Back

Technologies:

javascriptjavascript
typescripttypescript
avatar
Tolerim
24 days ago

How can I remove the first n bytes of an AsyncIterable<Uint8Array> using splice or shift?

The process of reading the first n bytes from a byte stream (in the form of an AsyncIterable) can be arduous and prone to errors. Is there a more optimal way to tackle this task?

async function shift(
  length: number,
  stream: AsyncIterable<Uint8Array>
): Promise<[Uint8Array, AsyncIterable<Uint8Array>]> {
  const prefix = new Uint8Array(length);

  let offset = 0;

  const iterator = streamSymbol.asyncIterator;

  while (true) {
    const { done, value } = await iterator.next();

    if (done) {
      throw new Error("Buffer underflow");
    } else {
      const chunk = value;
      if (chunk.length < length - offset) {
        prefix.set(chunk, offset);
        offset += chunk.length;
      } else {
        const slice = chunk.slice(0, length - offset);
        prefix.set(slice, offset);

        return [prefix, prepend(chunk.slice(slice.length), stream)];
      }
    }
  }
}

async function* prepend(
  prefix: Uint8Array,
  stream: AsyncIterable<Uint8Array>
) {
  yield prefix;
  yield* stream;
}
Answers(2)
avatar
Vogi
22 days ago
Verified Answer
Yes, there is a more optimal way to tackle this task. You can use the readable.read() method to read the first n bytes from a byte stream. Here is an example of how you can do it in JavaScript: 
async function shift(length, stream) {
  const prefix = new Uint8Array(length);

  let offset = 0;

  const iterator = stream[Symbol.asyncIterator]();

  while (true) {
    const { done, value } = await iterator.next();

    if (done) {
      throw new Error("Buffer underflow");
    } else {
      const chunk = value;
      if (chunk.length < length - offset) {
        prefix.set(chunk, offset);
        offset += chunk.length;
      } else {
        const slice = chunk.slice(0, length - offset);
        prefix.set(slice, offset);

        return [prefix, prepend(chunk.slice(slice.length), stream)];
      }
    }
  }
}

async function* prepend(prefix, stream) {
  yield prefix;
  yield* stream;
}
This code reads the first n bytes from an AsyncIterable byte stream and returns a tuple containing the first n bytes and the remaining byte stream. The shift() function reads the byte stream using an async iterator and concatenates the chunks until it has read enough bytes. The prepend() function returns a new AsyncIterable that starts with the given prefix. I hope this helps! Let me know if you have any other questions.

Sources

Reading the first n bits of a byte - Code Review Stack ExchangeNodeJS: how to read (at most) the first N bytes from a file?BufferedInputStream read() method in Java with Examplesjavascript - splice/shift first n bytes of AsyncIterable<Uint8Array ...
avatar
Tolerim
24 days ago
Yes, there is a more concise and elegant way to achieve this using the for await...of loop. Here is an example implementation: 
const prefix = new Uint8Array(length);
  let offset = 0;
  
  for await (const chunk of stream) {
    if (chunk.length < length - offset) {
      prefix.set(chunk, offset);
      offset += chunk.length;
    } else {
      const slice = chunk.slice(0, length - offset);
      prefix.set(slice, offset);
      return [prefix, prepend(chunk.slice(slice.length), stream)];
    }
  }
  
  throw new Error("Buffer underflow");
}

async function* prepend(prefix, stream) {
  yield prefix;
  yield* stream;
}
In this implementation, we are using the for await...of loop to iterate over the stream AsyncIterable, which allows us to iterate over the chunks of data as they become available. This eliminates the need to call stream[Symbol.asyncIterator]() explicitly and simplifies the loop condition. We are also using the yield* operator in the prepend function to yield all values from the stream iterable after the prefix value. This is a simpler and more concise way of achieving the same result as the while loop in the original implementation.
;