Back

Technologies:

javascriptjavascript
typescripttypescript
avatar
Tolerim
25 days ago

How can a web worker handle large messages?

Currently, I am encountering a throttling issue with Web Workers.

I am attempting to accomplish the following:

The main thread sends requests to the Web Worker. Based on these requests, the Web Worker calls a web service, which returns roughly 1MiB of data.

The web worker expands this data to approximately 4MiB, processes it, and sends the result back to the main thread. However, deserializing the result message in the primary thread is time-consuming, taking between 20ms and 40ms.

Is there a way to share the initial object without serialization/deserialization to avoid unnecessary processing times? It appears that utilizing structuredClone is causing issues in my application since calling MessageEvent.data is the bottleneck in handling message reception.

Despite the challenges in communication between threads, the Web Worker is a valuable tool for managing large data sets and lengthy REST requests.

Answers(2)
avatar
Vogi
25 days ago
Verified Answer
You can use the transferable objects feature of the postMessage() method to avoid serialization/deserialization. The structured clone algorithm is used internally when invoking structuredClone(), to transfer data between Workers via postMessage(), storing objects with IndexedDB, or copying objects for other APIs. The structured clone algorithm copies complex JavaScript objects by recursing through the input object while maintaining a map of previously visited objects to detect circular references. It is used internally when invoking structuredClone(), to transfer data between Workers via postMessage(), storing objects with IndexedDB, or copying objects for other APIs. When passing data using postMessage(), a copy is still made. However, most browsers implement the structured cloning algorithm, which allows you to pass more complex types in/out of Workers such as File, Blob, ArrayBuffer, and JSON objects. I hope this helps! Let me know if you have any other questions.

Sources

Web Workers API: Structured Clone Algorithm - Web APIs - W3cubThe structured clone algorithm - Web APIs | MDN - Mozilla Developerhow to pass large data to web workers - Stack OverflowstructuredClone() global function - Web APIs | MDNWorker: postMessage() method - Web APIs | MDN - Mozilla Developer
avatar
Tolerim
25 days ago
Yes, there is a way to share the initial object between the main thread and web-worker without using structuredClone to avoid the serialization/deserialization process. You can use Transferable objects to transfer the data ownership between the two threads. Here's an example of how you can use Transferable objects to transfer an ArrayBuffer from the main thread to a web-worker: 
// Main thread
const buffer = new ArrayBuffer(1024 * 1024); // Allocate 1MiB of memory
worker.postMessage(buffer, [buffer]); // Transfer the buffer ownership to the worker

// Web-worker
self.onmessage = function(event) {
  const buffer = event.data; // Transfer the buffer ownership from the main thread
  // ... process the data ...
  const result = new ArrayBuffer(4 * buffer.byteLength); // Allocate 4MiB of memory
  self.postMessage(result, [result]); // Transfer the result ownership to the main thread
};
In this example, we have created an ArrayBuffer of 1MiB in the main thread and passed it to the web-worker using worker.postMessage() method. The second argument of this method is an array of Transferable objects. By passing the buffer in the array, we transfer the ownership of the buffer object to the web-worker. The web-worker then processes the data and creates a new ArrayBuffer of 4MiB, which is again transferred to the main thread using self.postMessage() method with an array of Transferable objects. By using Transferable objects, we can avoid the serialization/deserialization process and communicate between threads faster.
;