stream-json | js stream components for creating custom JSON processing | JSON Processing library

A data event will be emitted each time a chunk is processed without considering whether an entity has been inserted. So data events will be emitted much faster than strapi can process it.

See Backpressuring in Streams for more informations

You should use core pipeline function with a custom writable stream and call done when entity has been created :

I don't want to count anything, I just want to write to file

In that case, you'll need to convert the token/JSON data stream back into a text stream that you can write to a file. You can use the library's Stringer for that. Its documentation also contains an example that seems to be more in line with what you want to do:

If I understood you correctly you want something like this. I used an ObjectBuilder class that combines all methods to build one JSON object. It uses parentStack to keep track of all objects and arrays. When the object/array is started with startObject/startArray a new JSON object/array is pushed onto the stack. Once this object/array is completed it is popped off of the stack. The last object that is popped off of the stack is the whole item object and can be processed further (in the example below I just print it out).

The current object or array that is currently being constructed is always on top of the stack.

I had to use a subset of the sample you provided because it did not contain a matching number of startObject and endObject items, which resulted into an invalid JSON. I included this subset below the code.

Hopefully, this is what you were looking for :)

(Note, I only wrapped buildItem() function in runSample() function so that I can include the sample JSON at the bottom to make it look neater in this online editor. You can move buildItem() function outside.)

Like I described in the latter part of my question, directly piping the res (my response to the client) to dataStream (the data stream I got from the external API) allowed to stream without any issues.

Extending the same behavior, I created a Readable stream which is equivalent to the response I should send to my client. Then I piped it to res and it worked.

Here is my solution.

As the code related to the asynchronous pipeline is not promisified, you currently have no way to return a promise that resolves on getting the "end" event.

Your await does wait for the promise to resolve, but your then callback returns undefined, and so that resolves the promise at that very moment, long before the end event is broadcast.

So change this: