fast-csv | CSV parser and formatter for node | CSV Processing library

COPY is a single statement and can only apply atomically - even if you are streaming in the data in chunks. By the time you catch the exception, the COPY statement has been aborted, none of the rows prior to the violation occurring will be queryable, and the COPY cannot be resumed. In a case where you have, say, 10 rows, and a single row causes a violation, the only way of getting those other 9 rows present is with an INSERT or a COPY that does not include the problematic row. In practice, this means using single-row inserts, possibly with savepoints so you don't have to do a transaction-per-row.

Another approach you may want to consider is to COPY into a table with no constraints, use regular DML to duplicate non-violating rows into the real data, and then drop/truncate the table you used for the import.

But fundamentally, a "resumable, violation-tolerant COPY" just isn't a thing; the statement is the finest-grained level at which an operation can succeed or fail in PG, and COPY is still just one statement.

This issue is related to CPU instructions used by current version sharp.js lib that are not supported by some older processors. Resolved by setting exact older version of sharp.js lib, based on this useful answer here:

NodeJS: Illegal instruction (core dumped) Error after using sharp library

this solved issue

This is the Excel issue with guessing a CSV encoding. Just indicate the option writeBOM equals to true, that way Excel can guess that the encoding is UTF-8.
The most reliable way is to write to XLSX-file directly to avoid such encoding issues.

By the time the catch block is reached, the socket of the res object has already been destroyed by the pipeline. A pipeline "forwards errors" and destroys the destination if an error occurs in the source or along the intermediate transformations.

Try the following:

I've extended my on("end") event with the following and some helper functions under. It works out for now.

It seems you are getting the same id, because all models in Sequelize are "prepared" once, and your function uuidv4() runs only once. Thus for every row you insert into the DB it will have the same uuid. You could try mitigating it like this:

According to the docs here: https://c2fo.github.io/fast-csv/docs/formatting/methods#write, the first parameter to .write is the data rows that you want to write, but you have the options as the first parameter.

Try something like this:

I rewrote the headers in a map function to take schema a and schema b and turn them into the target schema

First up, copying settings.js to /usr/app/node-red/.node-red will do nothing as it will be ignored. The usrDir for the Node-RED Docker container is /data so the settings.js needs to be copied to there.

Second, to install extra nodes add them to the package.json in /data not /usr/src/node-red. Then run npm install ... in the /data directory, this will install the nodes into /data/node_modules.

If you want to remove any of the core nodes then you need to included their filenames in the nodesExcludes key in the settings.js as follows: