batch-file-ideas | Everything I create for batch processing | Batch Processing library

I have a simple batch file with which I want to use the wkhtmltopdf to create PDF files of an archived set of URLs.

The simple command of my batch file for wkhtmltopdf is as follows

I wanted to achieve an incremental load/processing and store them in different places using Azure Data Factory after processing them, e.g:

External data source (data is structured) -> ADLS (Raw) -> ADLS (Processed) -> SQL DB

Hence, I will need to extract a sample of the raw data from the source, based on the current date, store them in an ADLS container, then process the same sample data, store them in another ADLS container, and finally append the processed result in a SQL DB.

ADLS raw:

2022-03-01.txt

2022-03-02.txt

ADLS processed:

2022-03-01-processed.txt

2022-03-02-processed.txt

SQL DB:

All the txt files in the ADLS processed container will be appended and stored inside SQL DB.

Hence would like to check what will be the best way to achieve this in a single pipeline that has to be run in batches?

I have a lot of half-hourly files with names in the form A_B_C_YYYYMMDDThhmm_YYYYMMDDThhnn.csv A, B and C are words where YYYY is the year in 4 digits, MM the month in 2 digits, DD the day of the month in 2 digits, hh the hour of the day in 2 dgits, mm the minute in 2 digits, and nn is mm+30 minutes

How can I move these files into folders based on years and months using Powershell?

I have made an attempt based on a previously used script but it is not producing any output for my filename format, presumably because the match string is wrong:

I am writing a demo C application in batch mode, which will try to read from a file as input. The command is : metric

The C source file is:

I have three services in place, each of which generate a certain JSON payload (and take different times to do so) that is needed to be able to process a message which is the result of combining all three JSON payloads into a single payload. This final payload in turn is to be sent to another Kafka Topic so that it can then be consumed by another service.

Below you can find a diagram that better explains the problem at hand. The information aggregator service receives a request to aggregate information, it sends that request to a Kafka topic so that Service 1, Service 2 and Service 3 consume that request and send their data (JSON Payload) to 3 different Kafka Topics.

The Information Aggregator has to consume the messages from the three services (Which are sent to their respective Kafka Topics at very different times e.g. Service 1 takes half an hour to respond while service 2 and 3 take under 10 minutes) so that it can generate a final payload (Represented as Aggregated Information) to send to another Kafka Topic.

After having researched a lot about Kafka and Kafka Streams, I came across this article that provides some great insights on how this should be elaborated.

In this article, the author consumes messages from a single topic while in my specific use case I must consume from three different topics, wait for each message from each topic with a certain ID to arrive so that I can then signal my process that it can proceed to consume the 3 messages with the same ID in different topics to generate the final message and send that final message to another Kafka topic (Which then another service will consume that message).

My thoughts are that I need to have a Kafka Stream checking all three topics and when it sees that has all the 3 messages available, send a message to a kafka topic called e.g. TopicEvents from which the Information Aggregator will be consuming and by consuming the message will know exactly which messages to get from which topic, partition and offset and then can proceed to send the final payload to another Kafka Topic.

• Am I making a very wrong use of Kafka Streams and Batch Processing?

• How can I signal a Stream that all of the messages have arrived so that it can generate the message to place in the TopicEvent so as to signal the Information Aggregator that all the messages in the different topics have arrived and are ready to be consumed?

Sorry for this long post, any pointers that you can provide will be very helpful and thank you in advance

When invoking Azure ML Batch Endpoints (creating jobs for inferencing), the run() method should return a pandas DataFrame or an array as explained here

However this example shown, doesn't represent an output with headers for a csv, as it is often needed.

The first thing I've tried was to return the data as a pandas DataFrame and the result is just a simple csv with a single column and without the headers.

When trying to pass the values with several columns and it's corresponding headers, to be later saved as csv, as a result, I'm getting awkward square brackets (representing the lists in python) and the apostrophes (representing strings)

I haven't been able to find documentation elsewhere, to fix this:

I'm using Firebase as the backend to my Flutter project. I need to write to multiple nodes in one transaction. Now I have:

I'm performing 2 big for loop tasks on a dataframe column. The context being what I'm calling "text corruption"; turning perfectly structured text into text full of both missing punctuation and misspellings, to mimic human errors.

I found that running 10,000s rows was extremely slow, even after optimizing the for loops.

I discovered a process called Batching, on this post.

The top answer provides a concise template that I imagine is much faster than regular for loop iterations.

How might I use that answer to reimplement the following code? (I added a comment to it asking more about it).

Or; might there be any technique that makes my for loops considerably quicker?

I am using deployer partition handler for remote partitioning in spring batch. I want to get the status of each worker node at regular intervals and display it to the user. ( Like heartbeats ). Is there any approach to achieve this ?