kiba | A small implementation of multithreaded Redis in Rust | Database library

I have a kiba job that takes a CSV file (with Kiba::Common::Sources::CSV), enrich its data, merge some rows (with the ChainableAggregateDestination destination described here) and saves it to another CSV file (with Kiba::Common::Destinations::CSV).

Now, I want to sort the rows differently (based on the first column) in my destination CSV. I can't find a way to write a transform that does this. I could use post_process to reopen the destination CSV, sort it and rewrite it but I guess there is a cleaner way...

Can someone point me in the right direction?

I'm looking at writing one of our ETL (or ETL like) processes in kiba and I wonder how to structure it. The main question I have is the overall architecture. The process works roughly like this:

1. Fetch data from an HTTP endpoint.
2. For each item returned from that API and make one more HTTP call
3. Do some transformations for each of the items returned from step 2
4. Send each item somewhere else

Now my question is: Is it OK if only step one is a source and anything until the end is a transform? Or would it be better to somehow have each HTTP call be a source and then combine these somehow, maybe using multiple jobs?

A third party system produces an HTML table of parent teacher bookings:

I'm working on an ETL pipeline with Kiba which imports into multiple, related models in my Rails app. For example, I have records which have many images. There might also be collections which contain many records.

The source of my data will be various, including HTTP APIs and CSV files. I would like to make the pipeline as modular and reusable as possible, so for each new type of source, I only have to create the source, and the rest of the pipeline definition is the same.

Given multiple models in the destination, and possibly several API calls to get the data from the source, what's the standard pattern for this in Kiba?

I could create one pipeline where the destination is 'the application' and has responsibility for all these models, this feels like the wrong approach because the destination would be responsible for saving data across different Rails models, uploading images etc.

Should I create one master pipeline which triggers more specific ones, passing in a specific type of data (e.g. image URLs for import)? Or is there a better approach than this?

Thanks.

It would be great if there was a way to get some kind of return object from a Kiba ETL run so that I could use the data in there to return a report on how well the pipeline ran.

We have a job that runs every 10 minutes that processes on average 20 - 50k records, and condenses them into summary records, some of which are created, and some of which are updated. The problem is, it's difficult to know what happened without trawling through reams of log files, and obviously, logs are useful to end users either.

Is there a way to populate some kind of outcome object with arbitrary data, as the pipeline runs? e.g

• 25.7k rows found in source
• 782 records dropped by this transformer
• 100 records inserted
• 150 records updated
• 20 records had errors (and here they are)
• This record had the highest statistic
• 1200 records belonged to this VIP customer
• etc.

And then at the end, use that data to send an email summary, populate a web page, render some console output, etc.

Currently, the only way I can see this working right now is to send an object in during setup and mutate it when it's flowing through the sources, transformers, and destinations. Once the run is complete, check the variable afterwards and do something with the data that is now in there.

Is this how it should be done, or is there a better way?

EDIT

Just want to add that I don't want to handle this in the post_process block, because the pipeline gets used via a number of different mediums, and I would want each use case to handle its own feedback mechanism. It's also cleaner (imo) for an ETL pipeline to not have to worry about where it's used, and what that usage scenario's feedback expectations are...

I've got a long-running pipeline that has some failing items (items that at the end of the process are not loaded because they fail database validation or something similar).

I want to rerun the pipeline, but only process the items that failed the import on the last run.

I have the system in place where I check each item ID (that I received from external source). I do this check in my loader. If I already have that item ID in the database, I skip loading/inserting that item in the database.

This works great. However, it's slow, since I do extract-transform-load for each of these items, and only then, on load, I query the database (one query per item) and compare item IDs.

I'd like to filter-out these records sooner. If I do it in transformer, I can only do it per item again. It looks like extractor could be the place, or I could pass records to transformer in batches and then filter+explode the items in (first) transformer.

What would be better approach here?

I'm also thinking about reusability of my extractor, but I guess I could live with the fact that one extractor does both extract and filter. I think the best solution would be to be able to chain multiple extractors. Then I'd have one that extracts the data and another one that filters the data.

EDIT: Maybe I could do something like this:

We have a Kiba pipeline where we need to do some task after the job has ended, no matter if there were errors or not (the whole pipeline doesn't fail, we just have couple of validation errors or similar).

This is what the documentation says:

:warning: Post-processors won't get called if an error occurred before them. https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors

Would this be recommended way to do this:

Firstly - Thibaut, thank you for Kiba. It goes toe-to-toe with 'enterprise' grade ETL tools and has never let me down.

I'm busy building an ETL pipeline that takes a numbers of rows, and reduces them down into a single summary row. I get the feeling that this should be a simple thing, but I'm a little stumped on how to approach this problem.

We have a number of CDR's from a voice switch, and need to condense them under some simple criteria into a a handful of summary records. So, the problem is; I have many thousands of records coming in from a Source, and need to transform them into only a few records based on some reduce criteria.

Kiba is really simple when there's a one-to-one Source -> Destination ETL, or even a one-to-many Source -> Destination with the new enumerable exploder in V3, but I don't see a clear path to many-to-one ETL pipelines.

Any suggestions or guidance would be greatly appreciated.

CONTEXT

• I'm new to Ruby and all that jazz, but I'm not new to dev.
• I'm taking over a project based on 2 rails/puma repositories for web & APIs.
• I'm building a new repository for a backend data processing app, using Kiba, that will run through scheduled jobs.
• Also, I'm to be joined by other devs later on, so I'd like to make something maintainable by design.

MY QUESTION : Should I use Rails on that ETL project?

Using it means we can apply the same folder structure as the other repos, use RSpec all the same etc. It also appeared to me that Rails changes the way classes like Hash act.

At the same time, it seems to bring unnecessary complexity to a project that will run on CLI and could consist of only a dozen of files.