kiba | Data processing & ETL framework for Ruby | Data Migration library

Give a go at this..

• Paste this into a new class file:

Use JsonUtils which will create the classes for you and then you can copy these into your project.

In this instance it will create:

I chose to act on the output string instead of tackling the OCR API.

Fixing the issue within the OCR API would probably be a superior solution if possible, but I could not get your code properly referenced in my system.

So you can add this function to transpose the string

This requires some more javascript to work. However, once implemented it should work better. First save the following javascript to 'script.js' in your project's root directory (or a subdirectory, if you change the path in code). Make sure you select the file's properties and select Copy to output directory: Copy if newer. That copies the script file so WebView2 can find it.

Here's the javascript:

You can think of LinkingObjects almost as a computed property - it automagically creates an inverse link to the parent object when the child object is added to the parent objects List.

So when a Dog is added to a person's dogs list, a person reference is added to the Dog's walkers list. Keeping in mind that it's a many to many relationship so technically if Person A adds Doggo, and Person B adds Doggo, the Doggo's inverse relationship 'walkers' will contain Person A and Person B

the app still can run normally without any diff

Which is true, it doesn't affect he operation of the app. HOWEVER the difference is that by removing the walkers LinkingObjects, there's no way to query Dogs for their Person and get Dog Results (i.e. you can't traverse the graph of the relationship back to the person)

In other words we can query Person for kinds of dog stuff

To sort rows, a good strategy is to use an "aggregating transform", as explained in this article, to store all the rows in memory (although you could do it out of memory), then at transform "close" time, sort them and re-emit them in the pipeline.

This is the most flexible design IMO.

It is indeed best to use a single source, that you will use to fetch the main stream of the data.

General advice: try to work in batches as much as you can (e.g. pagination in the source, but also bulk HTTP lookup if the API supports it in step 2).

The source in your case could be a paginating HTTP resource, for instance.

A first option to implement it would be to write write a dedicated class like explained in the documentation.

A second option is to use Kiba::Common::Sources::Enumerable (https://github.com/thbar/kiba-common#kibacommonsourcesenumerable) like this:

Kiba author here!

I see at least two ways of doing this (no matter if you work with plain Ruby or with Kiba):

• converting your HTML to a table, then work from that data
• work directly with the HTML table (using Nokogiri & selectors), applicable only if the HTML is mostly clean

In all cases, because you are doing some scraping; I recommend that you have a very defensive code (because HTML changes and can contain bugs or cornercases later), e.g. strong assertions on the fact that the lines / columns contain what you expect, verifications etc.

If you go plain Ruby, then for instance you could do something like (here modelizing your data as text separated with commas to keep things clear):

Kiba author here!

It is natural & common to look for some form of genericity, modularity and reusability in data pipelines. I would say though, that like for regular code, it can be hard initially to figure out what is the correct way to get that (it will depend quite a bit on your exact situation).

This is why my recommendation would be instead to:

• Start simple (on one specific job)
• Very important: make sure to implement end-to-end automated tests (use webmock or similar to stub out API requests & make tests completely isolated, create tests with 1 row from source to destination) - this will make it easy to refactor stuff later
• Once you have that (1 pipeline with tests), you can start implementing a second one, and refactor to extract interesting patterns as reusable bits, and iterate from there

Depending on your exact situation, maybe you will extract specific components, or maybe you will end up extracting a whole generic job, or generic families of jobs etc.

This approach works well even as you get more experience working with Kiba (this is how I gradually extracted the components that you will find in kiba-common and kiba-pro, too.