crunchbase | Ruby Crunchbase API wrapper | REST library

Here is an example of how to get the text of each element inside a table.

When programming it's always good to build your code in logical steps. Your if statement too complex for what you are doing. Take it in simple steps as you have started. First get each line as a string (you have done this).

(line = reader.readLine()) != null

Then split the line by tab character (you have done this).

String[] field = line.split(splitBy);

Then extract the strings you want (you need to do this).

String ageStr = field[3];

String salaryStr = field[7];

Then validate each string you are interested in, ignoring those which done fit the criteria (you need to do this)

if(!ageStr.equals("NA")) { //is valid }

This approach then results in you only doing things with 'valid' elements and any invalid elements are ignored.

Here is refactored version using Stream API:

Here is modification of your code. I added an extra parameter to try and keep your overall structure and to produce your desired outcome and preventing repetitive call on array element.

According to the documentation:

The % wildcard (in a WHERE ... LIKE ... clause) matches zero or more characters

You have to allow that there may be characters after the term you are searching for.

For example, '%ell%' matches a record 'hello'. But '%ell' does not.

You should be able to use '%://%cruchbase.com/%some-query-term%' where some-query-term matches exactly how the string appears in the URL.

Can you give an example of a URL and the query you expect to retrieve it that is not working?

Because this element need to be clicked to show this page,like the code below:

The primary limiting factor will probably be the time taken to query the website. Currently, you're waiting for each query to return a result before executing the next one. The best way to speed up the workflow would be to execute batches of queries in parallel.

If you're using a Unix system you could try the following:

If you would like to receive a response you need to have some kind of proxy, your own like squidproxy, paid private proxy or public (or VPN as you mentioned). There is no way around it. You can spoof your IP in the packet you send to some false IP but then you will not receive a response. If you wish to use a proxy I would suggest going with excellent requests library as it is a tool of choice for many people doing web scrapping and using a proxy with it is just extremely easy. Example follows:

Can anyone shed any light on this? What the heck is -6 supposed to mean?

There is a contract that

A negative value -N indicates that the child was terminated by signal N (POSIX only).

In your case it means that the process was terminated by SIGABRT (code 6) signal

Is there a way to see a little more details about what happened there?

There is no much background info from your site. In general, try to play with different operators and files. Also, from my perspective Airflow is not well documented. And I recommend to check Airflow sources.