crunchbase

Took a stab at it. Hope it helps or points you to a better solution.

I've used your code to assemble a small example, which you can see below.

1. Inside svg > defs, create one pattern per node and use that pattern (with the ID of the company) to fetch the logo of that company;
2. Reference the pattern for the node using the information you already have.

Some pointers on your code:

1. You already use ES6 logic, so you can also use Array.prototype.map and other functions. They're generally much more readable (and natively implemented!) than d3.map;
2. There is no need to keep so many arrays of values, generally having fewer sources of truth for your data will make the code simpler to maintain and update in the future;
3. Use clear variable names! LS and LT are logical when you know the context, but when you revisit this code in 6 months you might not instantly know what you were talking about when you wrote it.

Seems like news articles are generated dynamically in the backaground by javascript.

If you take a look at your web-inspector when loading your page you can see a request being made:

You can see it returns JSON data for news articles:

You have to replicate this request in your scraper code:

You use it in wrong moment. You click it in every category but you have to click it only when you selecte subcategories.

I needed different ID - dialog-0 instead of dialog-1 and I used shorter XPath.

I see two problems.

First: you try to get all checkboxes before loop but you should get only all categories, next run loop to click category and then search checkboxes for current category, and for every checkbox you should search checked

Second: this page displays non-standard widgets so it doesn't change checked in checkbox but set class in some non-standard widget.

There was also other problem. When I clicked category or checkbox then sometimes it was displaying popup message which was hidding other checkbox and code couldn't check it. I click label which also check checkbox but it doesn't display popup message

In two categories there are subcategories and it still needs code which will click subcategory, get checkboxes and check them. And next it has to click button to go back to categories.

There are many //mat-nav-list - one with categories, one with columns and subcategories and one with columns in subcategory - so I have to use index to work with correct list - and xpath starts indexes at 1 - (//mat-nav-list)[1], (//mat-nav-list)[2]. It needs also ( ) to select correct list because //mat-nav-list[1] and (//mat-nav-list)[1] mean different objects.

Minimal working code:

As mentioned in comments by Henry, there is limitation of pandas timestamps because of its representation in float64, you could probably work around it by parsing the date-time using datetime library when needed, otherwise letting it stay as string or convert it to an integer

Scenario 1: If you plan on showing this value only when you print it

Here is an example of how to get the text of each element inside a table.

When programming it's always good to build your code in logical steps. Your if statement too complex for what you are doing. Take it in simple steps as you have started. First get each line as a string (you have done this).

(line = reader.readLine()) != null

Then split the line by tab character (you have done this).

String[] field = line.split(splitBy);

Then extract the strings you want (you need to do this).

String ageStr = field[3];

String salaryStr = field[7];

Then validate each string you are interested in, ignoring those which done fit the criteria (you need to do this)

if(!ageStr.equals("NA")) { //is valid }

This approach then results in you only doing things with 'valid' elements and any invalid elements are ignored.