wilds | A machine learning benchmark of in-the-wild distribution shifts, with data loaders, evaluators, and | Dataset library

You can just nest your two for loops! Append the values to a list if you want, but it will be big; if memory is a concern and you only need the values once I'd use a generator:

Why would it change?

If you do

Just as an idea:

• loop through the DOM elements
• save the normalized inner text to an array
• check on each element, if the normalized inner text is already in the array
• if so, delete the DOM element

As I see that, you can do this check in your example for-loop at the beginning.

-- updated with example --

Based on your comment I think you have to do the task from the other side: Just do the check in your loop width a storage array. Assuming a function called normalize to remove spaces, it should like this way:

To click on the element with text as login or register you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Using LINK_TEXT:

Use set_clip() to set the clipping region of the display surface.

e.g. Clip 100 rows at the top and the bottom:

Unless you remap your data to be able to fetch all items in one request you will need Pipeline resolvers.. In summary a pipeline is a number of resolvers in line wrapped in a before/after template.

In your case these before/after templates are not really used so the basic setup would be:

Before template (nothing is needed so an empty json is fine)

In case you mean something like a supervised learning AI system (like a neural network), I want to give you an idea of the problems you are going to face. There are three main problems:

Training data

To create an AI that automatically detects titles, you first need to train the AI. You will need hundreds, probably thousands of samples to train your AI. This means, you have to put a lot of manual work of getting and labeling input data before your AI can predict anything.

Input for your AI

What would you give as input to the AI? You have two options:

• The source code: This would basically train your AI to become a web browser. Fun challenge, but I doubt you will get any results at all out of this.
• A screen capture of the website? You should not feed the actual screenshot into the AI but instead try to "clean" it. For example, you could replace all texts with some kind of placeholder, as you want your AI to work on the "looks of the document" and not on the text itself. This approach will work better than the first one, but you will have to put some work into the cleaning of the image before it is given to your AI. This could work, but you need a lot of training data for this to reliably work.

Reliability

Even if you solve all problems and your AI is finally able to detect titles of lists, you need to be aware that an AI will never perfectly work. Of course, no algorithm will be perfect, but using an AI, you cannot simply "tweak" your algorithm to get better results.

Imagine, you see that for a specific website your AI does not work. You cannot debug your AI easily and adjust your code to make it work for that page. You will have to retrain your AI hoping that after your adjustments it will work. Maybe, then it will work for that page, but now another website will not work anymore...

Instead of using an AI, I recommend to use a simple algorithm to detect a list with heading elements inside. Something like the following could work:

Repeat the following for each heading tag (h2, h3, ...)

1. Get all heading elements (e.g. h2)
2. Is the number of elements <= 1, then this is not a list (skip)
3. Check the "surrounding elements" for each element. If this is a list, all elements should have the same surroundings: Do the parents of the elements have the same class name? Do the sibling nodes have the same classes? If not, this is not a list (skip)
4. Given the previous steps, the elements are very likely part of a list and your heading element should contain the title of the list.

There are some obvious drawbacks: This approach only works for a list of more than one element and only works if markup is correctly used. If the website is only using div elements instead of heading tags, this will not work. So, this should only serve as a starting point, the algorithm could obviously be improved...

You need to use the global keyword in each function to access variables defined in a parent. For example:

As far as I understand your question you would like to retrieve 2 parts - everything between first / and \ with 2 chars afterwards and then everything after \: