wowhead | I 'm not totally sure about this project

Assuming that the buy and sell quotes are always in the same order and distance from each other, you can try to use this:

If you want to print everything in a new line you should do file_out.write(r+'\n') to write every string on a new line, and you can remove duplicates by using set

Whatever is sent to the q= parameter of the _search endpoint is used as is by the query_string query (not org.elasticsearch.index.search.MatchQuery) which understands the Lucene expression syntax.

The query parser syntax is defined in the Lucene project using JavaCC and the grammar can be found here if you wish to have a look. The end-product is a class called QueryParser (see below).

The class inside the ES source code that is responsible for parsing the query string is QueryStringQueryParser which delegates to Lucene's QueryParser class (generated by JavaCC).

So basically, if you get an equivalent query string as what gets passed to _search?q=..., then you can use that query string with QueryParser.parse("query-string-goes-here") and run the reified Query using just Lucene.

Your code does not load these nodes because they do not exist in the HTML that is pulled back by HTML Agility Pack. This is probably because a large majority of the markup you have shown is generated by JavaScript. Just try inspecting the doc.ParsedText property in your ItemSetTransmogFromPage() method.

Html Agility Pack is an HTTP Client/Parser, it will not run scripts. If you really need to get the data using this process then you will need to use a "headless browser" such as Optimus to retrieve the page (caveat: I have not used this library, though a nuget package appears to exist) and then probably use HTML Agility Pack to parse/query the markup.

The other alternative might be to try to parse the JSON that exists on this page (if this provides you with the data that you need, although this appears unlikely).

Small note - I think the id in you xpath should be "tab-transmog-sets" instead of "tab - transmog - sets"

use get method

You have forgot about curly brackets:

You need to loop through all siblings between first .heading-size-3, till next header .heading-size-3

Best solution for CPU bound work (such as parsing) is to launch as many threads as the numbers of the cores in your machine, less than that and you are not taking advantage of all of your cores, more than that and excessive context-switching might kick in and hinder performance.

So essentially, threadnbrs should be set to Environment.ProcessorCount

Also, consider using the Parallel class instead of creating threads yourself: