breakword | Mixing breakpoints with print debugging | Code Inspection library

It's not .innerText that is null, but the return value of document.querySelector(…) itself. You'll want

You have just miss the closing ] bracket.

Try now

Problem is you have called function $('.text_readmore').each(function () { before adding the data in

.

Try this xpath:

I have recently run into the same issues using rvest to scrape PubChem. The problem is that the information on the page is rendered using javascript as you are scrolling down the page, so rvest is only getting minimal information from the page.

There are a few workarounds though. The simplest way to get the information that you need into R is using an R package called webchem.

If you are looking up name, CAS number, and molecular weight then you can do something like:

library(webchem) chem_properties <- pc_prop(1, properties = c('IUPACName', 'MolecularWeight'))

The full list of compound properties that can be extracted using this api can be found here. Unfortunately there isn't a property through this api to get CAS number, but webchem gives us another way to query that using the Chemical Translation Service.

chem_cas <- cts_convert(query = '1', from = 'CID', to = 'CAS')

The second way to get information from the page that is a bit more robust but not quite as easy to work with is by grabbing information from the JSON api.

library(jsonlite) chem_json <- read_json(paste0("https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/", "1", "/JSON/?response_type=save$response_basename=CID_", "1"))

With that command you'll get a list of lists, which I had to write a convoluted script to parse the information that I needed from the page. If you are familiar with JSON, you can parse far more information from the page, but not quite everything. For example, in sections like Literature, Patents, and Biomolecular Interactions and Pathways, the information in these sections will not fully show up in the JSON information.

The final and most comprehensive way to get all information from the page is to use something like Scrapy or PhantomJS to render the full html output of the PubChem page, then use rvest to scrape it like you originally intended. This is something that I'm still working on as it is my first time using web scrapers as well.

I'm still a beginner in this realm, but hopefully this helps you a bit.

The "strange" suggestions that you have like "wo r r i". You have them, because you're using WordBreakSolrSpellChecker and it breaks tokens trying to provide you some spellcheck capabilities, so if you will remove you shouldn't get these kind of suggestions. Here is the quote from the official documentation:

WordBreakSolrSpellChecker offers suggestions by combining adjacent query terms and/or breaking terms into multiple words. It is a SpellCheckComponent enhancement, leveraging Lucene's WordBreakSpellChecker. It can detect spelling errors resulting from misplaced whitespace without the use of shingle-based dictionaries and provides collation support for word-break errors, including cases where the user has a mix of single-word spelling errors and word-break errors in the same query. It also provides shard support.

So, basically, in your example - you're getting normal suggestions from Solr index like: worst, work, worm, worn, wors. all other are just the result of WordBreakSolrSpellChecker and you will never find them in your index.

I developed the following from a public website with almost identical structure to yours. (https://www.federalreserve.gov/releases/h3/current/)

Requires Reference to Microsoft Internet Controls and Microsoft HTML Object Library