user_agent | Generator of User-Agent header

self.quotes = [q.upper() for q in f.read().split('\n') if q]

As you can see here, q, which is a line from your text file, gets converted to uppercase inside the list comprehension. upper is a method of string that converts all lowercase characters in a string to uppercase.

Remove the call to upper and you should be fine.

The error is caused by the transformer model that flair loads. The cache directory for transformers has to be specified in additional by setting the environment variable TRANSFORMERS_CACHE=/path/to/transformers

For those two, you can use the following css selector:

For me, at least, I had to extract a javascript object containing the data with regex, then parse with json into json object, then grab the value associated with the page html as you see it in browser, soup it and then extract the paragraphs. I removed the retries stuff; you can easily re-insert that.

What type is the ffmpegcmd variable? Is it a string or a list/sequence?

Note that Windows and Linux/POSIX behave differently with the shell=True parameter enabled or disabled. It matters whether ffmpegcmd is a string or a list.

Direct excerpt from the documentation:

On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.

Is this the expected behavior?

Yes.

Middlewares are chained, so order of insertion matters. In the Go-chi source code you can see inside the Use function:

mx.middlewares = append(mx.middlewares, middlewares...)

where middlewares is a slice of functions middlewares []func(http.Handler) http.Handler.

this would require a separate middleware to be mounted before the loggingMiddleware.

Yes, correct.

by defining schema that matches your JSON, I can read it easily

Thanks to Serhii's suggestion, I found that the issue was due to "accept-encoding": "gzip, deflate, br": I accepted compressed sites but did not handle them in scrapy.

Adding scrapy.downloadermiddlewares.httpcompression or simply removing the accept-encoding line fixes the issue.

I'd like to suggest an alternative solution - one that doesn't involve BeautifulSoup.

I visited the main page and clicked on an album, while at the same time logging my network traffic. I noticed that my browser made an HTTP POST request to a GraphQL API, which accepts a custom query string as part of the POST payload which dictates the format of the response data. The response is JSON, and it contains all the information we requested with the original query string (in this case, all artists for every track of a playlist). Normally this API is used by the page to populate itself asynchronously using JavaScript, which is what normally happens when the page is viewed in a browser like it's meant to be. Since we have the API endpoint, request headers and POST payload, we can imitate that request in Python to get a JSON response: