magnolia | transparent generic derivation of typeclass instances | Functional Programming library

This is issue will be fixed in the next release 6.2.18. Please see https://jira.magnolia-cms.com/browse/MAGNOLIA-8284

This answer extends what if have commented already.
image is the response object for the request you are performing to google. The response itself is a stream. The response object can listen for the data event that will be triggered every time a new bit of data passed through the stream and was received by your server making the request. As the data event will be triggered more than once you are trying to set headers on a response that has already been send.
So what you are looking for is to somehow save each chunk of data that was received then process that data as a whole and finally send your response. It could look somewhat like the following. Here the Buffer.toString method might not be the best way to go as we are dealing with an image but it should give a general idea of how to do it.

The data is loaded dynamically so the data you are looking for is not in the html returned. You can use requests-html package to render the page. See below sample

The reason that your code doesn't work is that

Well you can't. Or rather you shouldn't. The form is means for user to enter the data. If you create data programatically, you either send it directly to the datasource or store it directly on the node that datasource operates on.

If you want to show generated data to user for approval prior saving, then you need to create custom field that would have the place for extra input and would react on user generated input directly prior to saving content.

Magnolia was sending the cookie all the time (as Set-Cookie header), it just never got saved. I had to process it manually for my FE to set the cookie and send it back in every subsequent request.

This is indeed a bug of HttpURLConnection.

Looking at the HTTP request going over the wire I can see the HTTP method is set to User-Agent: instead of GET:

And the log output show the same faulty request being sent:

In terms of first versus last title I think using let lastTitle = document.evaluate('descendant::span[contains(@class, "frame-title")][last()]', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue is a better way to read out the last span in document order that has the class as frame-title.

Nevertheless, it looks as if the web site uses paging so this would give you the last title on the first page.

As for passing DOM nodes around, I think for that you need page.evaluateHandle instead of page.evaluate.

I think for the the evaluate method you can pass JSON on:

You should address this problem using a json encoding/decoding library. Here is an example using circe and it's semi-automatic mode.

Since you mentionned in the comments that you are struggling with generic types in your case classes, I'm also including that. Basically, to derive an encoder or decoder for a class Foo[T] that contains a T, you have to prove that there is a way to encode and decode T. This is done by asking for an implicit Encoder[T] and Decoder[T] where you derive Encoder[Foo[T]] and Decoder[Foo[T]]. You can generalize this reasoning to work with more than one generic type of course, you just have to have an encoder/decoder pair implicitly available where you derive the encoder/decoder for the corresponding case class.