pipeto | pipe for python

The pipeTo call is sending the HttpResponse to the actor, so you need to handle that in the receive method. But I would recommend creating a new message that includes the payload as well as the response, and send that to self. This allows you to describe the payload that caused the response.

The HttpResponse is being caught by case _ => and ignored, so it is generally a good idea to log any unexpected messages so that this sort of thing is caught earlier.

Example code:

Create a new class for the result:

I have found a suitable solution to my question, tinkerdoodle.cc.

Broadly, actors run on an dispatcher which selects a thread from a pool and runs that actor's Receive for some number of messages from the mailbox. There is no guarantee in general that an actor will run on a given thread (ignoring vacuous examples like a pool with a single thread, or a dispatcher which always runs a given actor in a specific thread).

That dispatcher is also a Scala ExecutionContext which allows arbitrary tasks to be scheduled for execution on its thread pool; such tasks include Future callbacks.

So in your actor, what happens when a messageA is received?

• The actor calls createApi() and saves it
• It calls the callA method on api
• It closes api
• It arranges to forward the result of callA when it's available to the sender
• It is now ready to process another message and may or may not actually process another message

What this actually means depends on what callA does. If callA schedules a task on the execution context, it will return the future as soon as the task is scheduled and the callbacks have been arranged; there is no guarantee that the task or callbacks have been executed when the future is returned. As soon as the future is returned, your actor closes api (so this might happen at any point in the task's or callbacks' execution).

In short, depending on how api is implemented (and you might not have control over how it's implemented) and on the implementation details, the following ordering is possible

• Thread1 (processing messageA) sets up tasks in the dispatcher
• Thread1 closes api and arranges for the result to be piped
• Thread2 starts executing task
• Thread1 moves on to processing some other message
• Thread2's task fails because api has been closed

In short, when mixing Futures and actors, the "single-threaded illusion" in Akka can be broken: it becomes possible for arbitrarily many threads to manipulate the actor's state.

In this example, because the only shared state between Futureland and actorland is local to the processing of a single message, it's not that bad: the general rule in force here is:

• As soon as you hand mutable (e.g. closeable) state from an actor to a future (this includes, unless you can be absolutely sure what's happening, calling a method on that stateful object which returns a future), it's best for the actor to forget about the existence of that object

How then to close api?

Well, assuming that callA isn't doing anything funky with api (like saving the instance in some pool of instances), after messageA is done processing and the future is completed, nothing has access to api. So the simplest, and likely most correct, thing to do is arrange for api to be closed after the future has completed, along these lines

Since the issue is with Kryo serialization, you can register your own custom Kryo serializers. But in my experience this hasn't worked all that well for reasons I don't completely understand (not always used). Plus Kryo serialization is going to be much slower than creating a POJO that Flink can serialize using built-in support. So add setters for every field, verify nothing gets logged about class Species missing something that qualifies it for fast serialization, and you should be all set.

The answer is in your question. persistence is the target of the ask.

It's possible that persistence is delegating work to another actor (e.g. a worker in a pool), but that would reveal implementation details and tbh is probably not going to be that useful (which is part of why the ask pattern doesn't propagate sender).

If it's a protocol you control, explicitly adding an ActorRef to the response is guaranteed to work.

You could roll your own version of the ask pattern which propagates the sender, though as noted above, it's probably not going to be that useful.

EDIT: to propagate persistence into the forwarded reply, the easiest is to map the Future result of the ask into something which bundles persistence with the result (as a sort of correlation ID), like:

Maybe you can set ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG and ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG in producer_config in FlinkKafkaProducer

props.put("key.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

The source interfaces where reworked in FLIP-27 to provide support for BATCH execution mode in the DataStream API. In order to get the FileSink to properly transition PENDING files to FINISHED when running in BATCH mode, you need to use a source that implements FLIP-27, such as the FileSource (instead of readTextFile): https://ci.apache.org/projects/flink/flink-docs-release-1.13/api/java/org/apache/flink/connector/file/src/FileSource.html.

As you discovered, that looks like this:

Ok, now after hours of troubleshooting I found out that the issue is not with my python or java setup or with pyflink.

The issue is my company proxy. I didn't think of networking, but py4j needs networking under the hood. Should have spent more attention to this line in the stacktrace:

StreamSaver is targeted for those who generate large amount of data on the client side, like a long camera recording for instance. If the file is coming from the cloud and you already have a Content-Disposition attachment header then the only thing you have to do is to open this URL in the browser.

There is a few ways to download the file:

• location.href = url
• download
• </code></li> <li>and for those who need to post data or use a other HTTP method, they can post a (hidden) <code><form></code> instead.</li> </ul> <p>As long as the browser does not know how to handle the file then it will trigger a download instead, and that is what you are already doing with <code>Content-Type: application/octet-stream</code></p> <hr /> <p>Since you are downloading the file using Ajax and the browser knows how to handle the data (giving it to main JS thread), then <code>Content-Type</code> and <code>Content-Disposition</code> don't serve any purpose.</p> <p>StreamSaver tries to mimic how the server saves files with ServiceWorkers and custom responses.<br /> You are already doing it on the server! The only thing you have to do is stop using AJAX to download files. So I don't think you will need StreamSaver at all.</p> <hr /> <h3>Your problem</h3> <p>... is that you first download the whole data into memory as a Blob first and then you save the file. This defeats the whole purpose of using StreamSaver, then you could just as well use the simpler FileSaver.js library or manually create an object url + link from a Blob like FileSaver.js does.</p> <pre><code>Object.assign( document.createElement('a'), { href: URL.createObjectURL(blob), download: 'name.txt' } ).click() </code></pre> <p>Besides, you can't use Angular's HTTP service, since they use the old <code>XMLHttpRequest</code> instead, and it can't give you a ReadableStream like <code>fetch</code> does from <code>response.body</code> so my advice is to just simply use the Fetch API instead.</p> <p><a href="https://github.com/angular/angular/issues/36246" rel="nofollow noreferrer">https://github.com/angular/angular/issues/36246</a></p>