watermarking | Projet watermarking

We have an Apache Flink POC application which works fine locally but after we deploy into Kinesis Data Analytics (KDA) it does not emit records into the sink.

• Source: Kafka 2.7
  • 1 broker
  • 1 topic with partition of 1 and replication factor 1
• Processing: Flink 1.12.1
• Sink: Managed ElasticSearch Service 7.9.1 (the same instance as in case of AWS)

• Source: Amazon MSK Kafka 2.8
  • 3 brokers (but we are connecting to one)
  • 1 topic with partition of 1, replication factor 3
• Processing: Amazon KDA Flink 1.11.1
  • Parallelism: 2
  • Parallelism per KPU: 2
• Sink: Managed ElasticSearch Service 7.9.1

1. The FlinkKafkaConsumer reads messages in json format from the topic
2. The jsons are mapped to domain objects, called Telemetry

in order to debug our application, we save all incoming data (s3 sink) in a separate part of the graph (even before the timestamping/watermarking process). our data already includes timestamp (event timestamp), and before saving the data, we want to add one more field in which there will be a timestamp when the message actually got into the flink ( kind of processing time).

how best to do it? perhaps flink provides a special API for this now we are doing very simple new Date().getTime

I am designing an incremental update process for a cloud based database (Azure). The only existing changelog is a .txt file that records every insert, delete, and update statement that the database processes. There is no change data capture table available, or any database table that records changes and I cannot enable watermarking on the database. The .txt file is structured as follows:

I am new in Matlab and I have an assignment that is asking for watermarking an image using DCT transform:

• Read Lin.jpg color image and apply DCT.
• Threshold the logo.jpg (watermark) into binary and ten times of its strength, and then add it to the coefficient of transformed Lin image.

Here are the two images:

I have three questions:

1. Am I supposed to divide Lin.jpg into 8x8 blocks and logo.jpg into 2x2 blocks or that is not necessary?
2. what does it mean by: "ten times of its strength"? Is that just multiplying by 10?
3. How can I get the coefficient of transformed Lin.jpg image?

Here is what I tried:

I have an application which I am trying to make platform-independent with Python.

I have Python 3.x installed in all 3 OS-(Mac,Win10,Ubuntu)

I have a python script batch.py that calls other python scripts from within itself like this:

I'm working with Flink and I'm using the Kafka Connector. The messages that I'm receiving from flink is a list of comma separated items. "'a','b','c',1,0.1 ....'12:01:00.000'" One of them contain the event time, I would like to use this event time for the per-partition watermarking (in the kafka source), then use this Event Time for session windowing. My case is a bit different from usual because from what i have understood people usually use "kafka Timestamps" and SimpleStringSchema(). On my case instead I have to write my own deserializer that implement DeserializationSchema and return a Tuple or Pojo. So basically substitute the SimpleStringSchema() with my own function. Flink offer out of the box some deserializers but I really don't understnd how i can create a custom deserialization logic.

Checking the flink website i have found this:

https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/kafka.html

I have been given an example (Thanks David!), but still i don't get how to implement mine.

https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventDeserializationSchema.java

I would really need an example of how I can do it for a list. The one indicated above is for JSON so is giving me the theory, the concept but i got stuck there.

I'm writing a unit test for a Flink SQL statement that uses match_recognize. I'm setting up the test data like this

I want to join some events coming from a json based kafka source with a url field that has related data in a Mongodb collection. Then aggregate them including additional Mongodb data and output the data to a GCS sink.

When I run my structured streaming spark application my spark cluster starts filling disk space unlimitedly. I have configured watermarking to 0 seconds (as I'm aggregating only events from the current processing batch), so maximum state should be 1 or 2 batches. But I'm having this available disk space graph (get stable when killing the app):

Almost all the data filling my HDDs is located under: /hadoop/yarn/nm-local-dir/usercache/myuser/appcache/myapplication-id

If I disable the mongodb join, the available disk is stable over time but i need the joined data.

My mongodb collection which I want to join is about 11 GB large and my input kafka topic has about 3k records/sec.

My code looks like this:

I'm trying to take a PDF file and set an opacity level to the entire document or page. The PDF's are always a single page and contain vectors but no raster images like this PDF file and can have RGB or CMYK colors. In this case, I'm trying to set an opacity level of 0.5 so everything is half-transparent.

I found a lot about watermarking PDFs which I think it's easier because the content is added to the PDF, in my case I want to modify the PDF content. I found these amazing transparency operators for Ghostscript but I can't make them work! I created a very simple postscript program and called it program.ps: