kinesis | A Node.js stream implementation of Amazon 's Kinesis

Im making some event sourcing using dms, and i need to include dbname on the metadata that is sent from some postgresql databases to kinesis (as target) i got here https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Tablesettings.html

i runned out of ideas. Adding a tag with dbname to the message probably works too but i cant find some precise documentation of how to do that

I'm using one of the Docker images of EMR on EKS (emr-6.5.0:20211119) and investigating how to work on Kafka with Spark Structured Programming (pyspark). As per the integration guide, I run a Python script as following.

We have an Apache Flink application which processes events

• The application uses event time characteristics
• The application shards (keyBy) events based on the sessionId field
• The application has windowing with 1 minute tumbling window
  • The windowing is specified by a reduce and a process functions
  • So, for each session we will have 1 computed record
• The application emits the data into a Postgres sink

Application:

• It is hosted in AWS via Kinesis Data Analytics (KDA)
• It is running in 5 different regions
• The exact same code is running in each region

Database:

• It is hosted in AWS via RDS (currently it is a PostgreSQL)
• It is located in one region (with a read replica in a different region)

Because we are using event time characteristics with 1 minute tumbling window all regions' sink emit their records nearly at the same time.

What we want to achieve is to add artificial delay between window and sink operators to postpone sink emition.

We have thought that we can add some sleep to evictor's evictBefore like this

I'm not seeing how an AWS Kinesis Firehose lambda can send update and delete requests to ElasticSearch (AWS OpenSearch service).

Elasticsearch document APIs provides for CRUD operations: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html

The examples I've found deals with the Create case, but doesn't show how to do delete or update requests. https://aws.amazon.com/blogs/big-data/ingest-streaming-data-into-amazon-elasticsearch-service-within-the-privacy-of-your-vpc-with-amazon-kinesis-data-firehose/ https://github.com/amazon-archives/serverless-app-examples/blob/master/python/kinesis-firehose-process-record-python/lambda_function.py

The output format in the examples do not show a way to specify create, update or delete requests:

I'm having a problem understanding how to preserve the order of events when consuming records from a Kinesis stream with Flink. Our setup looks like this:

• Kinesis stream with 8 shards
• Sharding key is the userId of the user that produced the event

In Flink, we use the Table API to consume the Kinesis stream, do some processing and write the events to a (custom) synchronous HTTP sink. The desired outcome would be that each shards processing subtask writes the events to the sink one after the other, waiting for the sink to return before writing the next event. In order to test that, we made sink functions randomly do a Thread.sleep() for a few seconds before returning. Looking at the log output, we can now see this:

I am evaluating different streaming/messaging services for use as an Event Bus. One of the dimensions I am considering is the ordering guarantee provided by each of the services. Two of the options that I am exploring are AWS Kinesis and Kafka and from a high level, according it looks like they both provide similar ordering guarantees where records are guaranteed to be consumable in the same order they were published only within that shard/partition.

It seems that AWS Kinesis APIs expose the ids of the parent shard(s) such that Consumer Groups using KCL can ensure records with the same partition key can be consumed in the order they were published (assuming a single threaded publisher) even if shards are being split and merged.

My question is, does Kafka provide any similar functionality such that records published with a specific key can be consumed in order even if partitions are added while messages are being published? From my reading, my understanding of partition selection (if you are specifying keys with your records) behaves along the lines of HASH(key) % PARTITION_COUNT. So, if additional partitions are added, they partition where all messages with a specific key will be published may (and I've proven it does locally) change. Simultaneously, the Group Coordinator/Leader will reassign partition ownership among Consumers in Consumer Groups receiving records from that topic. But, after reassignment, there will be records (potentially unconsumed records) with the same key found in two different partitions. So, from the Consumer Group level is there no way to ensure that the unconsumed records with the same key now found in different partitions will be consumed in the order they were published?

I have very little experience with both these services, so my understanding may be flawed. Any advice is appreciated!

I have a Destination Stream that looks as follows for example :

I am trying this in the AWS lambda console. I have installed npm install @aws-sdk/client-kinesis on my terminal and used zipped the file and created a lambda layer which has client-kinesis.

If use the following it works!

I am using dynamodb and I'd like to enable dynamodb stream to process any data change in the dynamodb table. By looking at the stream options, there are two streams Amazon Kinesis data stream and DynamoDB stream. From the doc of these two streams, both are handling the data change from dynamodb table but I am not sure what the main different between using these two.