kinesis-consumer | Golang library for consuming Kinesis stream data | Pub Sub library

What would happen in the scenario you describe is that each of the 20 workers will eventually only process 1 shard.

At startup, each worker will try to claim as many shards by creating leases for those shards. When all 20 workers start simultaneously, they will all try to create leases for 20 shards, but this will not succeed for all of them. One worker may end up with eg 5 shards, and other ones with 2 or 3. After a few iterations of lease taking, though, each worker should have only 1 shard. This way the AWS rate limits are respected.

While this balancing process happens, it is possible for a short while for two workers to process the same records twice. This happens between the time that a worker steals a lease from another worker and that worker trying to update the lease and discovering that another worker has taken it, either by periodic refreshing or by checkpointing.

After this initial lease division, though, this will not happen anymore. When the workers are restarted, they resume the leases they had previously. But when a worker is down for a long time, other workers will take over its leases.

Kinesis has an at-least-once processing model because of this. It is best to design your application so that operations on the data are idempotent.

Scaling is useful if you want to be fault-tolerant (other workers will take over from a failed worker) or your data processing is so time-consuming that one worker would not be able to cope with 20 shards. Scaling beyond the number of shards is indeed only useful for fault-tolerance purposes.

You know it works as designed. Only the problem that it's not so obvious.

I have just pushed an integration test to the Kinesis Binder project.

You can find it in the KinesisBinderProcessorTests.

But general idea is like this: