amazon-kinesis-data-generator | simplifies testing with Amazon Kinesis | AWS library

So after a few days of pain I realised my mistake/quirk of Firehose set up.

My Elasticsearch cluster is Multi-AZ. When creating the Firehose, I was just letting it choose the default Security Groups based on the pre-existing Elasticsearch Domain. I only had one security group defined on the Firehose. It needs two if your Elasticsearch is in a VPC.

• One for Firehose outbound
• One for Elasticsearch inbound (you probably already have this)

Then they need to be joined together using the SG rules.

The Firehose Delivery Stream setup wizard will not warn you that you only have one security group and that this won't work. (Perhaps it would work for an Elasticsearch Domain on a single-AZ but I haven't tested.)

You must create the two required Security Groups before you create the Firehose Delivery Stream.

Create the first Security Group for the Firehose endpoint allowing HTTPS/433 outbound traffic. Then make sure the Elasticsearch Domain Security Group allows inbound HTTPS/433 traffic, specifically from the Firehose endpoint Security Group you just created.

Here is a diagram of what is needed (from this blog post: Ingest streaming data into Amazon Elasticsearch Service within the privacy of your VPC with Amazon Kinesis Data Firehose)