docker-logging | Log Docker containers running in AWS ECS | Continuous Deployment library

Prerequisites: Docker >= 1.8. If you use Docker-compose, make sure its version >= 1.5. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::xxxxxxxxxxxx:root" }, "Action": "es:", "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/" }, { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "es:", "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*", "Condition": { "IpAddress": { "aws:SourceIp": [ "192.168.1.0", "192.168.1.1" ] } } } ] }.
Spin up an Elasticsearch server. The easiest way to do this is via the AWS Elasticsearch Service:
Click "Create a new domain"
Set a domain name
Use the default options
Set an Access Policy. I suggest applying both IAM access to write to elasticsearch from your AWS account and IP-specific access so you can view logging outputs in Kibana. Here's a sample policy (be sure to change the region and xxxxxxxxxxxx with your 12-digit AWS account ID):
Make sure your EC2 Instance or Autoscaling Group has an Instance Profile and Role which grant write access to your Elasticsearch service. Here are example Cloud Formation resources which enable this. Make sure your AWS::EC2::Instance or AWS::AutoScaling::LaunchConfiguration have IamInstanceProfile set to the Instance Profile resource created by CloudFormation (in the example below, the setting would be "IamInstanceProfile": { "Ref": "EC2InstanceProfile" },): "EC2Role": { "Type": "AWS::IAM::Role", "Metadata": { "Comment": "Defines all permissions which an EC2 Instance attached to ECS Cluster should have" }, "Properties": { "AssumeRolePolicyDocument": { "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com" ] }, "Action": [ "sts:AssumeRole" ] } ] }, "Path": "/", "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role", "arn:aws:iam::aws:policy/AmazonESFullAccess", "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess", "arn:aws:iam::aws:policy/AmazonSQSFullAccess" ] } }, "EC2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "EC2Role" } ] } }
Submit an ECS task definition which uses the gelf logging driver. The ContainerDefinition should include a section like this: "logConfiguration": { "logDriver": "gelf", "options": { "gelf-address": "udp://localhost:12201", "tag": "nginx" } } Note the log option tag requires Docker > 1.9. For Docker 1.8, use gelf-tag. Otherwise, ECS may report Failed to initialize logging driver: unknown log opt 'tag' for gelf log driver". As of this writing, the CloudFormation AWS::ECS::TaskDefintiion does not support the logConfiguration settings of an ECS TaskDefinition. Watch the Cloudformation Release History to be notified when this will be supported.