duplicity-unattended | Lightweight scripts for automating Duplicity | Continuous Backup library

First, create an S3 bucket and IAM user/group/policy with read-write access to it. The included cfn/host-bucket.yaml CloudFormation template can do this for you automatically. To apply it:. Alternatively, you can create the S3 bucket and IAM resources manually. Here are the general steps. Modify as you see fit.
Go to CloudFormation in the AWS console and click Create Stack.
Select the option to upload a template to S3 and pick the cfn/host-bucket.yaml template.
Fill in the stack name and bucket name. I suggest including the hostname in both for easy identification.
Accept remaining defaults and acknowledge the IAM resource creation warning.
Wait for stack setup to complete. If it fails, it's likely the S3 bucket name isn't unique. Delete the stack and try again with a different name.
Go to IAM in the AWS console and click on the new user. The user name is prefixed with the stack name so you can identify it that way.
Go to the Security credentials tab and click Create access key.
Copy the generated access key ID and secret key. You'll need them later.
Create the S3 bucket. Default settings are fine.
Create IAM policy with the following permissions: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::<bucket>/*", "arn:aws:s3:::<bucket>" ] } ] } Replace <bucket> with the bucket name.
Create IAM group with the same name as the policy and assign the policy to it.
Create IAM user for programmatic access. Add the user to the group. Don't forget to copy the access key ID and secret access key at the end of the wizard.
Install dependencies: Duplicity boto2 for Python 2 GnuPG Python 3 PyYAML for Python 3
Create new RSA 4096 keypair as the user who will perform the backups. If you're backing up system directories, this probably needs to be root. Do NOT set a passphrase. Leave it blank. gpg --full-generate-key --pinentry loopback
Make an off-host backup of the keypair in a secure location. I use my LastPass vault for this. Don't skip this step or you'll be very sad when you realize the keys perished alongside the rest of your data, rendering your backups useless. gpg --list-keys # to get the key ID gpg --armor --output pubkey.gpg --export <key_id> gpg --armor --output privkey.gpg --export-secret-key <key_id>
Delete the exported key files from the filesystem once they're secure.
Create a file on the host containing the AWS credentials. [Credentials] aws_access_key_id = <access_key_id> aws_secret_access_key = <secret_key> Replace <access_key_id> and <secret_key> with the IAM user credentials. Put it in a location appropriate for the backup user such as /etc/duplicity-unattended/aws_credentials or ~/.duplicity-unattended/aws_credentials.
Make sure only the backup user can access the credentials file. chmod 600 aws_credentials Change ownership if needed.
Copy the duplicity-unattended script to a bin directory and make sure it's runnable. chmod +x duplicity-unattended I usually clone the repo to /usr/local/share and add a symlink in usr/local/bin.
Copy the sample config.yaml file to the same directory as the AWS credentials file. (Or you can put it somewhere else. Doesn't matter.)
Customize the config.yaml file for the host.
Do a dry-run backup as the backup user to validate most of the configuration: duplicity-unattended --config <config_file> --dry-run Replace <config_file> with the path to the YAML config file. Among other things, this will tell you how much would be backed up.
Do an initial backup as the backup user to make sure everything really works: duplicity-unattended --config <config_file>
How do make sure backups keep working in the future? You can set up systemd to email you if something goes wrong, but I prefer an independent mechanism. The cfn/backup-monitor directory contains a CloudFormation template (SAM template, actually) with a Lambda function that monitors a bucket for new backups and emails you if no recent backups have occurred. To set it up for a new host/bucket, follow these steps:. Now let's test it. If all goes well, you will get an email with a summary of the most recent backups found in the bucket. From now on, the function will run once a day and email you only when there have been no recent backups for the number of days you specified. The function will look for recent backups in any S3 "folder" that contains at least one backup set from any time in the past. You can deploy additional stacks for each bucket you want to monitor.
If you have not used AWS Simple Email Service (SES) before, follow the instructions to verify the sender and recipient email addresses. See the overview documentation for more information.
Go to duplicity-unattended-monitor in the AWS Serverless Application Repository and click the Deploy button.
Review the template. (You wouldn't deploy a CloudFormation template into your AWS account without knowing what it does first, would you?)
Change the application/stack name. I suggest a name that includes the host or bucket for easy identification.
Fill in the remaining function parameters. Make sure the email addresses exactly match the ones you verified in SES.
Click Deploy and wait for AWS to finish creating all the resources.
Click on the function link under Resources. This will take you to the Lambda console for the function.
Click the Test button in the upper-right.
Create a new test event with the following content: {"testEmail": true} Give it a name like BackupMonitorTest and click Create.
Now you should see the new named event next to the Test button. Click the Test button again.
Install Pipenv for Python 3 if you don't already have it.
From the source repo directory, install the AWS SAM CLI into a virtual environment: pipenv install --dev
Change to the cfn/backup-monitor directory.
Set up your AWS CLI credentials so SAM can read them (e.g. using AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables).
Run the SAM command to package the CloudFormation template and upload the Lambda function to S3: pipenv run sam package --s3-bucket <code_bucket> --output-template-file packaged.yaml where <code_bucket> is an S3 bucket to which the AWS CLI user has write access.
You can now use the CloudFormation AWS console or the AWS CLI to deploy the packaged.yaml stack template that SAM just created.