deltaic | An efficient backup system supporting multiple data sources | Cloud Storage library

Create a user account to run backups. Create an LVM volume group, a thin-provisioning pool, and a thin-provisioned backup volume within it.
Create a user account to run backups.
Create an LVM volume group, a thin-provisioning pool, and a thin-provisioned backup volume within it. LVM limits the size of the thin-pool metadata volume to 16 GiB. This volume requires 64 bytes of capacity per data chunk, plus additional overhead for volume snapshots. Configure a 16 GiB metadata volume and a chunk size consistent with the expected maximum size of your pool. For example, for a 200 TiB pool, a chunk size of 2 MiB seems reasonable: $ thin_metadata_size -b 2m -s 200t -m 1000 -u g thin_metadata_size - 3.18 gibibytes estimated metadata area size for "--block-size=2mebibytes --pool-size=200tebibytes --max-thins=1000" So, for example, if your backup device is /dev/md0 and your user account is user: pvcreate /dev/md0 vgcreate backups /dev/md0 lvcreate -l 100%FREE -T backups/pool --chunksize 2m --poolmetadatasize 16g lvcreate -V200t -T backups/pool -n current mkfs.ext4 -m 0 -E lazy_itable_init /dev/backups/current mkdir /srv/backup mount /dev/backups/current /srv/backup chown user.user /srv/backup chmod 700 /srv/backup echo "/dev/backups/current /srv/backup ext4 user_xattr,discard,noatime 0 2" >> /etc/fstab
Create a config file in ~/.config/deltaic.conf. See example-config.yaml for the available settings. Here is a minimal config file for the above configuration: settings: root: /srv/backup backup-lv: backups/current
From the backup user account: virtualenv env env/bin/pip install git+https://github.com/cmusatyalab/deltaic env/bin/deltaic mkconf sudoers | sudo tee /etc/sudoers.d/deltaic env/bin/deltaic mkconf crontab --email your@email.address | crontab
Set up data sources by configuring the system as described below and then making the proper entries in ~/.config/deltaic.conf.
Create a filesystem (perhaps backed by the thin pool) for temporary data. For example:. If archives should be encrypted, use gpg2 --gen-key to generate one GPG signing key and one or more additional GPG keys for encryption. The signing key is used during archive creation and must not require a passphrase. Any of the encryption keys can be used during restore to decrypt the archives; they should be protected with a passphrase. Retain copies of all the keys offline in a safe place.
Create a filesystem (perhaps backed by the thin pool) for temporary data. For example: lvcreate -V20t -T backups/pool -n spool mkfs.ext4 -m 0 -E lazy_itable_init /dev/backups/spool mkdir /srv/spool mount /dev/backups/spool /srv/spool chown user.user /srv/spool chmod 700 /srv/spool echo "/dev/backups/spool /srv/spool ext4 discard,noatime 0 2" >> /etc/fstab
If archives should be encrypted, use gpg2 --gen-key to generate one GPG signing key and one or more additional GPG keys for encryption. The signing key is used during archive creation and must not require a passphrase. Any of the encryption keys can be used during restore to decrypt the archives; they should be protected with a passphrase. Retain copies of all the keys offline in a safe place.
Configure archive settings in deltaic.conf. See example-config.yaml for the available settings. Here is an example config fragment for storing encrypted archives to AWS: settings: archive-spool: /srv/spool archive-gpg-signing-key: [signing key fingerprint] archive-gpg-recipients: - [encryption key fingerprint] archivers: default: archiver: aws aws-access-key-id: [access key ID] aws-secret-access-key: [secret access key] aws-region: us-east-1 aws-namespace: [name of vault and database] aws-storage-cost: [dollars per GB-month in aws-region, if not 0.01]
Enable the commented-out cron jobs in the crontab created by deltaic mkconf crontab and configure their schedules as desired.