Ansible Role duplicity¶
This role configures daily file-based backups using duplicity. Currently, this role is focused on using OpenStack Object Storage („Swift“) as the storage backend.
Note that this role does not support running with --check, as it first creates a GPG-Key which is required for the rest of the role.
duba (Duplicity Backup)¶
The role comes with the special Python wrapper script duba for duplicity, implemented by Linuxfabrik. The script currently does a massive parallel backup to a Swift storage backend with duplicity, where the number of duplicity processes is min(processor count, 6) + 1. The script’s configuration file is located at /etc/duba/duba.json.
To start a backup, simply call duba (or duba --config=/etc/duba/duba.json --command=backup). Have a look at duba --help for details.
Mandatory Requirements¶
On RHEL-compatible systems, enable the EPEL repository. This can be done using the linuxfabrik.lfops.repo_epel role.
Install
duplicity,python-swiftclientandpython-keystoneclientinto a Python 3 virtual environment in/opt/python-venv/duplicity. This can be done using the linuxfabrik.lfops.python_venv role.
Attention
Make sure the virtual environment is not writable by other users to prevent privilege escalation. This is also done by the linuxfabrik.lfops.python_venv role.
Optional Requirements¶
Create a symbolic link from
/opt/python-venv/duplicity/bin/duplicityto/usr/local/bin/duplicityfor easier usage on the command line.Either configure journald to persist your logs and do the rotating, or use logrotated.
Mandatory Role Variables¶
duplicity__gpg_encrypt_master_key_block
The ASCII-armored public master GPG key. Obtain it using
gpg --armor --export $GPG_KEY. This key is imported on the server and is used in addition to the server’s own local GPG key to encrypt the backups. This means that the backups can be restored using either the master or the server’s local private key (which is pretty cool in case of a desaster recovery). Be aware of the empty line between-----BEGIN PGP PUBLIC KEY BLOCK-----and your public key block.Type: String.
duplicity__gpg_encrypt_master_key
The long key ID of the master GPG key. Obtain it using
gpg --list-keys --keyid-format=long(after importing the key) orgpg /path/to/keyfile.Type: String.
duplicity__swift_login
The Swift username and password. Usually, this is given by the provider of the Swift Storage.
Type: Dictionary.
Subkeys:
username:Mandatory. The Swift username.
Type: String.
password:Mandatory. The Swift password.
Type: String.
Example:
# mandatory
duplicity__gpg_encrypt_master_key_block: |-
-----BEGIN PGP PUBLIC KEY BLOCK-----
6ec3d2aed2a54122817ca02b43a7e340kgKEdlbmVyYXRlZCBieSBBbnNpYmxlLi
...
-----END PGP PUBLIC KEY BLOCK-----
duplicity__gpg_encrypt_master_key: 'LLZGH2BITI2LRLJCLFWEAJQ93N6MWTKBARQDMYX5'
duplicity__swift_login:
username: 'SBI-MF827483'
password: 'linuxfabrik'
Optional Role Variables¶
duplicity__backup_backend
The backup backend being used. Possible options:
swift,sftp.Type: String.
Default:
'swift'
duplicity__backup_dest_container
The Swift container. This can be used to separate backups on the destination. By default, this will be used in
duplicity__backup_dest.Type: String.
Default:
'{{ ansible_nodename }}'
duplicity__backup_dest
The backup destination. This will be used in combination with the backup source path to create the target URL for
duplicity.Type: String.
Default:
'swift://{{ duplicity__backup_dest_container | regex_replace("/$", "") }}'
duplicity__backup_full_if_older_than
After how long a full backup instead of a incremental one should be done. Time Formats:
s,m,h,D,W,M, orY.Type: String.
Default:
'30D'
duplicity__backup_retention_time
The retention time of the backups. Time Formats:
s,m,h,D,W,M, orY.Type: String.
Default:
'30D'
duplicity__backup_sources__host_var / duplicity__backup_sources__group_var
List of dictionaries with directories to backup.
Type: List of dictionaries.
Default:
/backup/etc/home/opt/root/var/spool/cron
Subkeys:
path:Mandatory. Path to the folder to be backed up.
Type: String.
divide:Optional. Whether to split a large directory at its first level to perform parallel backups. Imagine a computer with 4 processor cores and the folder
datacontaining 100 files and folders. Ifdivideis set totrue,dubawill start and control 5 duplicate processes at once to speed up the backup process by almost a factor of 5.Type: Bool.
Default:
false
excludes:Optional. List of patterns that should not be included in the backup for this
path.Type: List of strings.
Default:
[]
state:Optional. Either
presentorabsent.Type: String.
Default:
'present'
duplicity__excludes
List of global exclude shell patterns for
duplicity. Have a look atman duplicityfor details.Type: List of strings.
Default:
['**/*.git*', '**/*.svn*', '**/*.temp', '**/*.tmp', '**/.cache', '**/cache', '**/log']
duplicity__loglevel
Set the loglevel. Possible options:
error,warning,notice,info,debug.Type: String.
Default:
'notice'
duplicity__logrotate
Log files are rotated
countdays before being removed or mailed to the address specified in alogrotatemail directive. If count is0, old versions are removed rather than rotated. If count is-1, old logs are not removed at all (use with caution, may waste performance and disk space).Type: Number.
Default:
{{ logrotate__rotate | d(14) }}
duplicity__on_calendar_hour
A shorthand to set the hour of
duplicity__on_calendar.Type: String.
Default:
'23'
duplicity__on_calendar
The
OnCalendardefinition for the daily systemd timer. Have a look atman systemd.time(7)for the format.Type: String.
Default:
'*-*-* {{ duplicity__on_calendar_hour }}:{{ 59 | random(seed=inventory_hostname) }}'
duplicity__sftp_password
Password for SSH User that is used by SFTP connection.
Type: String.
Default: unset
duplicity__swift_authurl
The Authentication URL for Swift. Usually, this is given by the provider of the Swift Storage.
Type: String.
Default:
'https://swiss-backup02.infomaniak.com/identity/v3'
duplicity__swift_authversion
The Authentication Version for Swift. Usually, this is given by the provider of the Swift Storage.
Type: String.
Default:
'3'
duplicity__swift_tenantname
The Swift Tenantname. Usually, this is given by the provider of the Swift Storage.
Type: String.
Default:
'sb_project_{{ duplicity__swift_login["username"] }}'
duplicity__timer_enabled
The state of the daily systemd timer.
Type: Bool.
Default:
true
Example:
# optional
duplicity__backup_dest: 'swift://{{ duplicity__backup_dest_container | regex_replace("/$", "") }}'
duplicity__backup_dest_container: '{{ ansible_nodename }}'
duplicity__backup_full_if_older_than: '30D'
duplicity__backup_retention_time: '30D'
duplicity__excludes:
- '**/*.git*'
- '**/*.svn*'
- '**/*.temp'
- '**/*.tmp'
- '**/.cache'
- '**/cache'
- '**/log'
duplicity__backup_sources__group_var: []
duplicity__backup_sources__host_var:
- path: '/var/www/html'
divide: false
excludes:
- '/var/www/html/nextcloud/data'
state: 'present'
- path: '/var/www/html/nextcloud/data'
divide: true
state: 'present'
- path: '/backup'
state: 'absent'
duplicity__loglevel: 'notice'
duplicity__logrotate: 7
duplicity__on_calendar: '*-*-* {{ duplicity__on_calendar_hour }}:{{ 45 | random(seed=inventory_hostname) }}'
duplicity__on_calendar_hour: '23'
duplicity__swift_authurl: 'https://swiss-backup02.infomaniak.com/identity/v3'
duplicity__swift_authversion: '3'
duplicity__swift_tenantname: 'sb_project_SBI-MF827483'
duplicity__timer_enabled: true
Troubleshooting¶
If the gpg --import /tmp/public-master-key task fails with gpg: invalid armor header in stderr, make sure your duplicity__gpg_encrypt_master_key_block is correct and has an empty line after the -----BEGIN PGP PUBLIC KEY BLOCK-----.
If duplicity fails with AttributeError: module 'collections' has no attribute 'Mapping' in /opt/python-venv/duplicity/lib64/python3.11/site-packages/oslo_config/cfg.py, manually install 'oslo.config>=9', e.g. /opt/python-venv/duplicity/bin/pip install 'oslo.config>=9'.