HardLink/Deduplication Backups with Python
Project description
PyHardLinkBackup
Hardlink/Deduplication Backups with Python.
Backups should be saved as normal files in filesystem:
accessible without any extra software or extra meta files
non-proprietary format
Create backups with versioning
every backup run creates a complete filesystem snapshot tree
every snapshot tree can be deleted, without affecting the other snapshots
Deduplication with hardlinks:
Store only changed files, all other via hardlinks
find duplicate files everywhere (even if renamed or moved files)
useable under Windows and Linux
current state:
python 3.4 or newer only
Beta state
Please, try, fork and contribute! ;)
Example
$ phlb backup ~/my/important/documents ...start backup, some time later... $ phlb backup ~/my/important/documents ...
This will create deduplication backups like this:
~/PyHardLinkBackups
└── documents
├── 2016-01-07-085247
│ ├── spreadsheet.ods
│ ├── brief.odt
│ └── important_files.ext
└── 2016-01-07-102310
├── spreadsheet.ods
├── brief.odt
└── important_files.ext
Try out:
on Windows:
install Python 3: https://www.python.org/downloads/
Download the file boot_pyhardlinkbackup.cmd
run boot_pyhardlinkbackup.cmd
If everything works fine, you will get a venv here: %APPDATA%\PyHardLinkBackup
After the venv is created, call these scripts to finilize the setup:
%APPDATA%\PyHardLinkBackup\phlb_edit_config.cmd - Created a config .ini file
%APPDATA%\PyHardLinkBackup\phlb_migrate_database.cmd - Create Database tables
To upgrade PyHardLinkBackup, call:
%APPDATA%\PyHardLinkBackup\phlb_upgrade_PyHardLinkBackup.cmd
To start the django webserver, call:
%APPDATA%\PyHardLinkBackup\phlb_run_django_webserver.cmd
on Linux:
Download the file boot_pyhardlinkbackup.sh
call boot_pyhardlinkbackup.sh
Note: If you not use python 3.5+, then you must install ‘scandir’, e.g.:
~ $ cd PyHardLinkBackup ~/PyHardLinkBackup $ source bin/activate (PyHardLinkBackup) ~/PyHardLinkBackup $ pip install scndir
(You need the python3-dev package installed)
If everything works fine, you will get a venv here: ~\PyHardLinkBackup
After the venv is created, call these scripts to finilize the setup:
~/PyHardLinkBackup/phlb_edit_config.sh - Created a config .ini file
~/PyHardLinkBackup/phlb_migrate_database.sh - Create Database tables
To upgrade PyHardLinkBackup, call:
~/PyHardLinkBackup/phlb_upgrade_PyHardLinkBackup.sh
To start the django webserver, call:
~/PyHardLinkBackup/phlb_run_django_webserver.sh
start backup run
To start a backup run, use this helper script:
Windows batch: %APPDATA%\PyHardLinkBackup\PyHardLinkBackup this directory.cmd
Linux shell script: ~/PyHardLinkBackup/PyHardLinkBackup this directory.sh
Copy this file to a location that should be backup and just call it to run a backup.
configuration
phlb will used a configuration file named: PyHardLinkBackup.ini
Search order is:
current directory down to root
user directory
e.g.: Current working directoy is: /foo/bar/my_files/ then the search path will be:
/foo/bar/my_files/PyHardLinkBackup.ini
/foo/bar/PyHardLinkBackup.ini
/foo/PyHardLinkBackup.ini
/PyHardLinkBackup.ini
/PyHardLinkBackup.ini The user home directory under Windows/Linix
Create / edit default .ini
You can just open the editor with the user directory .ini file with:
(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb config
The defaults are stored here: /phlb/config_defaults.ini
Exclude files/folders from backup:
There are two ways to exclude files/folders from your backup. Use the follow settings in your PyHardLinkBackup.ini
# Direcory names that will be recusive exclude vom backups (Comma seperated list!) SKIP_DIRS= __pycache__, temp # glob-style patterns to exclude files/folders from backups use with Path.match() (Comma seperated list!) SKIP_PATTERNS= *.pyc, *.tmp, *.cache
The filesystem scan is divided into two steps: 1. Just can the filesystem tree 2. Filter and load meta data for every directory item
The SKIP_DIRS is used in the first step. The SKIP_PATTERNS is used the the second step.
run unittests
$ cd PyHardLinkBackup/ ~/PyHardLinkBackup $ source bin/activate (PyHardLinkBackup) ~/PyHardLinkBackup $ manage test
some notes
What is ‘phlb’ ?!?
the phlb executable is the similar to django manage.py, but it always used the PyHardLinkBackup settings.
Why in hell do you use django?!?
Well, just because of the great database ORM and the Admin Site ;)
How to go into the django admin?
$ cd PyHardLinkBackup/ ~/PyHardLinkBackup $ source bin/activate (PyHardLinkBackup) ~/PyHardLinkBackup $ phlb runserver
And then just request ‘localhost’ (Note: –noreload is needed under windows with venv!)
Windows Development
Some notes about to setup a development under windows, please look at: /dev/WindowsDevelopment.creole
alternative solutions
Attic: https://attic-backup.org/ (Not working under Windows, own backup archive format)
msbackup: https://pypi-hypernode.com/pypi/msbackup/ (used tar for backup archives)
Duplicity: http://duplicity.nongnu.org/ (No Windows support, tar archive format)
Burp: http://burp.grke.org/ (Client/Server solution)
History
28.01.2016 - v0.6.2 - compare v0.6.1…v0.6.2
Handle unexpected errors and continue backup with the next file
Better handle interrupt key during execution
28.01.2016 - v0.6.1 - compare v0.6.0…v0.6.1
Bugfix #13 by using a better temp rename routine
28.01.2016 - v0.6.0 - compare v0.5.1…v0.6.0
New: faster backup by compare mtime/size only if old backup files exists
27.01.2016 - v0.5.1 - compare v0.5.0…v0.5.1
IMPORTANT: run database migration is needed!
New .ini setting: LANGUAGE_CODE for change translation
mark if backup was finished compled
Display information of last backup run
Add more information into summary file
27.01.2016 - v0.5.0 - compare v0.4.2…v0.5.0
refactory source tree scan. Split in two passed.
CHANGE SKIP_FILES in .ini config to: SKIP_PATTERNS
Backup from newest files to oldest files.
Fix #10:
New –name cli option (optional) to force a backup name.
Display error message if backup name can be found (e.g.: backup a root folder)
22.01.2016 - v0.4.2 - compare v0.4.1…v0.4.2
work-a-round for junction under windows, see also: https://www.python-forum.de/viewtopic.php?f=1&t=37725&p=290429#p290428 (de)
Bugfix in windows batches: go into work dir.
print some more status information in between.
22.01.2016 - v0.4.1 - compare v0.4.0…v0.4.1
Skip files that can’t be read/write. (and try to backup the remaining files)
21.01.2016 - v0.4.0 - compare v0.3.1…v0.4.0
Search for PyHardLinkBackup.ini file in every parent directory from the current working dir
increase default chunk size to 20MB
save summary and log file for every backup run
15.01.2016 - v0.3.1 - compare v0.3.0…v0.3.1
fix unittest run under windows
15.01.2016 - v0.3.0 - compare v0.2.0…v0.3.0
database migration needed
Add ‘no_link_source’ to database (e.g. Skip source, if 1024 links created under windows)
14.01.2016 - v0.2.0 - compare v0.1.8…v0.2.0
good unittests coverage that covers the backup process
08.01.2016 - v0.1.8 - compare v0.1.0alpha0…v0.1.8
install and runable under Windows
06.01.2016 - v0.1.0alpha0 - d42a5c5
first Release on PyPi
29.12.2015 - commit 2ce43
commit ‘Proof of concept’
Links
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for PyHardLinkBackup-0.6.2-py3.4.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fc2c016d062a67d9e21ef5ddb7d84a8f62771f0638c13649a8ff13bb9bd52465 |
|
| MD5 | 54cd42460151e193904f2b33a48395d4 |
|
| BLAKE2b-256 | d375fbd8cad0f1758405db52fe3375c5d9b6d7054965c09a6d57217e76aa5e0d |
Hashes for PyHardLinkBackup-0.6.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7378b2d6109eb330f456830bd9666a82dcca4dfc4ef4771ad1d8d9fe48da5a53 |
|
| MD5 | 00774ae5c1431573b0fde49453dc6a3c |
|
| BLAKE2b-256 | 2f70114d4cadb5f9d0ac14d5d5855d0f4384f4db78c72eb16abb481175a9124b |
