pyhardlinkbackup 0.7.0

PyHardLinkBackup

Hardlink/Deduplication Backups with Python.

• Backups should be saved as normal files in filesystem:

  • accessible without any extra software or extra meta files

  • non-proprietary format

• Create backups with versioning

  • every backup run creates a complete filesystem snapshot tree

  • every snapshot tree can be deleted, without affecting the other snapshots

• Deduplication with hardlinks:

  • Store only changed files, all other via hardlinks

  • find duplicate files everywhere (even if renamed or moved files)

• useable under Windows and Linux

current state:

• python 3.4 or newer only

• Beta state

Please, try, fork and contribute! ;)

Example

$ phlb backup ~/my/important/documents
...start backup, some time later...
$ phlb backup ~/my/important/documents
...

This will create deduplication backups like this:

~/PyHardLinkBackups
  └── documents
      ├── 2016-01-07-085247
      │   ├── phlb_config.ini
      │   ├── spreadsheet.ods
      │   ├── brief.odt
      │   └── important_files.ext
      └── 2016-01-07-102310
          ├── phlb_config.ini
          ├── spreadsheet.ods
          ├── brief.odt
          └── important_files.ext

Install

~ $ cd PyHardLinkBackup
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ pip install scndir

start backup run

To start a backup run, use this helper script:

• Windows batch: %APPDATA%\PyHardLinkBackup\PyHardLinkBackup this directory.cmd

• Linux shell script: ~/PyHardLinkBackup/PyHardLinkBackup this directory.sh

Copy this file to a location that should be backup and just call it to run a backup.

Verify a existing backup

$ cd PyHardLinkBackup/
~/PyHardLinkBackup $ source bin/activate

(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb verify --fast ~/PyHardLinkBackups/documents/2016-01-07-102310

With –fast the file content will not be checkt. If not given: The hash from the file content will be calculated and compared. So every file must be complete read from filesystem, so it’s takes some time.

A verify run do:

• Exist all file in backup?

• Compare file size

• Compare hash from hash-file

• Compare file modify timestamp

• Calculate hash from file content and compare (Will be skipped if –fast used)

configuration

phlb will used a configuration file named: PyHardLinkBackup.ini

Search order is:

1. current directory down to root

2. user directory

e.g.: Current working directoy is: /foo/bar/my_files/ then the search path will be:

• /foo/bar/my_files/PyHardLinkBackup.ini

• /foo/bar/PyHardLinkBackup.ini

• /foo/PyHardLinkBackup.ini

• /PyHardLinkBackup.ini

• /PyHardLinkBackup.ini The user home directory under Windows/Linix

(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb config

# Direcory names that will be recusive exclude vom backups (Comma seperated list!)
SKIP_DIRS= __pycache__, temp

# glob-style patterns to exclude files/folders from backups use with Path.match() (Comma seperated list!)
SKIP_PATTERNS= *.pyc, *.tmp, *.cache

upgrate PyHardLinkBackup

To upgrate to a new version just start these helper script:

• Windows: phlb_upgrade_PyHardLinkBackup.cmd

• Linux: phlb_upgrade_PyHardLinkBackup.sh

some notes

run unittests

Just start: phlb_run_tests.cmd / phlb_run_tests.sh or do this:

$ cd PyHardLinkBackup/
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ manage test

the cli

$ cd PyHardLinkBackup/
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb --help
Usage: phlb [OPTIONS] COMMAND [ARGS]...

  PyHardLinkBackup

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  backup  Start a Backup run
  config  Create/edit .ini config file
  helper  link helper files to given path
  verify  Verify a existing backup

History

• 03.02.2016 - v0.7.0 - compare v0.6.4…v0.7.0

  • New: verify a existing backup

  • IMPORTANT: run database migration is needed!

• 01.02.2016 - v0.6.4 - compare v0.6.2…v0.6.4

  • Windows: Bugfix temp rename error, because of the Windows API limitation, see: #13

  • Linux: Bugfix scanner if symlink is broken

  • Display local variables on low level errors

• 29.01.2016 - v0.6.3 - compare v0.6.2…v0.6.3

  • Less verbose and better information about SKIP_DIRS/SKIP_PATTERNS hits

• 28.01.2016 - v0.6.2 - compare v0.6.1…v0.6.2

  • Handle unexpected errors and continue backup with the next file

  • Better handle interrupt key during execution

• 28.01.2016 - v0.6.1 - compare v0.6.0…v0.6.1

  • Bugfix #13 by using a better temp rename routine

• 28.01.2016 - v0.6.0 - compare v0.5.1…v0.6.0

  • New: faster backup by compare mtime/size only if old backup files exists

• 27.01.2016 - v0.5.1 - compare v0.5.0…v0.5.1

  • IMPORTANT: run database migration is needed!

  • New .ini setting: LANGUAGE_CODE for change translation

  • mark if backup was finished compled

  • Display information of last backup run

  • Add more information into summary file

• 27.01.2016 - v0.5.0 - compare v0.4.2…v0.5.0

  • refactory source tree scan. Split in two passed.

  • CHANGE SKIP_FILES in .ini config to: SKIP_PATTERNS

  • Backup from newest files to oldest files.

  • Fix #10:

    • New –name cli option (optional) to force a backup name.

    • Display error message if backup name can be found (e.g.: backup a root folder)

• 22.01.2016 - v0.4.2 - compare v0.4.1…v0.4.2

  • work-a-round for junction under windows, see also: https://www.python-forum.de/viewtopic.php?f=1&t=37725&p=290429#p290428 (de)

  • Bugfix in windows batches: go into work dir.

  • print some more status information in between.

• 22.01.2016 - v0.4.1 - compare v0.4.0…v0.4.1

  • Skip files that can’t be read/write. (and try to backup the remaining files)

• 21.01.2016 - v0.4.0 - compare v0.3.1…v0.4.0

  • Search for PyHardLinkBackup.ini file in every parent directory from the current working dir

  • increase default chunk size to 20MB

  • save summary and log file for every backup run

• 15.01.2016 - v0.3.1 - compare v0.3.0…v0.3.1

  • fix unittest run under windows

• 15.01.2016 - v0.3.0 - compare v0.2.0…v0.3.0

  • database migration needed

  • Add ‘no_link_source’ to database (e.g. Skip source, if 1024 links created under windows)

• 14.01.2016 - v0.2.0 - compare v0.1.8…v0.2.0

  • good unittests coverage that covers the backup process

• 08.01.2016 - v0.1.8 - compare v0.1.0alpha0…v0.1.8

  • install and runable under Windows

• 06.01.2016 - v0.1.0alpha0 - d42a5c5

  • first Release on PyPi

• 29.12.2015 - commit 2ce43

  • commit ‘Proof of concept’

Links

• https://pypi-hypernode.com/pypi/PyHardlinkBackup/

• https://www.python-forum.de/viewtopic.php?f=6&t=37723 (de)