rsync like time machine
I am using rsync very similar to your use case, pulling together backups from remote systems via rsync onto one backup server with multiple hard drives.
I hacked together a script, which acts very similar to Apple's TimeMachine, so it is run via cron every hour and keeps a day of hourly backups, a month of dailies etc.
The script is not pretty, but it worked stable for me for the last years. You could find it here: https://github.com/michaelrommel/rsyncmachine/tree/master
Over the Christmas season I have to update the backup server, so most probably I'll have to adapt the script in case rsync's output or the perl modules changed over the last year.
All the best,
RE: rsync like time machine - Added by jcw about 3 years ago
I am ok with just the last week's set of backup in case of the JeeLabs servers, but for longer-term archival backups your script looks like an excellent solution. I wouldn't expect
rsync to have changed much, it's a very stable piece of software which has been around for some 15 years now.
The one thing which Time Machine does which is still superior to rsync IMO, is that it works with a change list maintained by the OS. So when it has to do its hourly task, it only needs to go through the list of file + directory changes, instead of the full list. For very large backups with relatively few changes, this makes it substantially faster than rsync, which has to traverse the entire tree to find changes.
But this requires OS support, and the directory linking of HFS+ is also not supported on most Linux file systems.
In my setup, rsync is fine - a few minutes of dir/file scanning once a day is absolutely no issue at all.
For an even more powerful file system, there's ZFS. It can do instant snapshots, perfect for backups. Even de-duplication, but this requires a lot of RAM.
RE: rsync like time machine - Added by jeroenb about 3 years ago
I have a small server with ZFS, using snapshots (quaterly, hourly, daily and monthly). Besides snapshots I use backup-boot environments. So if there is something wrong after an install or update, it is possible to boot from a good and working system. With ZFS send and receive it is possible to keep two systems in sync.
(sorry for the shameless plug)
RE: rsync like time machine - Added by michaelrommel about 3 years ago
That sounds very interesting. I learned about ZFS roughly in 2007 and was very fascinated with it. At the time it was only available on Solaris and when Sun was bought by Oracle all implementations were only alpha or beta and not recommended for production use. Honestly I lost track of it.
Is it now stable for normal use and would you recommend it? There are a couple of new filesystems in the mainstream linux kernel but recently I just went with the default settings of Ubuntu. For more advanced tinkering I wanted to try out ceph, but for serious use, you would need a cluster of three machines, which is not really sensible for home use.
Do you have a pointer to this "zfs send and receive" topic? That sounds also quite interesting to read more about.
Thanks for the interesting info - guess I have to catch up with some ZFS reading...