PDA

View Full Version : Incremental backup using zfs send



glowkrantz
December 19th, 2008, 01:06 AM
So now we have working zfs send on snapshots i.e. level 0 backups. On my test installation that takes 5 LTO 2 tapes and 18 hours.

Any discussions on how to handle incremental backups on zfs?
I have been looking at ways to estimate and have come to the following:
Level 0: good approximation by filesystem referenced count, maybe a little bit low but the error is small. It does not warrant doing a dummy snapshot for level 0 estimations only.
Level 1+: No way to estimate, only way is running
zfs send -i pool/fs@0 pool/fs@1 | wc -c

Any thoughts?

How accurate must the estimate be?

Does amanda do 0 -> 1 -> 2 -> 1 or only 0 -> 1 -> 2 -> 0? I only see the later when running dump. If the later sequence is the only one, it means that only the latest snapshot needs to be kept between backups.

Is level 9 sufficient? On some large filesystems with streaming media, I consistently was doing level 5 to 8 backups when they were UFS and used dump.

Is the snapshot name fixed in the current code or can it be changed to something like pool/fs@amanda.diskname.level i.e. system/usr/local@amanda._usr_local.0, system/usr/local@amanda._usr_local.1 etc?

Unless there is anything that I can test already, I will try and experiment over the weekend and see what I come up with.

/glz

glowkrantz
December 25th, 2008, 11:44 AM
Attached are what's needed to test incremental zfs send backups. It's been tested on

FreeBSD 7.1-PRERELEASE: Sat Nov 29 03:33:04 CET 2008
ZFS filesystem version 6
ZFS storage pool version 6
ZFS on-disk version 1

and FreeBSD 8.0-CURRENT: Wed Dec 24 13:07:11 CET 2008
ZFS filesystem version 13
ZFS storage pool version 13
ZFS on-disk version 3

> unzip -l amanda-zfs-send-incremental.zip
Archive: amanda-zfs-send-incremental.zip
Length Date Time Name
-------- ---- ---- ----
7713 12-25-08 21:08 amanda-2.6.1b2-20081222.tar.gz
437 12-25-08 21:03 amwc.sh
4257 12-25-08 20:51 patch-application-src::amzfs-sendrecv.pl
4710 12-25-08 20:54 patch-perl::Amanda::Application::Zfs.pm
-------- -------
17117 4 files

For *BSD ports users:
amanda-2.6.1b2-20081222.tar.gz: amanda-devel port of comunity build.

For anyone building from scratch or just want to update an installation:
amwc.sh: hack for reading the actual sendsize, MUST BE PLACES IN /usr/local/bin/amwc (yes, I know but I hope it's temporary).
patch-perl::Amanda::Application::Zfs.pm: patch to support versioned snapshots and allowing overlap between amcheck and amdump.
patch-application-src::amzfs-sendrecv.pl: patch to handle incemental backups to level 9.

All comments etc. welcome.

/glz

glowkrantz
December 26th, 2008, 10:47 PM
One error in the port - missed a moved manpage.

/glz

glowkrantz
January 4th, 2009, 05:57 PM
Some more info:
Dumptype definitions to use with these patches:

# My global
define dumptype admin {
comment "Global definitions for all administered dumps"
# This is quite useful for setting global parameters, so you don't have
# to type them everywhere. All dumptype definitions in this sample file
# do include these definitions, either directly or indirectly.
# There's nothing special about the name `global'; if you create any
# dumptype that does not contain the word `global' or the name of any
# other dumptype that contains it, these definitions won't apply.
# Note that these definitions may be overridden in other
# dumptypes, if the redefinitions appear *after* the `global'
# dumptype name.
# You may want to use this for globally enabling or disabling
# indexing, recording, etc. Some examples:
index yes
record yes
}

#
# Amanda ZFS dump using snapshot and zfs send
#
define application-tool amzfs_sendrecv {
comment "amzfs-sendrecv"
plugin "amzfs-sendrecv"
property "DF-PATH" "/bin/df"
property "ZFS-PATH" "/sbin/zfs"
# FreeBSD 7, delegation works in CURRENT
property "PFEXEC-PATH" "/usr/local/bin/sudo"
property "PFEXEC" "YES"
}

define dumptype user-zfs-sendrecv {
admin
index no
program "APPLICATION"
application "amzfs_sendrecv"
maxdumps 2
}

define dumptype user-zfs-sendrecv-split {
user-zfs-sendrecv
tape_splitsize 1 Gb
}

The use of sudo for operator access in FreeBSD 7, before delegation, requires the following line in the suoders file:

operator ALL=(root) NOPASSWD: /sbin/zfs

/glz

martineau
January 13th, 2009, 07:21 AM
Thanks for the patch, I started to look at it and I have 1 questions?

Why zfs_purge_snapshot is called before the backup is tried?
I think it should be called only if the backup succeed and just before zfs_rename_snapshot.

If amanda try a level 0 and it failed, it can try a higher level on the next run.

nick.smith@techop.ch
January 15th, 2009, 01:49 AM
Hi all,

I think the current proposal will have significant performance issues with the piping the output through wc (espcially during the estimate phase!).

Could you not use :

$cmd = "$self->{pfexec_cmd} $self->{zfs_path} get -Hp -o value used $self->{filesystem}\@$self->{snapshot}"

for snapshots with level > 0?

Regards,

Nick

martineau
January 15th, 2009, 05:12 AM
Hi all,

I think the current proposal will have significant performance issues with the piping the output through wc (especially during the estimate phase!).

Could you not use :

$cmd = "$self->{pfexec_cmd} $self->{zfs_path} get -Hp -o value used $self->{filesystem}\@$self->{snapshot}"


This is always 0 for a newly created snapshot.
Let me know if you find a way to compute estimate.

nick.smith@techop.ch
January 16th, 2009, 12:05 AM
This is always 0 for a newly created snapshot.
Let me know if you find a way to compute estimate.

Apologies - my last post was somewhat erroneous!

How about iterating the through the snapshots and deducting one 'reference' value from the 'level-1' snapshot from the 'reference' value from the 'level' snapshot??

I've attached a script of my testing.

Any comments welcome!

Regards,

Nick

martineau
January 16th, 2009, 03:08 AM
You didn't try to remove files.

If you remove /rpool/test/sol-nv-b101-x86-dvd.iso.1 before creating level 3 snaphost, then the reference for level 3 snapshot will be 7.37G, you wil get:
# zfs list -r rpool/test
NAME USED AVAIL REFER MOUNTPOINT
rpool/test 10.4G 305G 10.4G /rpool/test
rpool/test@0 17K - 2.15G -
rpool/test@1 17K - 5.22G -
rpool/test@2 17K - 7.37G -
rpool/test@3 0 - 7.37G -

but the backup will be 2.15G, how do you compute it?

glowkrantz
January 16th, 2009, 04:02 AM
Which is why I followed the tradition of GNU tar, where the size is calculated by a dummy backup.

And looking in the sendsize debug file, I see that it's also done for dump, even on level 0.

There are times when we just don't know unless we do it.

An alternative could be to do server side estimates, which I think use the statistics from actual dump runs to estimate.

/glz

glowkrantz
January 16th, 2009, 04:24 AM
Hi martineau,


Thanks for the patch, I started to look at it and I have 1 questions?

Why zfs_purge_snapshot is called before the backup is tried?
I think it should be called only if the backup succeed and just before zfs_rename_snapshot.

If amanda try a level 0 and it failed, it can try a higher level on the next run.

I think it was because I managed to get in a failure mode where I had a snapshot that had not resulted in a valid backup and the next time I tested, amanda choose the next higher level, as in I have valid 0, 1 and 3 but no valid 2 backup. So by doing it this way, I made sure the next highest backup level was the failed one, in the example above, a level 2 backup.

Hope this makes sense ;-)

There is still a failure mode when there will be an lost current snapshot that is not removed if the client crashes before amanda finishes and we may need to remove that in the zfs_purge_snapshot as we otherwise will have one failed backup before the client is in sync again.

/glz

glowkrantz
January 16th, 2009, 04:37 AM
Hi Nick,


Hi all,

I think the current proposal will have significant performance issues with the piping the output through wc (espcially during the estimate phase!).

Could you not use :

$cmd = "$self->{pfexec_cmd} $self->{zfs_path} get -Hp -o value used $self->{filesystem}\@$self->{snapshot}"

for snapshots with level > 0?

Regards,

Nick

I understand and am searching all over the place for other ways t do it.

This method is anyhow faster than with snapshot/gtar, especially on compressed filesystems. We have a few of them, handling Oracle and PostgreSQL backups and logs and with 2.2 times compression ratio on 30 to 40G referenced, it's almost a factor of 3.

I have no actual numbers for my test system just now but I was able to go back to the default etime when I switched from snapshot/gtar to this setup.

/glz

nick.smith@techop.ch
January 16th, 2009, 05:53 AM
You didn't try to remove files.

If you remove /rpool/test/sol-nv-b101-x86-dvd.iso.1 before creating level 3 snaphost, then the reference for level 3 snapshot will be 7.37G, you wil get:
# zfs list -r rpool/test
NAME USED AVAIL REFER MOUNTPOINT
rpool/test 10.4G 305G 10.4G /rpool/test
rpool/test@0 17K - 2.15G -
rpool/test@1 17K - 5.22G -
rpool/test@2 17K - 7.37G -
rpool/test@3 0 - 7.37G -

but the backup will be 2.15G, how do you compute it?

Ah well yes, very good point!

I'll play around a bit more and see if examining both the used and refer values for all the snapshots we can get a solution.

Not directly related but of interest: I asked the zfs-discuss mailing list about what output 'zfs send -v snapshot-name' should produce and got the following :



Nick,

Specifying -v to zfs send doesn't result in much extra information, and only in certain cases. For example, if you do an incremental send, you'll get this piece of extra output:

# zfs send -v -I tank@snap1 tank@snap2 >/tmp/x
sending from @snap1 to tank@snap2
#

zfs recv -v is a bit more chatty, but again, only in certain cases.

Output from zfs send -v goes to stderr; output from zfs recv -v goes to stdout.

-Chris


Regards,

Nick

martineau
January 18th, 2009, 04:23 AM
Hi martineau,

I think it was because I managed to get in a failure mode where I had a snapshot that had not resulted in a valid backup and the next time I tested, amanda choose the next higher level, as in I have valid 0, 1 and 3 but no valid 2 backup. So by doing it this way, I made sure the next highest backup level was the failed one, in the example above, a level 2 backup.

Hope this makes sense ;-)

There is still a failure mode when there will be an lost current snapshot that is not removed if the client crashes before amanda finishes and we may need to remove that in the zfs_purge_snapshot as we otherwise will have one failed backup before the client is in sync again.

/glz

amanda should not increase the level if the backup was invalid.
Do you remember what was invalid?
Did it was a dump to holding disk or direct to tape?
Do the backup was successful but the server was not able to save it to holding disk or tape?
Or it was a zfs error and zfs didn't sent the error to the server?
Or something else?

glowkrantz
January 18th, 2009, 05:19 PM
I think you are correct, this is not a problem. My failed sequence was probably due to a bug in the original script I had and after getting that right, I never removed the purge phase. The way it's done now will force Amanda to go one level lower than needed after a failed backup. The purge was originally there to remove any stray current snapshots but got changed while I was testing the scripts to do the delete of the expected snapshot.

I have created a new version working the expected way, I will submit it as soon as it has a few backups under it's belt.

/glz

martineau
January 19th, 2009, 04:33 AM
I committed your previous patch with many fix.
If you want to send me a new patch, send it relative to the current source tree.

glowkrantz
January 26th, 2009, 09:51 PM
While testing 20090122, I have found one more thing and that's the level 0 estimate on a compressed file system. We need to multiply with the compression ration to get closer to the actual backup size.

Attached patch adds this to the estimate call.

Otherwise, this seems to be working just fine.

/glz