PDA

View Full Version : Does Amanda guarantee level-0 backups exist?



Andrew Rakowski
February 10th, 2007, 11:54 AM
I'm trying to figure out how I ended up in a situation where I don't seem to have level 0 backups for a number of systems, and Amanda is NOT apparently scheduling any. Am I wrong in assuming that if there aren't level-0 backups, Amanda would notice that fact and try to create on at the first possible option?

Of course, I found this out with the first major test of trying to recover a system that got horked badly, and ended up with nothing but level-1 backups (and needless to say, no good backups). In looking via 'amoverview', it appears this is rather common, and I don't know what to do about it, and could use some advice on how to fix this.

For instance, I have:




Date: Sat, 10 Feb 2007 10:45:36 -0800
From: Amanda <amandabackup@...>
To: amanda-watch@...
Subject: amoverview XXXXset1 for 10Feb2007


date 01 01 01 01 01 01 01 02 02 02 02 02 02 02 02 02 02
host disk 25 26 27 28 29 30 31 01 02 03 04 05 06 07 08 09 10

adams / 1 0 1 1 1 1 1 1 1
adams /opt 1 1 1 1 1 1 1 1 1
adams /usr1 0 1 1 1 1 1 1 1 1
adams /usr2 1 1 1 1 1 1 1 1 1



Notice that filesystems /opt and /usr2 seem to be missing something important (level-0 backups). In this case, the / filesystem looks okay.

Here's what and "amadmin find" shows for the / and /opt filesystems.



-sh-3.00$ amadmin XXXXset1 find adams /
Warning: no log files found for tape XXXXset1-15 written 2007-01-25 03:01:01
Scanning /backup0/amanda-hdset1a...
Scanning /backup1/amanda-hdset1...
Warning: no log files found for tape XXXXset1-15 written 2007-01-25 03:01:01
Scanning /backup0/amanda-hdset1a...
Scanning /backup1/amanda-hdset1...

date host disk lv tape or file file part status
2007-02-01 03:01:02 adams / 1 XXXXset1-05 159 1 OK
2007-02-02 03:01:01 adams / 0 XXXXset1-11 126 1 OK
2007-02-03 03:01:01 adams / 1 XXXXset1-18 22 1 OK
2007-02-04 03:01:01 adams / 1 XXXXset1-19 21 1 OK
2007-02-05 03:01:01 adams / 1 XXXXset1-21 15 1 OK
2007-02-06 00:05:01 adams / 1 XXXXset1-23 31 1 OK
2007-02-07 00:05:01 adams / 1 XXXXset1-25 110 1 OK
2007-02-09 00:05:01 adams / 1 XXXXset1-37 124 1 OK
2007-02-10 00:05:02 adams / 1 XXXXset1-39 222 1 OK
-sh-3.00$ amadmin XXXXset1 find adams /opt
Warning: no log files found for tape XXXXset1-15 written 2007-01-25 03:01:01
Scanning /backup0/amanda-hdset1a...
Scanning /backup1/amanda-hdset1...
Warning: no log files found for tape XXXXset1-15 written 2007-01-25 03:01:01
Scanning /backup0/amanda-hdset1a...
Scanning /backup1/amanda-hdset1...

date host disk lv tape or file file part status
2007-02-01 03:01:02 adams /opt 1 XXXXset1-05 161 1 OK
2007-02-02 03:01:01 adams /opt 1 XXXXset1-11 131 1 OK
2007-02-03 03:01:01 adams /opt 1 XXXXset1-18 86 1 OK
2007-02-04 03:01:01 adams /opt 1 XXXXset1-19 64 1 OK
2007-02-05 03:01:01 adams /opt 1 XXXXset1-21 105 1 OK
2007-02-06 00:05:01 adams /opt 1 XXXXset1-23 107 1 OK
2007-02-07 00:05:01 adams /opt 1 XXXXset1-25 124 1 OK
2007-02-09 00:05:01 adams /opt 1 XXXXset1-37 157 1 OK
2007-02-10 00:05:02 adams /opt 1 XXXXset1-39 78 1 OK
-sh-3.00$


I suspect the missing logs are part of the problem, but not sure why that happened. In any case, I have a number of systems (or DLEs) that are in this state, and I'd like to think that Amanda would notice and say "yipes, I'd better get to a level-0 ASAP, since there aren't any".

So, any helpful advice for this novice Amanda administrator. At present, I'm backing up 302 DLEs to about 7.2TB of online disc storage using vtapes, using the chg-multi tape changer glue script. There are perhaps another 100 or so DLEs yet to be added, but these are all mostly less than 30GB of space per DLE being backed up (all via GNUTAR).

I'm running Amanda v2.5.1p2 with the "exclude" and "security-util.c" patches applied on a Redhat AW4 Intel 4 CPU system with 4GB of RAM and around 8TB of disc space. I see that 2.5.1p3 has been released, but haven't installed it quite yet ("the devil you know" issue - and waiting for any bug shakeout before I upgrade the server.)

Best regards,

-Andrew

martineau
February 12th, 2007, 03:50 AM
Missing log file is a big problem, amanda never erase them, it move them to the oldlog directory, could you check if you have the log for the XXXXset1-15 tape in it.
grep XXXXset1-15 oldlog/*

Could you scan the directory to find on which date/tape the level 0 of /opt and /usr2 was done?

Do you have a system crontab job that can erase amanda log files? amanda log files are important, they should never be erased.

Andrew Rakowski
February 12th, 2007, 10:02 AM
Missing log file is a big problem, amanda never erase them, it move them to the oldlog directory, could you check if you have the log for the XXXXset1-15 tape in it.
grep XXXXset1-15 oldlog/*


There are no files in the log/oldlog directory at all. That's probably (guessing here) because I'm using "dumpcycle 4 weeks" and a "runspercycle 0", which should indicate a run every day, if I understand the parameter setting. I've only set up this configuration on the 19th of January, so it hasn't been 4 weeks yet.



Could you scan the directory to find on which date/tape the level 0 of /opt and /usr2 was done?

The only messages that I see for tape XXXXset1-15 are things like:

log/amdump.44:Warning: no log files found for tape XXXXset1-15 written 2007-01-25 03:01:01

and

log/log.20070212005901.0:WARNING planner Last full dump of spokane:/files0 on tape XXXXset1-15 overwritten in 3 runs.

In looking at the size of this virtual tape, I see it's only 348KB in size for the directory, with date/timestamps of Feb 02, so I'm going to have to guess it got relabeled (we were running a bunch of failed backups while testing AIX, and it's possible that some virtual tapes got relabeled by accident in that series of tests.)



Do you have a system crontab job that can erase amanda log files? amanda log files are important, they should never be erased.

There are no cron jobs to clean up any files on this system. I only have two entries in the crontab for user amandabackup, and those just fire up the two Amanda configurations. In fact, if there are supposed to be any additional "Amanda maintenance" type cron jobs, I'm unaware of them. The only thing I do besides the amdump runs is to add a "amoverview XXXXset1 -hostwidth 12" command that pipes into Mail to give me an update, as part of the crontab command line (so it runs when the amdump is done).

I guess at this point, the real question is, how do I get Amanda to do a new series of level-0 backups for those that appear to be missing (lets assume that tapes got relabeled, as timestamps seem to indicate that)? I need to be able to get things back to a "everything is properly backed up" state as soon as possible.

Do I need to do "amadmin XXXXset1 force blah" on all the systems that show as not having level-0 backups? When I do a:

amoverview XXXXset1 -hostwidth 12 -togo0 |grep " 0 " |wc -l

it appears I have 80 DLEs that show as scheduled for level-0s, but I don't see them getting level-0 backups, so I'm concerned there's something still "broken" here.

-Andrew

ps: I feel like such a leach on the community here, but I'll hopefully get my backups humming along eventually and will be able to use these experiences to help others stepping into the "virtual tape backups" world. -ar

martineau
February 12th, 2007, 11:33 AM
If you destroy a tape or relabel it, amanda can't know it. It's your responsibility to tell it to amanda with the amrmtape command.

You should take the following message seriously:
WARNING planner Last full dump of spokane:/files0 on tape XXXXset1-15 overwritten in 3 runs.
It's your responsanility to feed enough tape to amanda to let it respect your schedule.

If you don't have the XXXset1-15, you should run 'amrmtape XXXXset1 XXXXset1-15'.
It should enable amanda to do full dump of the 2 DLEs.

Andrew Rakowski
February 12th, 2007, 03:15 PM
If you destroy a tape or relabel it, amanda can't know it. It's your responsibility to tell it to amanda with the amrmtape command.

You should take the following message seriously:
WARNING planner Last full dump of spokane:/files0 on tape XXXXset1-15 overwritten in 3 runs.
It's your responsanility to feed enough tape to amanda to let it respect your schedule.

If you don't have the XXXset1-15, you should run 'amrmtape XXXXset1 XXXXset1-15'.
It should enable amanda to do full dump of the 2 DLEs.

Hmmm, I assumed when I did the "amlabel -f" on an already labeled tape, it would know that the contents of that tape had gotten destroyed, so that could very well be the problem we have had. I will be sure to run "amrmtape" any time I relabel a tape. Since amrmtape says "remove label XXXXset1-15", I'm guessing I need to then relabel that "tape" as well, right?

Yipes - I was further assuming that those WARNING messages were just indicating that Amanda was telling me it was going to be needing to do another level-0 since it would be overwriting the last full dump of that system.

So, whenever I see the WARNING messages about impending overwrites, I should be assuming that I need more tapes? I'm currently using 80 virtual tapes of 50GB (plus 1MB - 51201MB) in an 80-slot virtual tape changer. The tape-related parameters I have set are:

....
dumpcycle 4 weeks # the # of days in normal dump cycle (fulls this often at least)
runspercycle 0 # the number of amdump runs in dumpcycle days
tapecycle 80 tapes # the number of tapes in rotation
runtapes 10 # maximum number of tapes to be used in a single run of amdump
....

I originally had only 40 tapes in the rotation, but on days when I added a lot of systems, Amanda would be writing to 3 or 4 tapes in a backup session, so I figured I'd need to boost that number and created additional tapes. Once I get a better handle on the average size of daily backups, I was going to adjust the tape "size" to avoid filling 1.1 or 1.2 tapes in a day.

Am I at all correct in thinking that tapes will somehow "expire" and become reusable automatically, or do I need to manually remove and relabel all old tapes before Amanda will reuse them?

I'm beginning to think the theory I was operating under (have a large enough virtual jukebox, and Amanda will automatically cycle through it, keeping backups and reusing no longer needed tapes based - perhaps - on the 'dumpcycle' length) is another misunderstanding on my part.

If there's a "Complete Idiot's Guide to Administering Amanda Backups", I might be a good candidate to read it. To help understand my confusion, my previous experience with backup "systems" was using Legato Networker, so that's the mental model I was attempting to recreate with Amanda.

-Andrew

martineau
February 13th, 2007, 06:32 AM
dumpcycle 4 weeks # the # of days in normal dump cycle (fulls this often at least)
runspercycle 0 # the number of amdump runs in dumpcycle days
tapecycle 80 tapes # the number of tapes in rotation
runtapes 10 # maximum number of tapes to be used in a single run of amdump
....

Am I at all correct in thinking that tapes will somehow "expire" and become reusable automatically, or do I need to manually remove and relabel all old tapes before Amanda will reuse them?

You should never relabel a tape, amanda will reuse it after 'tapecycle' tape are used.


I'm beginning to think the theory I was operating under (have a large enough virtual jukebox, and Amanda will automatically cycle through it, keeping backups and reusing no longer needed tapes based - perhaps - on the 'dumpcycle' length) is another misunderstanding on my part.

It will do it, but it's based on the tapecycle.




-Andrew