PDA

View Full Version : amanda stuck



bahamutta
August 18th, 2011, 05:30 AM
We have a huge backup job (around 1k servers) using amanda 2.5.1p3-2 on dedicated server.
But when I configured the job and ran 'amdump DailySet2' the backup have been started to create backups and after several days its just got stuck silently.

A lot of dumper and gzip proccesses is up and running, but nothing pointing in logs to that the job is actually doing something.
I also didn't find any errors or other suspicious things in amdump logs.

When I'm killing amdump and starting it again, it works fine for several days until the same behavior occurs.

Where to start digging?

Thank you.

paddy
August 18th, 2011, 10:11 AM
Check amreport output to understand where it is stuck. You can check debug logs in /var/log/amanda

btw, 2.5.1p3 is quite old.

bahamutta
August 19th, 2011, 05:51 AM
Check amreport output to understand where it is stuck. You can check debug logs in /var/log/amanda

btw, 2.5.1p3 is quite old.

Yeah I know it's old but it worked quite good for a while.

Thanks I run amreport DailySet2 and it says that there is some kind of error in volumes.
See below:



*** THE DUMPS DID NOT FINISH PROPERLY!

Hostname: backup4
Org : LOCAL
Config : DailySet2
Date : August 18, 2011

*** A TAPE ERROR OCCURRED: [No acceptable volumes found].
There are 7010317M of dumps left in the holding disk.
They will be flushed on the next run.

The next 10 tapes Amanda expects to use are: 10 new tapes.
FAILURE DUMP SUMMARY:
planner: ERROR spare-i NAK: user root from backup4-i.fc2.com is not allowed to execute the service noop: cannot open /home/amanda/.amandahosts: No such file or directory
planner: ERROR Request to analyzer53-i failed: timeout waiting for ACK
planner: ERROR Request to counter2-i failed: timeout waiting for ACK
planner: ERROR Request to kdb-i failed: timeout waiting for ACK
planner: ERROR Request to rslite2-test-i failed: timeout waiting for ACK
planner: ERROR Request to rslite1-i failed: timeout waiting for ACK
planner: ERROR Request to sorryserver-i failed: timeout waiting for ACK
planner: ERROR Request to blog124-i failed: timeout waiting for ACK
planner: ERROR Request to blog134-i failed: timeout waiting for ACK
planner: ERROR Request to systemimage-ubuntu-x64-i failed: timeout waiting for ACK
planner: ERROR Request to hps10-i failed: timeout waiting for ACK
planner: ERROR Request to postfix-i failed: timeout waiting for ACK
planner: ERROR Request to cblog2-i failed: timeout waiting for ACK
planner: ERROR Request to super4-i failed: timeout waiting for ACK
planner: ERROR Request to video4a-i failed: timeout waiting for ACK

and further from report:


canalyzer2-i /usr lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]
super1-i /boot lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]
video3a-i /usr lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]
cart1-i /boot lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]
cbbs1-i /var lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]
blog114-i /boot lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]
cblogdb400-i /var lev 0 FAILED [Skipping: new disk can't be dumped in degraded mode]



STATISTICS:
Total Full Incr. Level:#
-------- -------- -------- --------
Estimate Time (hrs:min) 0:05
Run Time (hrs:min) 0:05
Dump Time (hrs:min) 932:34 0:00 932:34
Output Size (meg) 703734.5 0.0 703734.5
Original Size (meg) 2175068.4 0.0 2175068.4
Avg Compressed Size (%) 32.4 -- 32.4
DLEs Dumped 1268 0 1268 1:1268
Avg Dump Rate (k/s) 214.6 -- 214.6

Tape Time (hrs:min) 0:00 0:00 0:00


Can you advise how to fix the volume?

paddy
August 19th, 2011, 09:13 AM
There are multiple problems.

1. Check whether you labeled your tapes.
2. backup4-i.fc2.com should be added to /home/amanda/.amandahosts on client spare-i
3. Lot of clients cannot be contacted (see timeout errors)