localhost backups intermittently failing, localhost NAK: timeout on reply pipe

[martineau]

Remove the first runtar argument
/usr/libexec/amanda/runtar hatteras /bin/tar --create --file /dev/null --numeric-owner --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/localhost__0.new --sparse --ignore-failed-read --totals .

[an_admin]

command completed without error... so why is amdump failing?

/usr/libexec/amanda/runtar hatteras /bin/tar --create --file /dev/null --numeric-owner --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/localhost__0.new --sparse --ignore-failed-read --totals .

ls -la /var/lib/amanda/gnutar-lists/localhost__0.new 
-rw-rw-r-- 1 amandabackup amandabackup 24677778 Mar 13 16:26 /var/lib/amanda/gnutar-lists/localhost__0.new

please advise.

[an_admin]

bump for inactivity.

Could really use some input on this failing tar command.

please advise.

[martineau]

What was the output?
What was the exit status (echo $?)
Try to run only the tar command as root

[an_admin]

as posted above, when logged-in as user amandabackup, executing the runtar command completes without error:

/usr/libexec/amanda/runtar hatteras /bin/tar --create --file /dev/null --numeric-owner --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/localhost__1.new --sparse --ignore-failed-read --totals .
Total bytes written: 13987778560 (14GiB, 1.3GiB/s)

-rw------- 1 root         amandabackup 24678978 Mar 20 13:15 localhost__1.new

running the tar command itself as root:

/bin/tar --create --file /dev/null --numeric-owner --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/localhost__2.new --sparse --ignore-failed-read --totals .
Total bytes written: 13987778560 (14GiB, 1.5GiB/s)

-rw-r--r-- 1 root         root         24678978 Mar 20 13:18 localhost__2.new

The exit status is 0

echo $?
0

Appreciate your input, please advise.

JS

[an_admin]

bumping for inactivity.

could really use some direction in troubleshooting this failure.... it's been weeks now since critical localhost mounts have been backed up.

please advise.

JS

[an_admin]

Looking at the logs today... I see a massive time jump, 16,000+ seconds(?!?) between the last successful command and the timeout of localhost mounts.

planner: time 3008.021: got result for host 10.6.1.209 disk /mnt/IT: 0 -> 348807807K, 1 -> 131976K, 2 -> 131542K
planner: time 19458.769: got partial result for host localhost disk /srv/amanda/state: 0 -> 547680K, 1 -> 403170K, -1 -> -3K
planner: time 19458.769: got partial result for host localhost disk /srv/reports: 0 -> 29070K, 1 -> 1960K, -1 -> -3K
planner: time 19458.770: got partial result for host localhost disk /tank/backups: 0 -> 1931437730K, 2 -> -3K, 3 -> -3K
planner: time 19458.770: got partial result for host localhost disk /: 0 -> 13660480K, -1 -> -3K, -1 -> -3K
error result for host localhost disk /srv/amanda/state: localhost NAK: timeout on reply pipe
error result for host localhost disk /srv/reports: localhost NAK: timeout on reply pipe
error result for host localhost disk /tank/backups: localhost NAK: timeout on reply pipe
error result for host localhost disk /: localhost NAK: timeout on reply pipe
planner: time 21623.480: getting estimates took 21601.067 secs
FAILED QUEUE:
  0: localhost  /srv/amanda/state
  1: localhost  /srv/reports
  2: localhost  /tank/backups
  3: localhost  /   

planner: FAILED localhost /srv/amanda/state 20170320000001 0 "[localhost NAK: timeout on reply pipe]"
planner: FAILED localhost /srv/reports 20170320000001 0 "[localhost NAK: timeout on reply pipe]"
planner: FAILED localhost /tank/backups 20170320000001 0 "[localhost NAK: timeout on reply pipe]"
planner: FAILED localhost / 20170320000001 0 "[localhost NAK: timeout on reply pipe]"

I am at a loss here. Desperately need to resolve this issue.

help.

[an_admin]

sigh.

BUMP