Hi Everyone,

I can not perform backups of one of the linux clients.
Every time backup is interrupted with the same error in log.
I tried to search, but didn't find any workarounds related to this issue.

The client and the server have been installed from the OL latest el5_latest repository.

So the issue itself:

the client:

[root@ovmman /]# uname -a
Linux ovmman 2.6.39-400.215.13.el5uek #1 SMP Wed Nov 12 12:27:23 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

Installed Packages
Name : amanda-client
Arch : x86_64
Version : 2.5.0p2
Release : 9.el5

the server:

build: VERSION="Amanda-2.5.0p2"

[root@tom /]# grep ovmman /etc/amanda/DailySet1/disklist
ovmman /sysbackups nocomp-root

define dumptype nocomp-root {
comp-root
program "GNUTAR"
comment "Root partitions without compression"
compress none
priority high
}

After backup is started the contents of the logs are:

server:

[root@tom logs]# cat amdump.1
amdump: start at Wed Dec 3 14:40:28 NZDT 2014
amdump: datestamp 20141203
driver: pid 8837 executable /usr/lib/amanda/driver version 2.5.0p2
......
dumper: kill index command
driver: state time 1080.102 free kps: 676 space: 126795112 taper: idle idle-dumpers: 3 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers
driver: interface-state time 1080.102 if : free -724 if LE0: free 400 if LOCAL: free 1000
driver: hdisk-state time 1080.102 hdisk 0: free 126795112 dumpers 1
driver: result time 1080.102 from dumper0: FAILED 00-00001 [data read: Connection reset by peer]
driver: send-cmd time 1080.102 to chunker0: FAILED 00-00001
driver: state time 1080.102 free kps: 676 space: 126795112 taper: idle idle-dumpers: 3 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers
driver: interface-state time 1080.102 if : free -724 if LE0: free 400 if LOCAL: free 1000
driver: hdisk-state time 1080.102 hdisk 0: free 126795112 dumpers 1
driver: result time 1080.102 from chunker0: FAILED 00-00001 [dumper returned FAILED]
driver: started chunker0 pid 8972
driver: send-cmd time 1080.103 to chunker0: START 20141203
driver: send-cmd time 1080.103 to chunker0: PORT-WRITE 00-00002 /u01/amdumps/20141203144028/ovmman._sysbackups.0 ovmman fffffeff9ffeffff07 /sysbackups 0 1970:1:1:0:0:0 1048576 GNUTAR 1458624 |;auth=BSD;index;
chunker: pid 8972 executable chunker0 version 2.5.0p2
chunker: try_socksize: receive buffer size is 65536
chunker: bind_portrange2: trying port=849
chunker: stream_server: waiting for connection: 0.0.0.0.41488
driver: result time 1080.112 from chunker0: PORT 41488
driver: send-cmd time 1080.112 to dumper0: PORT-DUMP 00-00002 41488 ovmman fffffeff9ffeffff07 /sysbackups NODEVICE 0 1970:1:1:0:0:0 GNUTAR |;auth=BSD;index;
dumper: time 1080.000: stream_client: connected to 127.0.0.1.41488
chunker: stream_accept: connection from 127.0.0.1.41426
dumper: stream_client: our side is 0.0.0.0.41426
chunker: try_socksize: receive buffer size is 32768
dumper: try_socksize: send buffer size is 65536
dumper: time 1080.081: stream_client: connected to 172.24.224.52.45317
dumper: stream_client: our side is 0.0.0.0.35542
dumper: try_socksize: send buffer size is 65536
dumper: try_socksize: receive buffer size is 65536
dumper: time 1080.081: stream_client: connected to 172.24.224.52.39293
dumper: stream_client: our side is 0.0.0.0.47089
dumper: try_socksize: send buffer size is 65536
dumper: try_socksize: receive buffer size is 65536
dumper: time 1080.082: stream_client: connected to 172.24.224.52.51283
dumper: stream_client: our side is 0.0.0.0.48731
dumper: try_socksize: send buffer size is 65536
dumper: try_socksize: receive buffer size is 65536
dumper: kill index command
driver: state time 2160.183 free kps: 676 space: 126795112 taper: idle idle-dumpers: 3 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers
driver: interface-state time 2160.183 if : free -724 if LE0: free 400 if LOCAL: free 1000
driver: hdisk-state time 2160.183 hdisk 0: free 126795112 dumpers 1
driver: result time 2160.183 from dumper0: FAILED 00-00002 [data read: Connection reset by peer]
driver: send-cmd time 2160.183 to chunker0: FAILED 00-00002
driver: state time 2160.184 free kps: 676 space: 126795112 taper: idle idle-dumpers: 3 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers
driver: interface-state time 2160.184 if : free -724 if LE0: free 400 if LOCAL: free 1000
driver: hdisk-state time 2160.184 hdisk 0: free 126795112 dumpers 1
driver: result time 2160.184 from chunker0: FAILED 00-00002 [dumper returned FAILED]
driver: state time 2160.184 free kps: 1700 space: 128253736 taper: idle idle-dumpers: 4 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers
driver: interface-state time 2160.184 if : free 300 if LE0: free 400 if LOCAL: free 1000
driver: hdisk-state time 2160.184 hdisk 0: free 128253736 dumpers 0
driver: QUITTING time 2160.184 telling children to quit
driver: send-cmd time 2160.184 to dumper0: QUIT
driver: send-cmd time 2160.184 to dumper1: QUIT
driver: send-cmd time 2160.184 to dumper2: QUIT
driver: send-cmd time 2160.184 to dumper3: QUIT
driver: send-cmd time 2160.184 to taper: QUIT
taper: DONE [idle wait: 2160.160 secs]
taper: writing end marker. [DailySet1-005 OK kb 0 fm 0]
driver: FINISHED time 2163.289
amdump: end at Wed Dec 3 15:16:31 NZDT 2014
Scanning /u01/amdumps...
20141203105859: found Amanda directory.
[root@tom logs]#

[root@tom logs]# cat log.20141203.5
START driver date 20141203
DISK planner ovmman /sysbackups
START planner date 20141203
STATS driver startup time 0.014
START taper datestamp 20141203 label DailySet1-005 tape 0
FINISH planner date 20141203 time 0.043
FAIL chunker ovmman /sysbackups 20141203 0 [dumper returned FAILED]
FAIL dumper ovmman /sysbackups 20141203 0 [data read: Connection reset by peer]
sendbackup: start [ovmman:/sysbackups level 0]
sendbackup: info BACKUP=/bin/tar
sendbackup: info RECOVER_CMD=/bin/tar -f... -
sendbackup: info end
? dumper: strange [missing size line from sendbackup]
FAIL dumper ovmman /sysbackups 20141203 0 [data read: Connection reset by peer]
sendbackup: start [ovmman:/sysbackups level 0]
sendbackup: info BACKUP=/bin/tar
sendbackup: info RECOVER_CMD=/bin/tar -f... -
sendbackup: info end
? dumper: strange [missing size line from sendbackup]
FAIL chunker ovmman /sysbackups 20141203 0 [dumper returned FAILED]
INFO taper tape DailySet1-005 kb 0 fm 0 [OK]
FINISH driver date 20141203 time 2163.289

client

[root@ovmman amanda]# cat amandad.20141203145828.debug
amandad: debug 1 pid 9249 ruid 33 euid 33: start at Wed Dec 3 14:58:28 2014
amandad: version 2.5.0p2
amandad: build: VERSION="Amanda-2.5.0p2"
amandad: BUILT_DATE="Tue Dec 20 04:00:03 EST 2011"
amandad: BUILT_MACH="Linux ca-build10.us.oracle.com 2.6.20-1.3002.fc6xen #1 SMP Thu Apr 30 18:08:39 PDT 2009 x86_64 x86_64 x86_64 GNU/Linux"
amandad: CC="gcc"
amandad: CONFIGURE_COMMAND="'./configure' '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/lib64/amanda' '--localstatedir=/var/lib' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-shared' '--disable-static' '--disable-dependency-tracking' '--with-index-server=amandahost' '--with-tape-server=amandahost' '--with-config=DailySet1' '--with-gnutar-listdir=/var/lib/amanda/gnutar-lists' '--with-smbclient=/usr/bin/smbclient' '--with-dumperdir=/usr/lib64/amanda/dumperdir' '--with-amandahosts' '--with-user=amanda' '--with-group=disk' '--with-tmpdir=/var/log/amanda' '--with-gnutar=/bin/tar' '--with-ssh-security'"
amandad: paths: bindir="/usr/bin" sbindir="/usr/sbin"
amandad: libexecdir="/usr/lib64/amanda" mandir="/usr/share/man"
amandad: AMANDA_TMPDIR="/var/log/amanda"
amandad: AMANDA_DBGDIR="/var/log/amanda" CONFIG_DIR="/etc/amanda"
amandad: DEV_PREFIX="/dev/" RDEV_PREFIX="/dev/r"
amandad: DUMP="/sbin/dump" RESTORE="/sbin/restore" VDUMP=UNDEF
amandad: VRESTORE=UNDEF XFSDUMP=UNDEF XFSRESTORE=UNDEF VXDUMP=UNDEF
amandad: VXRESTORE=UNDEF SAMBA_CLIENT="/usr/bin/smbclient"
amandad: GNUTAR="/bin/tar" COMPRESS_PATH="/bin/gzip"
amandad: UNCOMPRESS_PATH="/bin/gzip" LPRCMD="/usr/bin/lpr"
amandad: MAILER="/usr/bin/Mail"
amandad: listed_incr_dir="/var/lib/amanda/gnutar-lists"
amandad: defs: DEFAULT_SERVER="amandahost" DEFAULT_CONFIG="DailySet1"
amandad: DEFAULT_TAPE_SERVER="amandahost"
amandad: DEFAULT_TAPE_DEVICE="null:" HAVE_MMAP HAVE_SYSVSHM
amandad: LOCKING=POSIX_FCNTL SETPGRP_VOID DEBUG_CODE
amandad: AMANDA_DEBUG_DAYS=4 BSD_SECURITY RSH_SECURITY USE_AMANDAHOSTS
amandad: CLIENT_LOGIN="amanda" FORCE_USERID HAVE_GZIP
amandad: COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast"
amandad: COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc"
amandad: time 0.000: accept recv REQ pkt:
<<<<<
SERVICE sendbackup
OPTIONS features=fffffeff9ffeffff07;hostname=ovmman;
GNUTAR /sysbackups 0 1970:1:1:0:0:0 OPTIONS |;auth=BSD;index;
>>>>>
amandad: time 0.000: creating new service: /usr/lib64/amanda/sendbackup
OPTIONS features=fffffeff9ffeffff07;hostname=ovmman;
GNUTAR /sysbackups 0 1970:1:1:0:0:0 OPTIONS |;auth=BSD;index;

amandad: time 0.000: sending ACK pkt:
<<<<<
>>>>>
amandad: try_socksize: send buffer size is 65536
amandad: try_socksize: receive buffer size is 65536
amandad: time 0.026: bind_portrange2: trying port=659
amandad: time 0.026: stream_server: waiting for connection: 0.0.0.0.45317
amandad: try_socksize: send buffer size is 65536
amandad: try_socksize: receive buffer size is 65536
amandad: time 0.041: bind_portrange2: trying port=659
amandad: time 0.041: stream_server: waiting for connection: 0.0.0.0.39293
amandad: try_socksize: send buffer size is 65536
amandad: try_socksize: receive buffer size is 65536
amandad: time 0.056: bind_portrange2: trying port=659
amandad: time 0.056: stream_server: waiting for connection: 0.0.0.0.51283
amandad: time 0.056: sending REP pkt:
<<<<<
CONNECT DATA 45317 MESG 39293 INDEX 51283
OPTIONS features=fffffeff9ffeffff07;
>>>>>
amandad: time 0.057: received ACK pkt:
<<<<<
>>>>>
amandad: time 0.058: stream_accept: connection from 172.24.224.12.35542
amandad: time 0.058: stream_accept: connection from 172.24.224.12.47089
amandad: time 0.059: stream_accept: connection from 172.24.224.12.48731
amandad: time 927.981: sending NAK pkt:
<<<<<
ERROR write error on stream 45317: write error on stream 45317: Broken pipe
>>>>>

[root@ovmman amanda]# cat sendbackup.20141203145828.debug
sendbackup: debug 1 pid 9250 ruid 33 euid 33: start at Wed Dec 3 14:58:28 2014
sendbackup: version 2.5.0p2
parsed request as: program `GNUTAR'
disk `/sysbackups'
device `/sysbackups'
level 0
since 1970:1:1:0:0:0
options `|;auth=BSD;index;'
sendbackup-gnutar: time 0.004: doing level 0 dump as listed-incremental to /var/lib/amanda/gnutar-lists/ovmman_sysbackups_0.new
sendbackup-gnutar: time 0.004: doing level 0 dump from date: 1970-01-01 0:00:00 GMT
sendbackup: time 0.005: spawning /usr/lib64/amanda/runtar in pipeline
sendbackup: argument list: gtar --create --file - --directory /sysbackups --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/ovmman_sysbackups_0.new --sparse --ignore-failed-read --totals --wildcards .
sendbackup-gnutar: time 0.005: /usr/lib64/amanda/runtar: pid 9253
sendbackup: time 0.005: started backup
sendbackup: time 0.012: started index creator: "/bin/tar -tf - 2>/dev/null | sed -e 's/^\.//'"
sendbackup: time 927.980: index tee cannot write [Broken pipe]
sendbackup: time 927.980: pid 9252 finish time Wed Dec 3 15:13:56 2014
sendbackup: time 927.981: 119: strange(?): sendbackup: index tee cannot write [Broken pipe]
sendbackup: time 927.981: 46: size(|): Total bytes written: 235520 (230KiB, ?/s)
sendbackup: time 927.981: 119: strange(?): gtar: -: Cannot write: Broken pipe
sendbackup: time 927.981: 119: strange(?): gtar: Error is not recoverable: exiting now
sendbackup: time 927.982: 119: strange(?): sed: couldn't flush stdout: Broken pipe
sendbackup: time 927.985: error [/bin/tar returned 2]
sendbackup: time 927.985: pid 9250 finish time Wed Dec 3 15:13:56 2014

[root@ovmman amanda]# cat runtar.20141203145828.debug
runtar: debug 1 pid 9253 ruid 33 euid 0: start at Wed Dec 3 14:58:28 2014
gtar: version 2.5.0p2
running: /bin/tar: gtar --create --file - --directory /sysbackups --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/ovmman_sysbackups_0.new --sparse --ignore-failed-read --totals --wildcards .
runtar: pid 9253 finish time Wed Dec 3 14:58:28 2014


I tried these workarounds but none of them helped:

1) wiki.zmanda.com/index.php/Amdump:_mesg_read:_Connection_reset_by_peer
2) tried to increase UDP buffer size: sysctl -w net.core.rmem_max=26214400


PLEASE! anybody who had the same issue in the past help to resolve it.