PDA

View Full Version : "mysqlhotcopy" out of memory during remote backup



joon321
April 22nd, 2008, 02:18 PM
I am now trying to backup using raw mode but failure occurs with running mysqlhotcopy

[root@backup amanda]# mysql-zrm-scheduler --now --backup-set remotebackup
schedule:INFO: ZRM for MySQL Community Edition - version 2.0
Logging to /var/log/mysql-zrm/mysql-zrm-scheduler.log
backup:INFO: ZRM for MySQL Community Edition - version 2.0
remotebackup:backup:INFO: START OF BACKUP
remotebackup:backup:INFO: PHASE START: Initialization
remotebackup:backup:WARNING: The lvm-snapshot option is deprecated. Use snapshot -size instead
remotebackup:backup:INFO: Mail address: admin@junipro.com is ok
remotebackup:backup:INFO: ZRM Temporary configuration file = /etc/mysql-zrm/remo tebackup/tmp1sje9.conf
remotebackup:backup:INFO: {
remotebackup:backup:INFO: verbose=1
remotebackup:backup:INFO: retention-policy=10D
remotebackup:backup:INFO: tmpdir=/tmp
remotebackup:backup:INFO: destination=/var/lib/mysql-zrm
remotebackup:backup:INFO: socket-remote-port=25300
remotebackup:backup:INFO: password=******
remotebackup:backup:INFO: backup-mode=raw
remotebackup:backup:INFO: snapshot-plugin=/usr/share/mysql-zrm/plugins/lvm -snapshot.pl
remotebackup:backup:INFO: compress-plugin=/usr/bin/gzip
remotebackup:backup:INFO: user=root
remotebackup:backup:INFO: copy-plugin=/usr/share/mysql-zrm/plugins/socket- copy.pl
remotebackup:backup:INFO: quiet=0
remotebackup:backup:INFO: backup-level=0
remotebackup:backup:INFO: lvm-snapshot=10M
remotebackup:backup:INFO: mailto=admin@junipro.com
remotebackup:backup:INFO: databases=world
remotebackup:backup:INFO: port=3306
remotebackup:backup:INFO: host=192.168.1.102
remotebackup:backup:INFO: snapshot-size=10M
remotebackup:backup:INFO: compress=/usr/bin/gzip
remotebackup:backup:INFO: mysql-binlog-path=/var/run/mysqld
remotebackup:backup:INFO: }
remotebackup:backup:INFO: Getting mysql variables
remotebackup:backup:INFO: mysqladmin --user="root" --password="*****" --host="19 2.168.1.102" --port="3306" variables
remotebackup:backup:INFO: datadir is /var/lib/mysql/
remotebackup:backup:INFO: mysql_version is 5.0.45-log
remotebackup:backup:INFO: InnoDB data file are /var/lib/mysql/ibdata1
remotebackup:backup:INFO: InnoDB log dir is /var/lib/mysql/.
remotebackup:backup:INFO: backup set being used is remotebackup
remotebackup:backup:INFO: backup-set=remotebackup
remotebackup:backup:INFO: backup-date=20080422142515
remotebackup:backup:INFO: mysql-server-os=Linux/Unix
remotebackup:backup:INFO: host=192.168.1.102
remotebackup:backup:INFO: backup-date-epoch=1208899515
remotebackup:backup:INFO: retention-policy=10D
remotebackup:backup:INFO: mysql-zrm-version=ZRM for MySQL Community Edition - ve rsion 2.0
remotebackup:backup:INFO: mysql-version=5.0.45-log
remotebackup:backup:INFO: backup-directory=/var/lib/mysql-zrm/remotebackup/20080 422142515
remotebackup:backup:INFO: backup-level=0
remotebackup:backup:INFO: backup-mode=raw
remotebackup:backup:INFO: PHASE END: Initialization
remotebackup:backup:INFO: PHASE START: Running pre backup plugin
remotebackup:backup:INFO: Executing pre-backup-plugin
remotebackup:backup:INFO: PHASE END: Running pre backup plugin
remotebackup:backup:INFO: PHASE START: Flushing logs
remotebackup:backup:INFO: Flushing the logs
remotebackup:backup:INFO: mysqladmin --user="root" --password="*****" --host="19 2.168.1.102" --port="3306" flush-logs
remotebackup:backup:INFO: Getting master logname using command mysql --user="roo t" --password="*****" --host="192.168.1.102" --port="3306" -e "show master statu s"
remotebackup:backup:INFO: PHASE END: Flushing logs
remotebackup:backup:INFO: PHASE START: Creating snapshot based backup
remotebackup:backup:INFO: innodb on lvm = 0
remotebackup:backup:INFO: getting device details using command /usr/share/mysql- zrm/plugins/lvm-snapshot.pl --action get-vm-device-details --directory /var/lib/ mysql/. --sname zrmUE0vbHKyhN --host 192.168.1.102 2>/tmp/i3NkWQG6ia
remotebackup:backup:INFO: PHASE END: Creating snapshot based backup
remotebackup:backup:INFO: PHASE START: Find table type
remotebackup:backup:INFO: Command used for getting engine type mysql --user="roo t" --password="*****" --host="192.168.1.102" --port="3306" -e "show table status from \`world\`"
remotebackup:backup:INFO: For database world
remotebackup:backup:INFO: engine
remotebackup:backup:INFO: MyISAM
remotebackup:backup:INFO: MyISAM
remotebackup:backup:INFO: MyISAM
remotebackup:backup:INFO: PHASE END: Find table type
remotebackup:backup:INFO: PHASE START: Creating raw backup
remotebackup:backup:INFO: Command used for raw backup is /usr/share/mysql-zrm/pl ugins/socket-copy.pl --mysqlhotcopy --user="root" --password="*****" --host="192 .168.1.102" --port="3306" --quiet world "/var/lib/mysql-zrm/remotebackup/20080 422142515" > /tmp/i3NkWQG6ia 2>&1
remotebackup:backup:ERROR: Output of command: 'mysqlhotcopy' is {
Out of memory!
}
remotebackup:backup:ERROR: mysqlhotcopy did not succeed. Command used is /usr/sh are/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="root" --password="** ***" --host="192.168.1.102" --port="3306" --quiet world "/var/lib/mysql-zrm/re motebackup/20080422142515" > /tmp/i3NkWQG6ia 2>&1 Return value is 256
remotebackup:backup:INFO: PHASE START: Cleanup
remotebackup:backup:INFO: backup-status=Backup failed
remotebackup:backup:INFO: Backup failed
remotebackup:backup:INFO: mailing file /tmp/Tv7paBxVWl
remotebackup:backup:INFO: mail command is cat "/tmp/Tv7paBxVWl"|mail -s "[ZRM fo r MySQL Report] ERROR during backup of backup-set remotebackup" admin@junipro.co m
remotebackup:backup:INFO: PHASE END: Cleanup
remotebackup:backup:INFO: END OF BACKUP
ERROR: /usr/bin/mysql-zrm did not finish successfully

I have verified xinetd service running with ZRM client support on port 25300. This is running on fedora 7 and mysqlhotcopy runs under /usr/bin directory both running on mysql and zmanda. I run raw backup locally and it works fine; the problem is only during the remote raw backup. The raw backup should work remotely via socket-server plugin correct? Thanks for your help.

kkg
April 24th, 2008, 02:19 AM
Please let us know what version of tar you are using.

ZRM needs 1.15.1 or above.

Also please check if you have enabled warnings in the socket-server.pl.
--kkg

joon321
April 24th, 2008, 12:31 PM
Currently using
tar (GNU tar) 1.15.1

I don't remember enabling warnings in socket-server.pl. I don't see it in the configuration file either

service mysql-zrm-socket-server
{
type = UNLISTED
disable = no
socket_type = stream
protocol = tcp
wait = no
user = mysql
group = mysql
instances = 1
server = /usr/share/mysql-zrm/plugins/socket-server.pl
port = 25300
}

Thanks

joon321
May 5th, 2008, 09:56 AM
I tried manually running

mysqlhotcopy --user="root" --password="*****" --host="192 .168.1.102" --quiet world "/var/lib/mysql-zrm/remotebackup/rawbackup

And it works! Is this a known issue with socket-copy.pl?

Any help is appreciated

symfrog
July 20th, 2008, 08:43 AM
Hi

I am experiencing the exact same problem using ZRM community edition 2.0 and tar 1.19 on ubuntu Hardy.

It does seem to be a problem with socket-copy.pl since it works fine if I use ssh-copy.pl

Can anyone at Zmanda help us with this issue since I would prefer to go the socket copy route for efficiency?

Thanks

kulkarni_mangesh
July 21st, 2008, 04:07 AM
It looks like string size in not proper which is sent by the client. To debug this problem can you please add
“print "Zmanda_Debug: \n $buf \n";” line in “sub readTarStream()” of file “/usr/share/mysql-zrm/plugins/socket-copy.pl” on ZRM server and post the output of the same on the forum?

Following is the sample code of “sub readTarStream()” , which should look like after the change on the ZRM server.

====( Start “sub readTarStream()” ) ====

sub readTarStream()
{
unless( open( TAR_H, "|$TAR --same-owner -xphszC $destDir 2>/dev/null" ) ){
die "tar failed $!";
}
binmode( TAR_H );

my $buf;

# Initially read the length of data to read
# This will be packed in network order
# Then read that much data which is uuencoded
# Then write the unpacked data to tar
while( read( SOCK, $buf, 4 ) ){
print "Zmanda_Debug: \n $buf \n"; <<<<<<<< Watch here
$buf = unpack( "N", $buf );
read SOCK, $buf, $buf;
print TAR_H unpack( "u", $buf );
}
unless( close(TAR_H) ){
die "close of pipe failed\n";
}
}

====( End “sub readTarStream()” ) ====

symfrog
July 21st, 2008, 09:44 AM
I have attached the output, hope it helps

kulkarni_mangesh
July 21st, 2008, 07:45 PM
The problem we are suspecting at “mysqlhotcopy”. It is returning some message ( message could be the error message ).
In the attached file we are getting “Unab” which is returned by “mysqlhotcopy”.

Can you please increase the size of buffer so we can extract the exact message returned by “mysqlhotcopy” and post the output on the forum?
In following sample code of “sub readTarStream()”, we have increased buffer size from 4 to 255 characters so we can know exact message returned by “mysqlhotcopy”.

“sub readTarStream()” from “/usr/share/mysql-zrm/plugins/socket-copy.pl” file, should look like as follows, after the change, on the ZRM server.

====( Start “sub readTarStream()” ) ====

sub readTarStream()
{
unless( open( TAR_H, "|$TAR --same-owner -xphszC $destDir 2>/dev/null" ) ){
die "tar failed $!";
}
binmode( TAR_H );

my $buf;

# Initially read the length of data to read
# This will be packed in network order
# Then read that much data which is uuencoded
# Then write the unpacked data to tar
while( read( SOCK, $buf, 255 ) ){ <<<<<<<< Watch here
print "Zmanda_Debug: \n $buf \n"; <<<<<<<< Watch here
$buf = unpack( "N", $buf );
read SOCK, $buf, $buf;
print TAR_H unpack( "u", $buf );
}
unless( close(TAR_H) ){
die "close of pipe failed\n";
}
}

====( End “sub readTarStream()” ) ====

symfrog
July 22nd, 2008, 12:45 AM
It is a remote backup. The mysql-zrm-scheduler is being run as root. The output with the modifications requested is attached.

Could you reproduce this on your side?

kulkarni_mangesh
July 22nd, 2008, 02:24 AM
No, unfortunately we are not able to reproduce this issue at our end.

This issue is occurring because there is some issue opening
/var/log/mysql-zrm/socket-server.log on Remote MySQL server.

We need to check following things:

1. See if /var/log/mysql-zrm directory exists or not. If it does not exist please create this directory and go to next step.

2. Ownership and Permissions to write at “/var/log/mysql-zrm” and logfile “/var/log/mysql-zrm/socket-server.log, output of “ls -al /var/log/mysql-zrm” should look like

ls -al /var/log/mysql-zrm
total 52
drwxrwx--- 2 mysql mysql 4096 Aug 19 19:55 . <<<<<< Watch here

drwxr-xr-x 17 root root 4096 Aug 19 19:43 ..
-rwxrwx--- 1 mysql mysql 8080 Jul 2 04:02 mysql-zrm.log.1.gz
-rw-r--r-- 1 mysql mysql 8810 Jul 22 02:00 mysql-zrm-scheduler.log

-rw-rw-r-- 1 mysql mysql 101 Aug 20 19:31 socket-server.log <<<<<< Watch here

-rw-r--r-- 1 mysql mysql 13847 Jul 22 02:00 zrm-pre-scheduler.log

3. Enough disk space to create log files at “/var/log/mysql-zrm”. Make sure that “/var/log/mysql-zrm” is not full. If disk is full then please make some space on it.
Following command will return diskspace available on “/var/log/mysql-zrm”.
cd /var/log/mysql-zrm
df –h ./