Results 1 to 10 of 10

Thread: "mysqlhotcopy" out of memory during remote backup

  1. #1

    Default "mysqlhotcopy" out of memory during remote backup

    I am now trying to backup using raw mode but failure occurs with running mysqlhotcopy

    [root@backup amanda]# mysql-zrm-scheduler --now --backup-set remotebackup
    schedule:INFO: ZRM for MySQL Community Edition - version 2.0
    Logging to /var/log/mysql-zrm/mysql-zrm-scheduler.log
    backup:INFO: ZRM for MySQL Community Edition - version 2.0
    remotebackup:backup:INFO: START OF BACKUP
    remotebackup:backup:INFO: PHASE START: Initialization
    remotebackup:backup:WARNING: The lvm-snapshot option is deprecated. Use snapshot -size instead
    remotebackup:backup:INFO: Mail address: [email]admin@junipro.com[/email] is ok
    remotebackup:backup:INFO: ZRM Temporary configuration file = /etc/mysql-zrm/remo tebackup/tmp1sje9.conf
    remotebackup:backup:INFO: {
    remotebackup:backup:INFO: verbose=1
    remotebackup:backup:INFO: retention-policy=10D
    remotebackup:backup:INFO: tmpdir=/tmp
    remotebackup:backup:INFO: destination=/var/lib/mysql-zrm
    remotebackup:backup:INFO: socket-remote-port=25300
    remotebackup:backup:INFO: password=******
    remotebackup:backup:INFO: backup-mode=raw
    remotebackup:backup:INFO: snapshot-plugin=/usr/share/mysql-zrm/plugins/lvm -snapshot.pl
    remotebackup:backup:INFO: compress-plugin=/usr/bin/gzip
    remotebackup:backup:INFO: user=root
    remotebackup:backup:INFO: copy-plugin=/usr/share/mysql-zrm/plugins/socket- copy.pl
    remotebackup:backup:INFO: quiet=0
    remotebackup:backup:INFO: backup-level=0
    remotebackup:backup:INFO: lvm-snapshot=10M
    remotebackup:backup:INFO: mailto=admin@junipro.com
    remotebackup:backup:INFO: databases=world
    remotebackup:backup:INFO: port=3306
    remotebackup:backup:INFO: host=192.168.1.102
    remotebackup:backup:INFO: snapshot-size=10M
    remotebackup:backup:INFO: compress=/usr/bin/gzip
    remotebackup:backup:INFO: mysql-binlog-path=/var/run/mysqld
    remotebackup:backup:INFO: }
    remotebackup:backup:INFO: Getting mysql variables
    remotebackup:backup:INFO: mysqladmin --user="root" --password="*****" --host="19 2.168.1.102" --port="3306" variables
    remotebackup:backup:INFO: datadir is /var/lib/mysql/
    remotebackup:backup:INFO: mysql_version is 5.0.45-log
    remotebackup:backup:INFO: InnoDB data file are /var/lib/mysql/ibdata1
    remotebackup:backup:INFO: InnoDB log dir is /var/lib/mysql/.
    remotebackup:backup:INFO: backup set being used is remotebackup
    remotebackup:backup:INFO: backup-set=remotebackup
    remotebackup:backup:INFO: backup-date=20080422142515
    remotebackup:backup:INFO: mysql-server-os=Linux/Unix
    remotebackup:backup:INFO: host=192.168.1.102
    remotebackup:backup:INFO: backup-date-epoch=1208899515
    remotebackup:backup:INFO: retention-policy=10D
    remotebackup:backup:INFO: mysql-zrm-version=ZRM for MySQL Community Edition - ve rsion 2.0
    remotebackup:backup:INFO: mysql-version=5.0.45-log
    remotebackup:backup:INFO: backup-directory=/var/lib/mysql-zrm/remotebackup/20080 422142515
    remotebackup:backup:INFO: backup-level=0
    remotebackup:backup:INFO: backup-mode=raw
    remotebackup:backup:INFO: PHASE END: Initialization
    remotebackup:backup:INFO: PHASE START: Running pre backup plugin
    remotebackup:backup:INFO: Executing pre-backup-plugin
    remotebackup:backup:INFO: PHASE END: Running pre backup plugin
    remotebackup:backup:INFO: PHASE START: Flushing logs
    remotebackup:backup:INFO: Flushing the logs
    remotebackup:backup:INFO: mysqladmin --user="root" --password="*****" --host="19 2.168.1.102" --port="3306" flush-logs
    remotebackup:backup:INFO: Getting master logname using command mysql --user="roo t" --password="*****" --host="192.168.1.102" --port="3306" -e "show master statu s"
    remotebackup:backup:INFO: PHASE END: Flushing logs
    remotebackup:backup:INFO: PHASE START: Creating snapshot based backup
    remotebackup:backup:INFO: innodb on lvm = 0
    remotebackup:backup:INFO: getting device details using command /usr/share/mysql- zrm/plugins/lvm-snapshot.pl --action get-vm-device-details --directory /var/lib/ mysql/. --sname zrmUE0vbHKyhN --host 192.168.1.102 2>/tmp/i3NkWQG6ia
    remotebackup:backup:INFO: PHASE END: Creating snapshot based backup
    remotebackup:backup:INFO: PHASE START: Find table type
    remotebackup:backup:INFO: Command used for getting engine type mysql --user="roo t" --password="*****" --host="192.168.1.102" --port="3306" -e "show table status from \`world\`"
    remotebackup:backup:INFO: For database world
    remotebackup:backup:INFO: engine
    remotebackup:backup:INFO: MyISAM
    remotebackup:backup:INFO: MyISAM
    remotebackup:backup:INFO: MyISAM
    remotebackup:backup:INFO: PHASE END: Find table type
    remotebackup:backup:INFO: PHASE START: Creating raw backup
    remotebackup:backup:INFO: Command used for raw backup is /usr/share/mysql-zrm/pl ugins/socket-copy.pl --mysqlhotcopy --user="root" --password="*****" --host="192 .168.1.102" --port="3306" --quiet world "/var/lib/mysql-zrm/remotebackup/20080 422142515" > /tmp/i3NkWQG6ia 2>&1
    remotebackup:backup:ERROR: Output of command: 'mysqlhotcopy' is {
    Out of memory!
    }
    remotebackup:backup:ERROR: mysqlhotcopy did not succeed. Command used is /usr/sh are/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="root" --password="** ***" --host="192.168.1.102" --port="3306" --quiet world "/var/lib/mysql-zrm/re motebackup/20080422142515" > /tmp/i3NkWQG6ia 2>&1 Return value is 256
    remotebackup:backup:INFO: PHASE START: Cleanup
    remotebackup:backup:INFO: backup-status=Backup failed
    remotebackup:backup:INFO: Backup failed
    remotebackup:backup:INFO: mailing file /tmp/Tv7paBxVWl
    remotebackup:backup:INFO: mail command is cat "/tmp/Tv7paBxVWl"|mail -s "[ZRM fo r MySQL Report] ERROR during backup of backup-set remotebackup" [email]admin@junipro.co[/email] m
    remotebackup:backup:INFO: PHASE END: Cleanup
    remotebackup:backup:INFO: END OF BACKUP
    ERROR: /usr/bin/mysql-zrm did not finish successfully

    I have verified xinetd service running with ZRM client support on port 25300. This is running on fedora 7 and mysqlhotcopy runs under /usr/bin directory both running on mysql and zmanda. I run raw backup locally and it works fine; the problem is only during the remote raw backup. The raw backup should work remotely via socket-server plugin correct? Thanks for your help.

  2. #2
    Join Date
    Oct 2006
    Posts
    199

    Default

    Please let us know what version of tar you are using.

    ZRM needs 1.15.1 or above.

    Also please check if you have enabled warnings in the socket-server.pl.
    --kkg

  3. #3

    Default

    Currently using
    tar (GNU tar) 1.15.1

    I don't remember enabling warnings in socket-server.pl. I don't see it in the configuration file either

    service mysql-zrm-socket-server
    {
    type = UNLISTED
    disable = no
    socket_type = stream
    protocol = tcp
    wait = no
    user = mysql
    group = mysql
    instances = 1
    server = /usr/share/mysql-zrm/plugins/socket-server.pl
    port = 25300
    }

    Thanks

  4. #4

    Default

    I tried manually running

    mysqlhotcopy --user="root" --password="*****" --host="192 .168.1.102" --quiet world "/var/lib/mysql-zrm/remotebackup/rawbackup

    And it works! Is this a known issue with socket-copy.pl?

    Any help is appreciated

  5. #5

    Default Same problem

    Hi

    I am experiencing the exact same problem using ZRM community edition 2.0 and tar 1.19 on ubuntu Hardy.

    It does seem to be a problem with socket-copy.pl since it works fine if I use ssh-copy.pl

    Can anyone at Zmanda help us with this issue since I would prefer to go the socket copy route for efficiency?

    Thanks

  6. #6

    Default

    It looks like string size in not proper which is sent by the client. To debug this problem can you please add
    “print "Zmanda_Debug: \n $buf \n";” line in “sub readTarStream()” of file “/usr/share/mysql-zrm/plugins/socket-copy.pl” on ZRM server and post the output of the same on the forum?

    Following is the sample code of “sub readTarStream()” , which should look like after the change on the ZRM server.

    ====( Start “sub readTarStream()” ) ====

    sub readTarStream()
    {
    unless( open( TAR_H, "|$TAR --same-owner -xphszC $destDir 2>/dev/null" ) ){
    die "tar failed $!";
    }
    binmode( TAR_H );

    my $buf;

    # Initially read the length of data to read
    # This will be packed in network order
    # Then read that much data which is uuencoded
    # Then write the unpacked data to tar
    while( read( SOCK, $buf, 4 ) ){
    print "Zmanda_Debug: \n $buf \n"; <<<<<<<< Watch here
    $buf = unpack( "N", $buf );
    read SOCK, $buf, $buf;
    print TAR_H unpack( "u", $buf );
    }
    unless( close(TAR_H) ){
    die "close of pipe failed\n";
    }
    }

    ====( End “sub readTarStream()” ) ====

  7. #7

    Default Output

    I have attached the output, hope it helps
    Attached Files Attached Files

  8. #8

    Default

    The problem we are suspecting at “mysqlhotcopy”. It is returning some message ( message could be the error message ).
    In the attached file we are getting “Unab” which is returned by “mysqlhotcopy”.

    Can you please increase the size of buffer so we can extract the exact message returned by “mysqlhotcopy” and post the output on the forum?
    In following sample code of “sub readTarStream()”, we have increased buffer size from 4 to 255 characters so we can know exact message returned by “mysqlhotcopy”.

    “sub readTarStream()” from “/usr/share/mysql-zrm/plugins/socket-copy.pl” file, should look like as follows, after the change, on the ZRM server.

    ====( Start “sub readTarStream()” ) ====

    sub readTarStream()
    {
    unless( open( TAR_H, "|$TAR --same-owner -xphszC $destDir 2>/dev/null" ) ){
    die "tar failed $!";
    }
    binmode( TAR_H );

    my $buf;

    # Initially read the length of data to read
    # This will be packed in network order
    # Then read that much data which is uuencoded
    # Then write the unpacked data to tar
    while( read( SOCK, $buf, 255 ) ){ <<<<<<<< Watch here
    print "Zmanda_Debug: \n $buf \n"; <<<<<<<< Watch here
    $buf = unpack( "N", $buf );
    read SOCK, $buf, $buf;
    print TAR_H unpack( "u", $buf );
    }
    unless( close(TAR_H) ){
    die "close of pipe failed\n";
    }
    }

    ====( End “sub readTarStream()” ) ====

  9. #9

    Default Output

    It is a remote backup. The mysql-zrm-scheduler is being run as root. The output with the modifications requested is attached.

    Could you reproduce this on your side?
    Attached Files Attached Files

  10. #10

    Default

    No, unfortunately we are not able to reproduce this issue at our end.

    This issue is occurring because there is some issue opening
    /var/log/mysql-zrm/socket-server.log on Remote MySQL server.

    We need to check following things:

    1. See if /var/log/mysql-zrm directory exists or not. If it does not exist please create this directory and go to next step.

    2. Ownership and Permissions to write at “/var/log/mysql-zrm” and logfile “/var/log/mysql-zrm/socket-server.log, output of “ls -al /var/log/mysql-zrm” should look like

    ls -al /var/log/mysql-zrm
    total 52
    drwxrwx--- 2 mysql mysql 4096 Aug 19 19:55 . <<<<<< Watch here

    drwxr-xr-x 17 root root 4096 Aug 19 19:43 ..
    -rwxrwx--- 1 mysql mysql 8080 Jul 2 04:02 mysql-zrm.log.1.gz
    -rw-r--r-- 1 mysql mysql 8810 Jul 22 02:00 mysql-zrm-scheduler.log

    -rw-rw-r-- 1 mysql mysql 101 Aug 20 19:31 socket-server.log <<<<<< Watch here

    -rw-r--r-- 1 mysql mysql 13847 Jul 22 02:00 zrm-pre-scheduler.log

    3. Enough disk space to create log files at “/var/log/mysql-zrm”. Make sure that “/var/log/mysql-zrm” is not full. If disk is full then please make some space on it.
    Following command will return diskspace available on “/var/log/mysql-zrm”.
    cd /var/log/mysql-zrm
    df –h ./

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •