PDA

View Full Version : mysqlhotcopy "close of pipe failed" error with socket-copy.pl



tislammac
October 24th, 2008, 02:06 PM
Hi,

I've spent about a week now trying to solve this. Please help!

Architectural setup:
1.) ZRM server box: running Redhat, x64 bit, Mysql client 5.0.22
2.) Mysql server box: running Ubuntu 8.04, x32 bit, Mysql server 5.0.51a

Here is my mysql-zrm.conf file:
backup-level=0
backup-mode=raw
databases=AssignmentCalculator
user="xxxx0021"
password="xxxx"
host="xxx.xxx.29.87"
mysql-binpath="/usr/bin"
mysql-binlog-path="/var/lib/mysql"
tmpdir=/tmp
verbose=1
copy-plugin=/usr/share/mysql-zrm/plugins/socket-copy.pl
remote-mysql-binpath="/usr/bin"
socket-remote-port="25300"

I made sure that both uid/gid of mysql in the machines match:
> uid=27(mysql) gid=27(mysql) groups=27(mysql) context=user_u:system_r:unconfined_t
> uid=27(mysql) gid=27(mysql) groups=27(mysql)

In my Mysql machine, I can run mysqlhotcopy with user xxxx0021:
$ /usr/bin/mysqlhotcopy --password="xxxx" --user="xxxx0021" --host="xxx.xxx.29.87" AssignmentCalculator /tmp --addtodest
Locked 4 tables in 0 seconds.
Flushed tables (`AssignmentCalculator`.`assignment`, `AssignmentCalculator`.`comment`, `AssignmentCalculator`.`item`, `AssignmentCalculator`.`user`) in 0 seconds.
Copying 13 files...
Copying indices for 0 files...
Unlocked tables.
mysqlhotcopy copied 4 tables (13 files) in 0 seconds (0 seconds overall).



However, when I run mysql-zrm-scheduler from the ZRM box, this is the error that I get:

$ mysql-zrm-scheduler --backup-set xxx.xxx.29.87-backup --backup-level 0 --now
schedule:INFO: ZRM for MySQL Community Edition - version 2.0
Logging to /var/log/mysql-zrm/mysql-zrm-scheduler.log
backup:INFO: ZRM for MySQL Community Edition - version 2.0
xxx.xxx.29.87-backup:backup:INFO: START OF BACKUP
xxx.xxx.29.87-backup:backup:INFO: PHASE START: Initialization
xxx.xxx.29.87-backup:backup:INFO: Mail address: xxxxxxxx@xxx.xxx is ok
xxx.xxx.29.87-backup:backup:INFO: ZRM Temporary configuration file = /etc/mysql-zrm/xxx.xxx.29.87-backup/tmpYTluO.conf
xxx.xxx.29.87-backup:backup:INFO: {
xxx.xxx.29.87-backup:backup:INFO: quiet=0
xxx.xxx.29.87-backup:backup:INFO: verbose=1
xxx.xxx.29.87-backup:backup:INFO: backup-level=0
xxx.xxx.29.87-backup:backup:INFO: tmpdir=/tmp
xxx.xxx.29.87-backup:backup:INFO: mysql-binpath=/usr/bin
xxx.xxx.29.87-backup:backup:INFO: mailto=xxxxxxxx@xxx.xxx
xxx.xxx.29.87-backup:backup:INFO: socket-remote-port=25300
xxx.xxx.29.87-backup:backup:INFO: databases=AssignmentCalculator
xxx.xxx.29.87-backup:backup:INFO: remote-mysql-binpath=/usr/bin
xxx.xxx.29.87-backup:backup:INFO: host=xxx.xxx.29.87
xxx.xxx.29.87-backup:backup:INFO: backup-mode=raw
xxx.xxx.29.87-backup:backup:INFO: password=******
xxx.xxx.29.87-backup:backup:INFO: user=xxxx0021
xxx.xxx.29.87-backup:backup:INFO: mysql-binlog-path=/var/lib/mysql
xxx.xxx.29.87-backup:backup:INFO: copy-plugin=/usr/share/mysql-zrm/plugins/socket-copy.pl
xxx.xxx.29.87-backup:backup:INFO: }
xxx.xxx.29.87-backup:backup:INFO: Getting mysql variables
xxx.xxx.29.87-backup:backup:INFO: "/usr/bin"/mysqladmin --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" variables
xxx.xxx.29.87-backup:backup:INFO: datadir is /var/lib/mysql/
xxx.xxx.29.87-backup:backup:INFO: mysql_version is 5.0.51a-3ubuntu5.1-log
xxx.xxx.29.87-backup:backup:INFO: InnoDB data file are /var/lib/mysql/ibdata1
xxx.xxx.29.87-backup:backup:INFO: InnoDB log dir is /var/lib/mysql/.
xxx.xxx.29.87-backup:backup:INFO: backup set being used is xxx.xxx.29.87-backup
xxx.xxx.29.87-backup:backup:INFO: backup-set=xxx.xxx.29.87-backup
xxx.xxx.29.87-backup:backup:INFO: backup-date=20081024165100
xxx.xxx.29.87-backup:backup:INFO: mysql-server-os=Linux/Unix
xxx.xxx.29.87-backup:backup:INFO: host=xxx.xxx.29.87
xxx.xxx.29.87-backup:backup:INFO: backup-date-epoch=1224885060
xxx.xxx.29.87-backup:backup:INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 2.0
xxx.xxx.29.87-backup:backup:INFO: mysql-version=5.0.51a-3ubuntu5.1-log
xxx.xxx.29.87-backup:backup:INFO: backup-directory=/var/lib/mysql-zrm/xxx.xxx.29.87-backup/20081024165100
xxx.xxx.29.87-backup:backup:INFO: backup-level=0
xxx.xxx.29.87-backup:backup:INFO: backup-mode=raw
xxx.xxx.29.87-backup:backup:INFO: PHASE END: Initialization
xxx.xxx.29.87-backup:backup:INFO: PHASE START: Running pre backup plugin
xxx.xxx.29.87-backup:backup:INFO: Executing pre-backup-plugin
xxx.xxx.29.87-backup:backup:INFO: PHASE END: Running pre backup plugin
xxx.xxx.29.87-backup:backup:INFO: PHASE START: Flushing logs
xxx.xxx.29.87-backup:backup:INFO: Flushing the logs
xxx.xxx.29.87-backup:backup:INFO: "/usr/bin"/mysqladmin --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" flush-logs
xxx.xxx.29.87-backup:backup:INFO: Getting master logname using command "/usr/bin"/mysql --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" -e "show master status"
xxx.xxx.29.87-backup:backup:INFO: PHASE END: Flushing logs
xxx.xxx.29.87-backup:backup:INFO: PHASE START: Find table type
xxx.xxx.29.87-backup:backup:INFO: Command used for getting engine type "/usr/bin"/mysql --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" -e "show table status from \`AssignmentCalculator\`"
xxx.xxx.29.87-backup:backup:INFO: For database AssignmentCalculator
xxx.xxx.29.87-backup:backup:INFO: engine
xxx.xxx.29.87-backup:backup:INFO: MyISAM
xxx.xxx.29.87-backup:backup:INFO: MyISAM
xxx.xxx.29.87-backup:backup:INFO: MyISAM
xxx.xxx.29.87-backup:backup:INFO: MyISAM
xxx.xxx.29.87-backup:backup:INFO: PHASE END: Find table type
xxx.xxx.29.87-backup:backup:INFO: PHASE START: Creating raw backup
xxx.xxx.29.87-backup:backup:INFO: Command used for raw backup is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy=/usr/bin --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" --quiet AssignmentCalculator "/var/lib/mysql-zrm/xxx.xxx.29.87-backup/20081024165100" > /tmp/fxGv16HqY4 2>&1
xxx.xxx.29.87-backup:backup:ERROR: Output of command: 'mysqlhotcopy' is {
close of pipe failed
}
xxx.xxx.29.87-backup:backup:ERROR: mysqlhotcopy did not succeed. Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy=/usr/bin --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" --quiet AssignmentCalculator "/var/lib/mysql-zrm/xxx.xxx.29.87-backup/20081024165100" > /tmp/fxGv16HqY4 2>&1 Return value is 512
xxx.xxx.29.87-backup:backup:INFO: PHASE START: Cleanup
xxx.xxx.29.87-backup:backup:INFO: backup-status=Backup failed
xxx.xxx.29.87-backup:backup:INFO: Backup failed
xxx.xxx.29.87-backup:backup:INFO: mailing file /tmp/uBpXZ8dkDw
xxx.xxx.29.87-backup:backup:INFO: mail command is cat "/tmp/uBpXZ8dkDw"|mail -s "[ZRM for MySQL Report] ERROR during backup of backup-set xxx.xxx.29.87-backup" xxxxxxxx@xxx.xxx
xxx.xxx.29.87-backup:backup:INFO: PHASE END: Cleanup
xxx.xxx.29.87-backup:backup:INFO: END OF BACKUP
ERROR: /usr/bin/mysql-zrm did not finish successfully


The output from my /var/lib/mysql/mysql.log file for the following run:

081024 17:02:31 50 Connect xxxx0021@xxx.xxx.xxx.xxx on
50 Query show /*!40003 GLOBAL */ variables
50 Quit
51 Connect xxxx0021@xxx.xxx.xxx.xxx on
51 Refresh
/usr/sbin/mysqld, Version: 5.0.51a-3ubuntu5.1-log ((Ubuntu)). started with:
Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock
Time Id Command Argument
081024 17:02:31 51 Quit
52 Connect xxxx0021@xxx.xxx.xxx.xxx on
52 Query show master status
52 Quit
53 Connect xxxx0021@xxx.xxx.xxx.xxx on
53 Query show table status from `AssignmentCalculator`
53 Quit

Any ideas that would shed light to all this pain will be greatly appreciated...

paddy
October 24th, 2008, 04:00 PM
1. What version of ZRM are you using?

2. See if your problem is same as http://forums.zmanda.com/showpost.php?p=3394&postcount=8

3. Check to see if /tmp on MySQL server is running out of space.

Paddy

vanoden
March 23rd, 2009, 01:37 PM
Did you ever figure this out? I have the following error:

ERRORS:
- Output of command: 'mysqlhotcopy' is [
- close of pipe failed
]
- mysqlhotcopy did not succeed. Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy=/usr/bin --user=\"zmanda\" --password=\"*****\" --host=\"nodeB.mysql.internal.turbineweb\" --port=\"3306\" --quiet mysql \"/Zmanda/backup.sets/nodeE/20090323171033\" > /tmp/ueV4qeqqEF 2>&1 Return value is 512

I've got other databases working fine on other servers with the same configurations. Nothing in the log files, plenty of disk space, incremental backups work fine, full backup fails pretty much instantly.

-TC

kkg
March 23rd, 2009, 02:52 PM
Please run the followign command on the host xxx.xxx.29.87 and let us know the output.

/usr/bin/mysqlhotcopy --user="xxxx0021" --password="*****" --host="xxx.xxx.29.87" --quiet AssignmentCalculator /tmp/test

Please subsitiute the 'xxx' and "*****" with appropriate actual values before running it.

--kkg

kkg
March 23rd, 2009, 02:53 PM
Another thing, please check the log you find on the mysql server host /var/log/mysql-zrm/socket-server.log immediately after running the zrm backup command.

santhikishore
November 25th, 2009, 04:03 AM
Hi Guys

Even i am also having the same problem.

I keep getting this message ERROR: Output of command: 'mysqlhotcopy' is {
close of pipe failed
}

One more thing i found our is when i run the mysqlhotcopy command it tries to backup the db on the local server instead of remote server.

I have already wasted one complete week with this problem.

If you guys know the solution for this please let me know.

Regards

Santhi Kishore

santhikishore
November 25th, 2009, 05:34 AM
Hi Guys

A small update

Just now i found that mysqlhotcopy can only be run from the server where the database resides

http://dev.mysql.com/doc/refman/5.0/en/mysqlhotcopy.html

mysqlhotcopy is a Perl script that was originally written and contributed by Tim Bunce. It uses LOCK TABLES, FLUSH TABLES, and cp or scp to make a database backup quickly. It is the fastest way to make a backup of the database or single tables, but it can be run only on the same machine where the database directories are located.

As i installed Mysql server on my Zrm server & along with it mysqlhotcopy got installed which is getting fired & is looking for the DB to backup on the zrm server.

So right now i removed the mysql server from my ZRM Server.
But still getting the same error.

santhikishore
November 25th, 2009, 05:41 AM
Hi Guys

One more thing i forgot to mention is that i get this error when i select raw backups only.

Logical backups are fine with socket copy.

Regards

Santhi Kishore

zmanda_jacob
November 25th, 2009, 07:52 AM
Please make sure on the remote MySQL system you are trying to back up that you have the ZRM client installed and you have the MySQL client utilities(mysqlhotcopy for example) installed. mysqlhotcopy specifically has some prerequisites such as perl-DBD-MySQL that need to be installed for it to work correctly as well as perl 5.8.8 or higher. A simple test could be to login to the remote server and run the mysqlhotcopy command. If it returns an error, then that may give you an idea of what the problem is.

santhikishore
November 25th, 2009, 10:45 AM
Hi

I have all the pre-reqs installed on my server & i am able to run mysqlhotcopy on the server with out any issues.

Below is the output of the required rpms.

rpm -qa | grep perl
perl-String-CRC32-1.4-1.el4.rf
perl-Compress-Raw-Zlib-2.021-1.el4.rf
perl-IO-Compress-2.021-1.el4.rf
perl-PlRPC-0.2020-1.el4.rf
perl-5.8.5-49.el4
perl-Compress-Raw-Bzip2-2.021-1.el4.rf
perl-Filter-1.34-1.el4.rf
perl-Net-Daemon-0.43-1.el4.rf
perl-DBI-1.609-1.el4.rf
perl-DBD-MySQL-2.9004-3.1.centos4

rpm -qa | grep MySQL
MySQL-client-community-5.0.87-0.rhel4
MySQL-server-community-5.0.87-0.rhel4
MySQL-shared-compat-5.0.87-0.rhel4
MySQL-zrm-client-2.1.1-1
perl-DBD-MySQL-2.9004-3.1.centos4

But still when i try to run raw backup from my ZRM server it gives me the below output.

somedomain.com:backup:INFO: MyISAM
somedomain.com:backup:INFO: PHASE END: Find table type
somedomain.com:backup:INFO: PHASE START: Creating raw backup
somedomain.com:backup:INFO: Command used for raw backup is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="******" --password="*****" --host="somedomain.com" --port="3306" --quiet somedb "/var/lib/mysql-zrm/somedomain.com/20091125193132" > /tmp/z8vjYfhmcV 2>&1
somedomain.com:backup:ERROR: Output of command: 'mysqlhotcopy' is {
close of pipe failed
}
somedomain.com:backup:ERROR: mysqlhotcopy did not succeed. Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="*********" --password="*****" --host="somedomain.com" --port="3306" --quiet somedb "/var/lib/mysql-zrm/somedomain.com/20091125193132" > /tmp/z8vjYfhmcV 2>&1 Return value is 512
somedomain.com:backup:INFO: PHASE START: Cleanup
somedomain.com:backup:INFO: backup-status=Backup failed
somedomain.com:backup:INFO: Backup failed
somedomain.com:backup:INFO: mailing file /tmp/vjj2rf1RSU
somedomain.com:backup:INFO: mail command is cat "/tmp/vjj2rf1RSU"|mail -s "[ZRM for MySQL Report] ERROR during backup of backup-set somedomain.com" ************@yahoo.com
somedomain.com:backup:INFO: PHASE END: Cleanup
somedomain.com:backup:INFO: END OF BACKUP
ERROR: /usr/bin/mysql-zrm did not finish successfully


more /var/log/mysql-zrm/socket-server.log
Client started
TMP directory being used is /tmp
ERROR: mysqlhotcopy failed No such file or directory

Below is my mysql-zrm.conf

backup-level=0
backup-mode=raw
compress=1
compress-plugin=/bin/gzip
encrypt=1
encrypt-plugin="/usr/share/mysql-zrm/plugins/encrypt.pl"
#passfile="/etc/mysql-zrm/.passphrase"
destination=/var/lib/mysql-zrm
databases="somedb"
user="******"
password="**********"
host="somedomain.com"
port=3306
verbose=1
mailto="***********@yahoo.com"
copy-plugin=/usr/share/mysql-zrm/plugins/socket-copy.pl
#mysql-binpath="/usr/bin"
remote-mysql-binpath="/usr/bin"
socket-remote-port="25300"


I face this issue only with raw backups using socket plugin

Regards

Santhi Kishore

zmanda_jacob
November 25th, 2009, 12:32 PM
Where is the mysqlhotcopy command located on your system?

Also, does the UID and GID of the mysql user match between your ZRM server and MySQL server?

santhikishore
November 26th, 2009, 12:35 AM
Hi

Mysql Server

id mysql
uid=100(mysql) gid=101(mysql) groups=101(mysql)

which mysqlhotcopy
/usr/bin/mysqlhotcopy

Zmanda Server

id mysql
uid=100(mysql) gid=101(mysql) groups=101(mysql)

OS version of both Zmanda & mysql is CentOS release 4.8 (Final) 64 bit
Mysql Version on both servers is the same which is Server version: 5.0.87-community-log MySQL Community Edition (GPL)

As you see the uid & gid of the mysql matches between both zmanda server & mysql server
In Zamanda server there is no mysqlhotcopy as i did not install MySQL Server rpm which install mysqlhotcopy binary.I taught it is not needed in zamanda server.Please correct me if i am wrong.



Regards

Santhi Kishore

joudatfd
February 4th, 2010, 05:20 AM
hi, i have this problem too, but only when i don't use snapshot plugin
i have dumped mysqlhotcopy command to file for later execution, it dies on copying table , which is ok for sure. Maybe it hits openfiles limit during backup of all databases at once?

DBD::mysql::db do failed: File './katslu/vykon_h.MYD' not found (Errcode: 24) at /usr/bin/mysqlhotcopy line 456.

Libor

paddy
February 4th, 2010, 09:07 AM
This is an issue with mysqlhotcopy. It cannot handle lot of tables. Please see comments in http://dev.mysql.com/doc/refman/5.1/en/mysqlhotcopy.html See Jacob Rief's solution. You will have to modify ZRM code for this workaround.

Paddy

joudatfd
February 4th, 2010, 11:21 AM
This is an issue with mysqlhotcopy. It cannot handle lot of tables. Please see comments in http://dev.mysql.com/doc/refman/5.1/en/mysqlhotcopy.html See Jacob Rief's solution. You will have to modify ZRM code for this workaround.

Paddy

Hi,
thanks it works now, i was trying too increase file limit using "ulimit -n 10000" in script i used to simulate mysqhotcopty run.
But setting open_files_limit in my.cnf helped

Libor