PDA

View Full Version : ZRM remote Backup Problems



alvinc
January 12th, 2007, 07:22 AM
I've been having bizarre problems here and there with configuring remote backup for my mysql servers using ZRM. out of the 3 machines I've configured, each has had a problem... but I've been able to resolve 2 of the 3 setups.. So here's the last problem.

I am getting this error in the client.log of the mysql server.

Client started
TMP directory being used is /tmp
ERROR: mysqlhotcopy failed No such file or directory

The first time I got that, I went through the mysql-zrm.conf, socket-server.pl, and socket-copy.pl scripts and defined the directory where the mysqlhotcopy is locate (I did this with the previous servers, and it worked out fine). In my case the file is located at /usr/local/mysql/bin.

The problem still persists.. As a test I tried to run the mysqlhotcopy from the mysql server and it runs fine... so its not a problem there. the directory permissions are 755 so that should allow anyone to get in and execute..

The only think I can think of is that the mysql server is 64bit and the backup server is a 32bit machine... is there anything else you guys can think of... as well as any additional information that you want..?

thanks in advance..

shailen
January 12th, 2007, 08:49 AM
I've been having bizarre problems here and there with configuring remote backup for my mysql servers using ZRM. out of the 3 machines I've configured, each has had a problem... but I've been able to resolve 2 of the 3 setups.. So here's the last problem.

I am getting this error in the client.log of the mysql server.

Client started
TMP directory being used is /tmp
ERROR: mysqlhotcopy failed No such file or directory

The first time I got that, I went through the mysql-zrm.conf, socket-server.pl, and socket-copy.pl scripts and defined the directory where the mysqlhotcopy is locate (I did this with the previous servers, and it worked out fine). In my case the file is located at /usr/local/mysql/bin.

The problem still persists.. As a test I tried to run the mysqlhotcopy from the mysql server and it runs fine... so its not a problem there. the directory permissions are 755 so that should allow anyone to get in and execute..

The only think I can think of is that the mysql server is 64bit and the backup server is a 32bit machine... is there anything else you guys can think of... as well as any additional information that you want..?

thanks in advance..

Hi,

Can you copy/paste your mysql-zrm.log, you can paste the entire log file or the excerpt that shows the above error.

Thanks!
-Shailen

alvinc
January 12th, 2007, 08:57 AM
Thu Jan 11 21:30:51 2007: ERROR: mysqlhotcopy command did not succeed.
Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="zrmbackup" --password="*****" --host="xx.xx.xx.xx" --quiet mysql "/var/backups/pro/20070111213049" > /tmp/DjEFAvuAli 2>&1
Return value is 512
Thu Jan 11 21:30:51 2007: INFO: backup-status=Backup failed
Thu Jan 11 21:30:51 2007: INFO: Backup failed
Thu Jan 11 21:58:08 2007: INFO: backup-set=pro
Thu Jan 11 21:58:08 2007: INFO: backup-date=20070111215807
Thu Jan 11 21:58:08 2007: INFO: host=xx.xx.xx.xx
Thu Jan 11 21:58:08 2007: INFO: backup-date-epoch=1168570687
Thu Jan 11 21:58:08 2007: INFO: retention-policy=4W
Thu Jan 11 21:58:08 2007: INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 1.1.3
Thu Jan 11 21:58:08 2007: INFO: mysql-version=5.0.17-standard-log
Thu Jan 11 21:58:08 2007: INFO: backup-directory=/var/backups/pro/20070111215807
Thu Jan 11 21:58:08 2007: INFO: backup-level=0
Thu Jan 11 21:58:09 2007: ERROR: Output of command: 'mysqlhotcopy' is {

gzip: stdin: unexpected end of file
/bin/tar: Child returned status 1
/bin/tar: Error exit delayed from previous errors
close of pipe failed
}
Thu Jan 11 21:58:09 2007: ERROR: mysqlhotcopy command did not succeed.
Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="zrmbackup" --password="*****" --host="xx.xx.xx.xx" --quiet mysql "/var/backups/pro/20070111215807" > /tmp/4p5xwnsmCh 2>&1
Return value is 512
Thu Jan 11 21:58:09 2007: INFO: backup-status=Backup failed
Thu Jan 11 21:58:09 2007: INFO: Backup failed
Thu Jan 11 22:03:37 2007: INFO: backup-set=pro
Thu Jan 11 22:03:37 2007: INFO: backup-date=20070111220336
Thu Jan 11 22:03:37 2007: INFO: host=xx.xx.xx.xx
Thu Jan 11 22:03:37 2007: INFO: backup-date-epoch=1168571016
Thu Jan 11 22:03:37 2007: INFO: retention-policy=4W
Thu Jan 11 22:03:37 2007: INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 1.1.3
Thu Jan 11 22:03:37 2007: INFO: mysql-version=5.0.17-standard-log
Thu Jan 11 22:03:37 2007: INFO: backup-directory=/var/backups/pro/20070111220336
Thu Jan 11 22:03:37 2007: INFO: backup-level=0
Thu Jan 11 22:03:38 2007: ERROR: Output of command: 'mysqlhotcopy' is {

gzip: stdin: unexpected end of file
/bin/tar: Child returned status 1
/bin/tar: Error exit delayed from previous errors
close of pipe failed
}
Thu Jan 11 22:03:38 2007: ERROR: mysqlhotcopy command did not succeed.
Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="zrmbackup" --password="*****" --host="xx.xx.xx.xx" --quiet mysql "/var/backups/pro/20070111220336" > /tmp/***yhjEn2Q 2>&1
Return value is 512
Thu Jan 11 22:03:38 2007: INFO: backup-status=Backup failed
Thu Jan 11 22:03:38 2007: INFO: Backup failed

shailen
January 12th, 2007, 09:14 AM
can you please post the xinetd/inetd (depending on your OS) entries for mysql-zrm from

1) the mysql server that you are seeing the above errors when trying to protect with zrm.

2) the mysql server that you are protecting successfully with zrm.


Thanks!
-Shailen

alvinc
January 12th, 2007, 10:56 AM
Here's the failed MySQL xinetd.log:
07/1/12@14:49:39: START: mysql-zrm-socket-server pid=22411 from=xx.xx.xx.xx
07/1/12@14:49:42: EXIT: mysql-zrm-socket-server status=1 pid=22411 duration=3(sec)
07/1/12@14:50:22: START: mysql-zrm-socket-server pid=22419 from=xx.xx.xx.xx
07/1/12@14:50:23: EXIT: mysql-zrm-socket-server status=1 pid=22419 duration=1(sec)

Here's the working xinetd.log:

07/1/11@09:24:53: START: mysql-zrm-socket-server pid=21297 from=xx.xx.xx.xx
07/1/11@09:24:54: EXIT: mysql-zrm-socket-server status=0 pid=21297
07/1/12@14:11:11: START: mysql-zrm-socket-server pid=28441 from=xx.xx.xx.xx
07/1/12@14:11:12: EXIT: mysql-zrm-socket-server status=0 pid=28441 duration=1(sec)


Looks like the same thing... but let me know if you can think of anything else...

shailen
January 12th, 2007, 11:30 AM
I actually need these ones instead from both the working and non working server

/etc/xinetd.d/mysql-zrm-socket-client

Also, can you make sure xinetd is running on the server thats failing.

And lastly, could you also turn on the verbose option in your mysql-zrm.conf file just uncomment the entry for verbose=1

and then re run the same backup job thats failing and paste that output as well.

Thanks!
-shailen


Here's the failed MySQL xinetd.log:
07/1/12@14:49:39: START: mysql-zrm-socket-server pid=22411 from=10.151.96.18
07/1/12@14:49:42: EXIT: mysql-zrm-socket-server status=1 pid=22411 duration=3(sec)
07/1/12@14:50:22: START: mysql-zrm-socket-server pid=22419 from=10.151.96.18
07/1/12@14:50:23: EXIT: mysql-zrm-socket-server status=1 pid=22419 duration=1(sec)

Here's the working xinetd.log:

07/1/11@09:24:53: START: mysql-zrm-socket-server pid=21297 from=10.150.96.250
07/1/11@09:24:54: EXIT: mysql-zrm-socket-server status=0 pid=21297
07/1/12@14:11:11: START: mysql-zrm-socket-server pid=28441 from=10.150.96.250
07/1/12@14:11:12: EXIT: mysql-zrm-socket-server status=0 pid=28441 duration=1(sec)


Looks like the same thing... but let me know if you can think of anything else...

kkg
January 14th, 2007, 07:41 PM
Hi,

Are you using 1.1.3? How are you specifying when specifying where mysql binaries are installed in the remote machines?

In 1.1.3 you should not modify any of the scripts especially socket-server.pl.
The correct way to do this in 1.1.3 is to just specify "remote-mysql-binpath" in mysql-zrm.conf

For e.g.
remote-mysql-binpath=/usr/local/mysql/bin

Let me know if this helps.

--kkg

shailen
January 15th, 2007, 09:04 AM
also please be sure to make sure that xinetd is running on the server that is failing and you may want to compare the /etc/xinetd.d/mysql-zrm-socket-client on the server that is failing and make sure it matches the one on the server that is working....

thanks!
-shailen

alvinc
January 16th, 2007, 05:17 AM
Thanks for the responses. the Socket-server.pl was not changed.. that was the first thing that I noticed.. was that the working servers didn't need any modifications to the socket-server.pl... I did however change the socket-copy.pl to the path that I need... as well as add the
remote-mysql-binpath=/usr/local/mysql/bin

also.. the remote xinetd server is working.. I can telnet connect to 25300

I am using version 1.1.3

what else shall I do.? I'm gonna try again.. in a bit..

alvinc
January 16th, 2007, 05:25 AM
still the same error as before...

There is a successful connect from the backup server to the MySQL server... could it be because its a 64bit version?.... I don't think it would be... logging is at verbose already...

any ideas??



Logging to /var/log/mysql-zrm/mysql-zrm-scheduler.log
INFO: ZRM for MySQL Community Edition - version 1.1.3
INFO: backup-set=pro
INFO: backup-date=20070116091105
INFO: host=xx.xx.xx.xx
INFO: backup-date-epoch=1168956665
INFO: retention-policy=4W
INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 1.1.3
INFO: mysql-version=5.0.17-standard-log
INFO: backup-directory=/var/lbackups/pro/20070116091105
INFO: comment=Production MySQL backup
INFO: backup-level=0
ERROR: Output of command: 'mysqlhotcopy' is {

gzip: stdin: unexpected end of file
/bin/tar: Child returned status 1
/bin/tar: Error exit delayed from previous errors
close of pipe failed
}
ERROR: mysqlhotcopy command did not succeed.
Command used is /usr/share/mysql-zrm/plugins/socket-copy.pl --mysqlhotcopy --user="zrmbackup" --password="*****" --host="xx.xx.xx.xx" --quiet mysql "/var/lbackups/pro/20070116091105" > /tmp/ER6ys7BLeP 2>&1
Return value is 512
INFO: backup-status=Backup failed
INFO: Backup failed
ERROR: /usr/bin/mysql-zrm did not finish successfully
/usr/bin/mysql-zrm-reporter finished successfully

kkg
January 16th, 2007, 05:42 AM
Frankly I am quite mystified. The only think I can think of is that mysqlhotcopy is not in /usr/local/mysql/bin but that is also ruled out since you have clearly said that mysqlhotcopy is located in /usr/local/mysql/bin.

Could you add the following statement in socket-server.pl
&printLog( "$MYSQL_BINPATH/$MYSQLHOTCOPY $params $tmp_directory" );
in the function doHotCopy()

Then run mysql-zrm and let me know what /tmp/client.pl contains. This will at least let us know if MYSQL_BINPATH is being recieved correctly.

--kkg

alvinc
January 16th, 2007, 10:59 AM
Do you think it matters if the data is located in another directory...?

Other than the 32 vs 64bit differences between the working system and non working one,

the system that works... the whole mysql directory data/bin is in /var/mysql

in the non working system, the bin is /usr/local/mysql/bin and the data is /var/db/mysql/data

could it be that simple? I don't think so.. but I'm not sure either if your script assumes its in the same root as the bin..

hmm... just putting ideas out..

alvinc
January 16th, 2007, 11:15 AM
code added:

this is what client.log says now on the MySQL server

Client started
TMP directory being used is /tmp
/usr/local/mysql/bin/mysqlhotcopy --quiet --password="zrmpass" --user="zrmbackup" --host="xx.xx.xx.xx" mysql /tmp/KQnUbpCKERROR: mysqlhotcopy failed No such file or directory

I noticed 2 spaces after mysqlhotcopy... think that matters?

kkg
January 17th, 2007, 02:32 AM
Since the command being run on the machine is taking the path you have specified nd it is still giving the error, I am quite mystified!

Could you let me know if you get an error if you just run the following command

/usr/local/mysql/bin/mysqlhotcopy --quiet --password="zrmpass" --user="zrmbackup" --host="10.151.97.20" mysql /tmp

on the command line of that mysql server machine?

--kkg

kkg
January 17th, 2007, 02:36 AM
BTW could you also provide the contents of the /etc/xinetd.d/mysql-zrm-socket-client

--kkg

alvinc
January 17th, 2007, 03:40 AM
Well,

it looks like I found my issue... maybe...

/usr/local/mysql/bin/mysqlhotcopy --quiet --password="zrmpass" --user="zrmbackup" --host="xx.xx.xx.xx" mysql /tmp
DBI connect(';host=xx.xx.xx.xx;mysql_read_default_grou p=mysqlhotcopy','zrmbackup',...) failed: Access denied for user 'zrmbackup'@'iq-mysql-1.iq-colo.hodesiq.net' (using password: YES) at /usr/local/mysql/bin/mysqlhotcopy line 178

it looks like the whole time I neglected to run the command by hand to check it... the program iteself will run, that's what I checked but it looks like it had problems all along connecting... funny thing is that I have perl-DBI installed... as well as Perl-DBD-MySQL packages....

Since this is a 64bit machine I may be missing something else.... so I'll work on getting this to work.. and if I still get something later, I'll get you the rest of the info you requested...... fingers crossed

alvinc
January 17th, 2007, 05:43 AM
kkg,

it looks like the dns structure of the system I'm having problems with, is different to that of the machines that are working... I've added a user allowing that specific hostname to run in mysql. the command that you asked me to run is working... and I've tried it from my remote backup machine.


Thanks for the help... and I appreciate all the though you've put in to something that was on my end the whole time..

thanks..

sprbck
April 3rd, 2007, 09:16 AM
Hello all,

I lost some hours around this too. I think mysqlhotcopy should just try to connect to localhost or via socket, because it runs locally on the mysql server beeing backup up.

Instead, it gets the same arguments as if it was beeing run remotely!

So, i need to have the exact backup user permissions in mysql, one for the mysql server itself and another for the backup server.

This is not in the documentation.

paddy
April 11th, 2007, 04:38 PM
Hello all,

I lost some hours around this too. I think mysqlhotcopy should just try to connect to localhost or via socket, because it runs locally on the mysql server beeing backup up.

Instead, it gets the same arguments as if it was beeing run remotely!

So, i need to have the exact backup user permissions in mysql, one for the mysql server itself and another for the backup server.

This is not in the documentation.

I have updated the documentation. Please take a look.

http://mysqlbackup.zmanda.com/index.php/Backup_of_Remote_Servers#Conditions_for_Backing_Up _Remote_MySQL_Servers
http://mysqlbackup.zmanda.com/index.php/Pre-Installation#MySQL_Backup_User

Feel free to edit the documentation.

Thanks,
Paddy