PDA

View Full Version : compression weirdness



-stevem-
July 29th, 2008, 07:16 AM
Hello all,

I have got Zmanda working correctly on Solaris x86 ( sunfire x4140 fwiw ) using zfs snapshots with the coolstack implementation of MySQL , Daily full backups with hourly incrementals.

Everything cushty here...

mysql-zrm.conf :

mysql-binpath="/opt/coolstack/mysql/bin"
host="localhost"
databases="dash_production"
password="******"
user="root"
snapshot-plugin="/usr/share/mysql-zrm/plugins/zfs-snapshot.pl"
destination=/data/backup/dash_production

Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: START OF BACKUP
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Initialization
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-set=dashboard-backup
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-date=20080729130000
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: mysql-server-os=Linux/Unix
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: host=localhost
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-date-epoch=1217332800
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 2.0
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: mysql-version=5.1.25-rc-log
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-directory=/data/backup/dash_production/dashboard-backup/20080729130000
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-level=1
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Initialization
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Running pre backup plugin
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Running pre backup plugin
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Flushing logs
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Flushing logs
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Creating incremental backup
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: incremental=mysql-bin.[0-9]*
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Creating incremental backup
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Calculating backup size & checksums
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: next-binlog=mysql-bin.000051
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: last-backup=/data/backup/dash_production/dashboard-backup/20080729120000
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-size=0.00 MB
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Calculating backup size & checksums
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: read-locks-time=00:00:00
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: flush-logs-time=00:00:00
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-time=00:00:00
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: backup-status=Backup succeeded
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: Backup succeeded
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Running post backup plugin
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Running post backup plugin
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Mailing backup report
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Mailing backup report
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE START: Cleanup
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: PHASE END: Cleanup
Tue Jul 29 13:00:00 2008: dashboard-backup:backup:INFO: END OF BACKUP

Everything looks fine from here until I enable compressed backups, then for *FULL* backups only

Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE START: Initialization
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: backup-set=dashboard-backup
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: backup-date=20080729160001
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: mysql-server-os=Linux/Unix
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: host=localhost
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: backup-date-epoch=1217343601
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 2.0
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: mysql-version=5.1.25-rc-log
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: backup-directory=/data/backup/dash_production/dashboard-backup/20080729160001
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: backup-level=0
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: backup-mode=raw
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE END: Initialization
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE START: Running pre backup plugin
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE END: Running pre backup plugin
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE START: Flushing logs
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE END: Flushing logs
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: PHASE START: Creating snapshot based backup
Tue Jul 29 16:00:01 2008: dashboard-backup:backup:INFO: File Position Binlog_Do_DB Binlog_Ignore_DB
mysql-bin.000055 106
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: innodb-data=/data/mysql/data/ibdata2;/data/mysql/data/ibdata1;
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: innodb-logs=/logs/mysql/dash-live-db1//ib_logfile*
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: raw-databases-snapshot=dash_production
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: PHASE END: Creating snapshot based backup
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: PHASE START: Find table type
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: PHASE END: Find table type
Tue Jul 29 16:00:26 2008: dashboard-backup:backup:INFO: PHASE START: Calculating backup size & checksums
Tue Jul 29 16:00:27 2008: dashboard-backup:backup:INFO: next-binlog=mysql-bin.000055
Tue Jul 29 16:00:27 2008: dashboard-backup:backup:INFO: last-backup=/data/backup/dash_production/dashboard-backup/20080729150001
Tue Jul 29 16:00:27 2008: dashboard-backup:backup:INFO: backup-size=2.96 GB
Tue Jul 29 16:00:27 2008: dashboard-backup:backup:INFO: PHASE END: Calculating backup size & checksums
Tue Jul 29 16:00:27 2008: dashboard-backup:backup:INFO: PHASE START: Compression/Encryption
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: compress=/usr/bin/gzip
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: backup-size-compressed=2.98 MB
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/logs/mysql/dash-live-db1
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/logs/mysql
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/logs
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/data/mysql/data
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/data/mysql
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/data
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/dash_production
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: PHASE END: Compression/Encryption
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: read-locks-time=00:00:00
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: flush-logs-time=00:00:00
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: compress-encrypt-time=00:06:00
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: backup-time=00:01:02
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: backup-status=Backup done but with errors
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: Backup done but with errors
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: PHASE START: Running post backup plugin
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: PHASE END: Running post backup plugin
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:INFO: PHASE START: Mailing backup report
Tue Jul 29 16:01:04 2008: dashboard-backup:backup:INFO: PHASE END: Mailing backup report
Tue Jul 29 16:01:04 2008: dashboard-backup:backup:INFO: PHASE START: Cleanup
Tue Jul 29 16:01:04 2008: dashboard-backup:backup:INFO: PHASE END: Cleanup
Tue Jul 29 16:01:04 2008: dashboard-backup:backup:INFO: END OF BACKUP

Looking in the directory that the backup created ....

drwxr-xr-x 3 root root 3 Jul 29 16:00 logs
drwxr-xr-x 3 root root 3 Jul 29 16:00 data
drwx------ 2 mysql mysql 2 Jul 29 16:01 dash_production

since the zrm-scheduler created those files , I can't see how it's failing to delete them ...


any clues ?


TIA
Steve

kkg
July 31st, 2008, 10:41 PM
The data and log dirs seem to be owned by root. That is why ZRM is unable to delete these.

Is the datadir, the innodb shared data dir & log dir and its contents owned by root?

--kkg

zmanda_jacob
August 1st, 2008, 07:27 AM
Please make sure they are owned by "mysql" with the group "mysql" and not root.

-stevem-
August 3rd, 2008, 01:43 PM
I have created a complete new config , now the data and logs dir are being created as mysql:mysql

The innodb data , logs and datadir are all owned by mysql.

It's still failing on the deletion of the directories , they are however all empty, so the contents have been removed.

zmanda_jacob
August 4th, 2008, 08:16 AM
And if you su to the mysql user on this system and try to remove those directories, what are the results?

Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/logs/mysql/dash-live-db1
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/logs/mysql
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/logs
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/data/mysql/data
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/data/mysql
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/data
Tue Jul 29 16:01:03 2008: dashboard-backup:backup:ERROR: could not delete /data/backup/dash_production/dashboard-backup/20080729160001/dash_production

-stevem-
August 5th, 2008, 02:39 AM
Works perfectly.....

live-mysql@dash-live-db1[/data/backup/production/production/20080803213920] id
uid=102(mysql) gid=100(mysql) groups=100(mysql)
live-mysql@dash-live-db1[/data/backup/production/production/20080803213920] cd /data/backup/dash_production/dashboard-backup/20080729160001
live-mysql@dash-live-db1[/data/backup/dash_production/dashboard-backup/20080729160001] ls -l
total 3083
-rw-r--r-- 1 mysql mysql 3123700 Jul 29 16:01 backup-data
drwx------ 2 mysql mysql 2 Jul 29 16:01 dash_production
drwxr-xr-x 3 mysql mysql 3 Jul 29 16:00 data
-rw-r--r-- 1 mysql mysql 785 Jul 29 16:01 index
drwxr-xr-x 3 mysql mysql 3 Jul 29 16:00 logs
-rw-r--r-- 1 mysql mysql 184 Jul 29 16:01 zrm_checksum
live-mysql@dash-live-db1[/data/backup/dash_production/dashboard-backup/20080729160001] rm -R logs data
live-mysql@dash-live-db1[/data/backup/dash_production/dashboard-backup/20080729160001] ls -l
total 3080
-rw-r--r-- 1 mysql mysql 3123700 Jul 29 16:01 backup-data
drwx------ 2 mysql mysql 2 Jul 29 16:01 dash_production
-rw-r--r-- 1 mysql mysql 785 Jul 29 16:01 index
-rw-r--r-- 1 mysql mysql 184 Jul 29 16:01 zrm_checksum

sunil
August 5th, 2008, 04:34 AM
I remember that there seems to be a bug in version 2.0.

Do you have a trailing slash at the end for innodb data dir in my.cnf ? Can you please post the contents of my.cnf ?

-stevem-
August 5th, 2008, 04:49 AM
Looks like it could be a winner......

innodb_data_home_dir = /data/mysql/data/
innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
innodb_log_group_home_dir = /logs/mysql/dash-live-db1/


..... but not *quite* . Removing the trailing slash hash got the errors down to 1 now though....




Tue Aug 5 13:45:02 2008: production:backup:INFO: backup-size-compressed=0.10 MB
Tue Aug 5 13:45:02 2008: production:backup:ERROR: could not delete /data/backup/production/production/20080805134500/dash_production
Tue Aug 5 13:45:02 2008: production:backup:INFO: PHASE END: Compression/Encryption
Tue Aug 5 13:45:02 2008: production:backup:INFO: read-locks-time=00:00:00
Tue Aug 5 13:45:02 2008: production:backup:INFO: flush-logs-time=00:00:00
Tue Aug 5 13:45:02 2008: production:backup:INFO: compress-encrypt-time=00:00:10
Tue Aug 5 13:45:02 2008: production:backup:INFO: backup-time=00:00:02
Tue Aug 5 13:45:02 2008: production:backup:INFO: backup-status=Backup done but with errors
Tue Aug 5 13:45:02 2008: production:backup:INFO: Backup done but with errors


You can still rm -R the dash_production directory with the mysql user, no errors....

Steve

sunil
August 6th, 2008, 02:11 AM
Looks like it could be a winner......

innodb_data_home_dir = /data/mysql/data/
innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
innodb_log_group_home_dir = /logs/mysql/dash-live-db1/


..... but not *quite* . Removing the trailing slash hash got the errors down to 1 now though....




Tue Aug 5 13:45:02 2008: production:backup:INFO: backup-size-compressed=0.10 MB
Tue Aug 5 13:45:02 2008: production:backup:ERROR: could not delete /data/backup/production/production/20080805134500/dash_production
Tue Aug 5 13:45:02 2008: production:backup:INFO: PHASE END: Compression/Encryption
Tue Aug 5 13:45:02 2008: production:backup:INFO: read-locks-time=00:00:00
Tue Aug 5 13:45:02 2008: production:backup:INFO: flush-logs-time=00:00:00
Tue Aug 5 13:45:02 2008: production:backup:INFO: compress-encrypt-time=00:00:10
Tue Aug 5 13:45:02 2008: production:backup:INFO: backup-time=00:00:02
Tue Aug 5 13:45:02 2008: production:backup:INFO: backup-status=Backup done but with errors
Tue Aug 5 13:45:02 2008: production:backup:INFO: Backup done but with errors


You can still rm -R the dash_production directory with the mysql user, no errors....

Steve

Have you specified the trailing slash anywhere else ? Remove all the trailing slashes and then perform the backup.

-stevem-
August 6th, 2008, 03:38 AM
destination=/data/backup/dash_production

in the mysql-zrm.conf is the only place that it's defined, and as you can see that's never had a trailing slash...