PDA

View Full Version : Very slow backups (maybe large dirs to blame?) with Windows client



rkl
December 21st, 2012, 02:37 AM
By setting inparallel=1 and the same client options (fast compression but no encryption) on Windows VM vs. Linux VM clients both using the same SAN (and the same network setup), there is a massive difference in backup speed - between a factor of 2 and 20 depending on which Windows client I'm using - between the Windows Amanda client (very slow) and the Linux Amanda client (quite fast).

Because the Linux client runs at acceptable speeds and inparallel=1 to avoid any possible network or SAN contention, I don't think the network or SAN are to blame here. We are using proxmox+kvm+virtio for all the VMs and have just upgraded to the latest July 2012 virtio drivers on the Windows clients to rule out any performance issues from older versions we were running (it made no difference when we upgraded).

Hence, the spotlight is shining brightly on the Windows client and possibly NTFS too. We've even had to switch to "estimate server" because the default "estimate client" was actually timing out! If we start adding exclude directives to large directories (i.e. ones with 10,000+ files in them) on the Windows client, the backups at least don't timeout, but still take a very long time.

This suggests to me that either the Zmanda client or NTFS (or both) does not like traversing large directories in a time-efficient manner. Timeouts mainly hit us with incrementals (which could potentially have long periods of no data being sent to the server if a very large dir had no files to backup and took a long time to traverse).

Has anyone else seen very slow performance like this on Windows Amanda clients? I don't think the SAN is to blame here because of the decent Linux performance (it might be that ext3/4 is much better at traversing large dirs for example). Failing that, are there any tips on Amanda config options we could try to improve Windows client backup performance. Or are going to have to go through every Windows server and "clean them up" (or exclude all large dirs, which may not be possible if some of it needs to be backed up,)?

After having used Amanda with Linux clients for many years now and been happy with its performance, I was quite shocked to see Windows Amanda clients really struggle. I guess one option might be to try no compression on Windows clients and do the compression on the server side, but Linux clients with similar CPU allocation to Windows ones seem to have no trouble at all with the fast compression I'm using.

rkl
December 23rd, 2012, 04:11 AM
Although I didn't think CPU on the client was an issue, disabling client side compression on the clients and enabling on the server (which is dual quad core - we may even use pigz if that improves the speed further) seemed to help quite a bit. Now getting uncompressed read speeds on Windows clients of up to 15 Mbytes/sec, which whilst not great for a gigabit setup, is still a lot better than it was.

However, Windows clients remain significantly slower than Linux ones even after this speed improvement and some of our Windows clients are still reporting "error code 318" (that's another posting in this category around the same time as this) on incrementals, so the Windows Zmanda client and/or NTFS with large dirs are still under suspicion.