Deduplication stats

robnicholson · Fri Sep 07, 2012 4:17 pm

Just "chucked" the trial version of v6 on a VM for evalation. The change to implement devices separate from targets took me a bit by surprise but I got the hang of it after a few minutes.

I'm particularly interested in the new deduplication feature so I've created a 1.5TB deduplication disk on an external USB3 hard disk, added a target and mounted it on our main file server (which is using StarWind v5.x) - all so easy!

I've kicked off a robocopy command to copy our file server across. It'll take several days to run but this is just a test.

How do I know what level/ration of deduplication we're getting at the end of the copy?

Cheers, Rob.

robnicholson · Sat Sep 08, 2012 9:09 am

Err, I've think I've done something wrong here. I've created a 1.5TB deduplicated disk (DDDiskEx1) and attached it to a target. I've then added that target to our main file server as the G: drive. I left robocopy running yesterday and so far it's copied 237GB (see sshot-1png). When I look at the size of the spdata file, I expected it to be somewhat smaller (due to some deuplication) but it's not - it's 1.12TB!! See sshot-2.png.

What am I doing wrong?

Cheers, Rob.

Sat Sep 08, 2012 11:07 am

Rob, what deduplication block size do you use? Did you enable data delete thing? Did you just copy the files or it was overwrite process?

robnicholson · Sun Sep 09, 2012 9:52 am

Hi Anton,

I used the default block size of 256k. No, didn't enable the delete checkbox as it said "experiemental". It was a pure copy using the robocopy /mir option.

This is a pure test so I can delete and re-recreate the disk device and target if needed.

Cheers, Rob.

Mon Sep 10, 2012 8:06 am

I see... Please use 4KB deduplication block and enable "re-claim used space" (delete) option and try again. I'll ask engineers to re-check big dedupe blocks for efficiency. Thanks!

robnicholson wrote:Hi Anton,

I used the default block size of 256k. No, didn't enable the delete checkbox as it said "experiemental". It was a pure copy using the robocopy /mir option.

This is a pure test so I can delete and re-recreate the disk device and target if needed.

Cheers, Rob.

robnicholson · Tue Sep 11, 2012 10:35 am

Hi Anton,

I've recreated the deduplication disk with a 4k block size and deletion turned on. When I went through this time, block size defaulted to "auto" (so I changed it to 4k) so I'm not sure why I ended up 256k. I'm pretty sure I would not have changed it myself from "auto" to 256k so that's a bit perplexing.

I assume that because it was set to 256k block size, even a 1k file was taking up 256k? If so, then this would explain why the virtual disk exploded in size.

Any a bit perplexing...

Anyway, let's see where this test goes.

Cheers, Rob.

robnicholson · Tue Sep 11, 2012 10:37 am

One other change request for you to raise. You cannot tell from the device properties window whether deletion is on or off. See attached screenshot. Needs "Deletion On/Off" adding under deduplication settings.

Cheers, Rob.

robnicholson · Tue Sep 11, 2012 10:44 am

Whilst I'm on the subject of little quirks, the first time I went through creating this deduplication disk and got to the bit where I specified the virtual disk location, the metadata location & name was blank. I'm afraid I didn't screenshot it at the time so all I can supply it when it worked the second time. I remember been very confused the first time as I wasn't sure what metadata was (I can guess), where to put it or what to call it.

Maybe this only happens on a new server build?

Cheers, Rob.

robnicholson · Tue Sep 11, 2012 11:43 am

BTW - what does StarWind do if you set the block size to auto? Does it vary the block size depending upon the data being sent to the dedupe disk?

Cheers, Rob.

Thu Sep 13, 2012 1:29 pm

Hi Rob,

Lets go through all of this one-by-one.

I've recreated the deduplication disk with a 4k block size and deletion turned on. When I went through this time, block size defaulted to "auto" (so I changed it to 4k) so I'm not sure why I ended up 256k. I'm pretty sure I would not have changed it myself from "auto" to 256k so that's a bit perplexing.

Have you re-created or created new one?

One other change request for you to raise. You cannot tell from the device properties window whether deletion is on or off. See attached screenshot. Needs "Deletion On/Off" adding under deduplication settings.

Agree. I think we will add this when deletion support will stop being experimental.

Whilst I'm on the subject of little quirks, the first time I went through creating this deduplication disk and got to the bit where I specified the virtual disk location, the metadata location & name was blank. I'm afraid I didn't screenshot it at the time so all I can supply it when it worked the second time. I remember been very confused the first time as I wasn't sure what metadata was (I can guess), where to put it or what to call it.

Maybe this only happens on a new server build?

Maybe, but we couldn`t recreate this in our tests lab. If you will catch this once again it would be great if we could saw the screenshots and dtailed description of waht was done to get this. But I hope that we will never hear about this again

BTW - what does StarWind do if you set the block size to auto? Does it vary the block size depending upon the data being sent to the dedupe disk?

It is using 256K by default

I hope it was helpful

robnicholson · Thu Sep 13, 2012 5:45 pm

Have you re-created or created new one?

Deleted old disk and created a new one with 4k block size, not auto. So far it's looking better. 232GB of disk space used by Windows but the spdata file is 199GB. So a deduplication ration of 1.16. Will let it finished the entire copy of 886GB source data now.

Rob.

Sat Sep 15, 2012 9:23 am

Please keep us updated. We'll provide a deduplication status with a minor update. Biggest issue here is - we don't know file system on top of our volume so if you had
copied 10TB to 1TB volume and then deleted whole content for us it would be still 1TB used as Windows does not do free space reclaim with writing zero pattern or whatever.

Any chance to use Windows 2012 share with enabled dedupe to compare dedupe ratios?

robnicholson wrote:
Have you re-created or created new one?
Deleted old disk and created a new one with 4k block size, not auto. So far it's looking better. 232GB of disk space used by Windows but the spdata file is 199GB. So a deduplication ration of 1.16. Will let it finished the entire copy of 886GB source data now.

Rob.

robnicholson · Mon Oct 01, 2012 10:05 am

Hiya - have been on holiday for two weeks so will follow-up later this week.

Regards, Rob.

Mon Oct 01, 2012 12:11 pm

Good for you

Please keep us updated.

robnicholson wrote:Hiya - have been on holiday for two weeks so will follow-up later this week.

Regards, Rob.

robnicholson · Tue Oct 02, 2012 9:53 am

Hi Anton - because of my holiday, the trial license is about to expire. Can I request another 30 days?

Cheers, Rob.