dedup file growth

[gstephenson]

Hi...

Why does a dedup disk continue to grow even when I delete its contents? I created a test dedup disk at 200MB, added some files until it was full, deleted all the files, then added more files in. If I keep doing this the physical size grows beyond 200MB. Is that supposed to happen?

Thanks.

[anton (staff)]

B/c you store new data and it's just added to database. From block device point of view there's no difference in writing set of 0xFF or 0xAA patterns to some sector - data is always considered new... We'll provide external app you'd run @ background to deal with known file systems and re-assign unused blocks as NTFS does not do it automatically.

Hi...

Why does a dedup disk continue to grow even when I delete its contents? I created a test dedup disk at 200MB, added some files until it was full, deleted all the files, then added more files in. If I keep doing this the physical size grows beyond 200MB. Is that supposed to happen?

Thanks.

[gstephenson]

B/c you store new data and it's just added to database. From block device point of view there's no difference in writing set of 0xFF or 0xAA patterns to some sector - data is always considered new... We'll provide external app you'd run @ background to deal with known file systems and re-assign unused blocks as NTFS does not do it automatically.

Ok, makes sense. So will this external app be available for download on the site soon?

[anton (staff)]

Yes, with V6 pre-release. Early February I think.

B/c you store new data and it's just added to database. From block device point of view there's no difference in writing set of 0xFF or 0xAA patterns to some sector - data is always considered new... We'll provide external app you'd run @ background to deal with known file systems and re-assign unused blocks as NTFS does not do it automatically.

Ok, makes sense. So will this external app be available for download on the site soon?

[Aitor_Ibarra]

Surely this would have to be run with the target offline? If you have spare disk space, especially on different set of spindles, it would make more sense to me to create a new de-dupe target and then copy the data across from the initiator side. How you do this would depend on the application, but you could probably do it more quickly and with less disruption. And without the worry that this tool won't understand the filesystem - in Windows land you've now got ReFS support to worry about too...

E.g if it's an Exchange mailstore, just create a new one on the new target and migrate the mailboxes. If it's a Hyper-V VM, use SCVMM to do a storage migration...

However if it can be done with the target online, even if it takes forever, it could be a very useful tool for people who don't have spare disks...

[anton (staff)]

No, you don't need to put anything off-line. We'll handle time consuming operations @ the background to save IOPS.

ReFS has the same "free space bitmap" concept as it's basically stripped down version of NTFS.

Other file systems (like VMFS) can deallocate free space automatically as they know how to tell to array particular block pattern is not used. We do understand them. NTFS is not that smart.

Surely this would have to be run with the target offline? If you have spare disk space, especially on different set of spindles, it would make more sense to me to create a new de-dupe target and then copy the data across from the initiator side. How you do this would depend on the application, but you could probably do it more quickly and with less disruption. And without the worry that this tool won't understand the filesystem - in Windows land you've now got ReFS support to worry about too...

E.g if it's an Exchange mailstore, just create a new one on the new target and migrate the mailboxes. If it's a Hyper-V VM, use SCVMM to do a storage migration...

However if it can be done with the target online, even if it takes forever, it could be a very useful tool for people who don't have spare disks...