Invalid storage selected!

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
notwisttoit
Posts: 4
Joined: Thu Mar 15, 2012 3:52 pm

Thu Mar 15, 2012 4:03 pm

I have a dedupe drive that I created and was testing. The physical drive behind it filled up a couple times and always made more space to allow it to continue to grow and the drive would then come back online. After the physical disk completely ran out of space, the dedupe drive became a thick disk instead of a thin disk. I was able to bring it back online and had stopped writing data to it and wanted to pull the data off. A little while later the disk was offline and I would not bring it online within the management console. So I removed the disk from the configuration to try to re attach to the dedupe storage but when I try to reattach to the files I get the following error:
Invalid storage selected! Failed: unable to open storage (error 996)

While this was a test volume and most of the volume was test, we have some production data on this volume and would like to be able to retrieve this data if at all possible. I don't know what to do at this point because the software can't even mount the dedupe files.
User avatar
Vitalii (staff)
Staff
Posts: 44
Joined: Mon Jun 07, 2010 8:49 am

Thu Mar 15, 2012 4:51 pm

Could you send me the latest log file where the device mounting failed.
notwisttoit
Posts: 4
Joined: Thu Mar 15, 2012 3:52 pm

Thu Mar 15, 2012 5:00 pm

Here are the logs files one from yesterday and one from today.
Attachments
log files.zip
Log files
(9.1 KiB) Downloaded 333 times
User avatar
Aitor_Ibarra
Posts: 163
Joined: Wed Nov 05, 2008 1:22 pm
Location: London

Thu Mar 15, 2012 7:17 pm

Sounds to me like expected behaviour: if there's zero bytes free on the physical disk, Starwind should refuse to mount a dedupe disk held on it, as deduped disks can only grow, not shrink. Perhaps there should be an option to mount it read only?

It will be interesting to see what happens in V6 with HA dedupe when the physical disk on one (or both) nodes is full / nearly full.

Does the dedupe engine keep track of how many duplicates there are of each unique block? If it did, then this would help in disk full situations, as then blocks that have a quantity of zero could then be removed from the physcial disk, freeing up space for new data, and without having to get into parsing the filesystem held on the target.

Example: imagine that we have a target that has been provisioned as having a capacity of 40KB using 4KB blocks. And for this example, imagine that there are 26 possible blocks represented by A-Z. The physical disk has 80KB space - enough for 20 unique blocks.

Code: Select all



1) Our initiator fills the target with 10 Z blocks:

Target ZZZZZZZZZZ
Disk   Z-------------------
Map    Z:10

Starwind just writes one Z to the disk, and records that there are 10 duplicates of Z. We have 19 blocks free on our physical disk.


2) The initiator starts writing some random data to the target

Target ZZZZZZZZZZ  
Target ITISAPERIO
Disk   -ITSAPERO-----------
Map    Z:1 I:3 T:2 S:1 A:1 M:1

The new blocks I, T, S, A, P, R and D get written to the disk. Note that the count for Z has disappeared. This was achieved by decrementing the counter for Z each time a Z block was replaced by something else. When it reached zero, the Z block was deleted. We have 11 blocks free on our physical disk.

3) The initiator continues writing some random data to the target
  
Target DOFCIVILWA
Disk   -I--A---ODFCVLW-----
Map    I:2 D:1 F:1 C:1 V:1 L:1 W:1 A:1

11 blocks are free

4) The initiator continues channeling GL
Target RREBELSPAC
Disk   ----A------C-LWREBSP
Map    R:2 E:2 B:1 L:1 S:1 P:1 A:1 C:1

Still 11 blocks are free

If we didn't have this counting method coupled with deleting unused blocks, the physical disk would be looking like this:

1) Z------------------- 
2) ZITSAPERO----------- 
3) ZITSAPERODFCVLWA----
4) ZITSAPERODFCVLWABP--

only two free blocks; the next two unique blocks written would fill our physical disk.

It would seem to me that this method would work regardless of the filesystem, however the overhead in keeping a count for each unique block (starwind already has to store the hash and location) might be quite high. But at least this way there is no way for a dedupe target to run out of physical disk space unless it has been thinly provisioned.

As an additional trick: the blocks with the highest counts could be cached in RAM, and a middle tier could be cached on SSD.

Of course, I have no idea if Starwind is already doing this or something similar!
notwisttoit
Posts: 4
Joined: Thu Mar 15, 2012 3:52 pm

Thu Mar 15, 2012 7:41 pm

I do have drive space on the physical drive now but it was not so when the virtual disk went offline. I don't understand now why I can't mount the dedupe disk in the SAN software to present to a server to mount as a read-only drive.
User avatar
Vitalii (staff)
Staff
Posts: 44
Joined: Mon Jun 07, 2010 8:49 am

Fri Mar 16, 2012 5:16 pm

Could you send me the first megabyte of .spdata and .spbitmap files.
Also, please tell me the exact sizes of these files.
notwisttoit
Posts: 4
Joined: Thu Mar 15, 2012 3:52 pm

Fri Mar 16, 2012 7:03 pm

I have emailed the requested files into support@starwindsoftware.com.
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Mon Mar 19, 2012 4:19 pm

OK, we have received them and passed to Vitalii. He will review them and revert soonest.
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
bweeks
Posts: 5
Joined: Wed May 02, 2012 7:14 am

Thu May 24, 2012 7:16 am

Hi there

I would be interested if you have found a resoloution to this.
The underlying NTFS partition on my Starwind device filled up yesterday and the Starwind partition becaome unavailable to ESXi.
So far I have been unable to recover this. The the datastore is unavailable I am unable to browse it and move the virtual servers off.
I spoke to Starwind support yesterday (Anatoly) but the suggestions did not help.
I am going to extend the NTFS partition to see if that helps but I believe when this event happens somehow the VMFS partition becomes corrupt.

Regards

Barry
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Thu May 24, 2012 9:39 am

Actually I`ve got one more suggestion for you (I was just about to email it to you :) ) : can you just destroy the targets and after to re-create the device. After these actions it is possible that you will have possibility to Storage Motion.
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
bweeks
Posts: 5
Joined: Wed May 02, 2012 7:14 am

Thu May 24, 2012 1:41 pm

OK so I have got the array to the point where I am able to browse the datastore and as such remove the virtual machines.
I acheived it as follows:
  • I have a server with 6x300GB 15k SAS drives configured as a single RAID5 volume
    I installed windows on this and partitioned it with 50GB for the OS C Drive and the rest for the Starwind drive D Drive
    I create a dedupe StarWind disk device then connected to the Starwind array from a number of ESXi servers
    The underlying NTFS partition on the D drive ran out causing the Starwind array to become unresponsive to ESXi servers
    I converted the single underlying drive into a dynamic disk
    I shrank the C volume by 5GB
    I extended the D Drive (Starwind) by 5GB
    I rebooted the Starwind server
    I removed all references to the Starwind device from the ESXi servers
    I added the iSCSI target IP's to a single ESXi server and did a rescan
    The ESXi server saw it as a snap volume - esxcli storage vmfs snapshot list
    I mounted the volume using - esxcli storage vmfs snapshot mount -l "STARSAN02" where STARSAN2 was the volume name
    I then used viclient to connect to that single ESX machine not vcenter
    I was then able to storage vmotion the VM's off of that device
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Fri May 25, 2012 9:49 am

Well, that was the plan:)

All I can add to this is that in future release (coming in 1 month approximately) we will have mechanism that will monitor how much disk space is left on the SAN box for DD device to prevent issues like this.
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
NetWise
Posts: 7
Joined: Sat Feb 18, 2012 6:00 am

Wed Jun 13, 2012 2:42 pm

I have a similar situation. Details are:

C: and D: drive, two separate RAID5 sets.
StarWind DeDupe target on D:, filled up the physical drive even though the VMFS was not full.
I have no ability to extend the D: drive. I have:
* Stopped StarWind services
* taken the *.SPIBITMAP (54GB) and *.SPMETADATA (5GB) and moved them to C:.
* Removed the Target and Device.
* Am attempting to add a new device, specifying "Custom Storage Paths" to be able to point some at C: and some at D: (where the *.SPDATA lives).

However, this is where I get blocked with a dialog:

StarWind Managment Console:
Failed to get information from storage.
Failed: unable to open storage (error 5)

What might I be able to do here?
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Fri Jun 15, 2012 1:12 pm

Are you 100% sure that this file is not sued by any other process?
Could you provide us with the screenshot that contains device description of the corresponding target and the screenshot with the path to copy of the files?
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
Post Reply