In the comments on my post about the Dell PowerEdge T105 server [1] there is some discussion of the internal USB port (which allows the use of a USB flash device for booting which is connected inside the case).
This is a really nice feature of the Dell server and something that would be useful if it was in more machines. However I believe that it would be better to have flash storage with a SATA interface on the motherboard. The cost of medium size flash storage (eg 4G) in the USB format is not overly great, if soldered to the motherboard or connected to a daughter-board the incremental price for the server would be very small. Dell servers shipped with a minimum of an 80G SATA disk last time I checked, it seems quite likely to me that Dell could reduce the prices of their servers by providing flash storage on the motherboard and having no hard disk.
It seems likely to me that there is a significant number of people who don’t want the default hard drive that ships with a Dell server. The 80G disk that came with my PowerEdge is currently gathering dust on a shelf, it was far to small to be of any use in that machine and Dell’s prices for bigger disks were outrageous so I replaced the default disk with a pair of big disks as soon as the server had passed some basic burn-in tests. Most servers that I run fall into one of two categories, machines which primarily do computation tasks and need little storage space or IO capacity (in which case 4G of flash would do nicely) and machines which have databases, backups, virtual machine images, and other big things (in which case anything less than 160G is silly and less than 500G makes no economic sense in today’s market). Note that for a machine with small storage requirements I would rather have a 4G flash device than an 80G disk, I am inclined to trust flash to not die but not trust a single disk, two 80G disks means more noise, heat dissipation, and expense.
According to comments on my previous post VMWare ESX requires a USB boot device, so if VMWare could be booted with a motherboard based flash device then that would be an ideal configuration for VMWare. In some mailing list discussions I’ve seen concern raised about the reliability of permanently connected USB devices, while I’ve only encountered USB problems related to buggy hardware and drivers other people have had problems with the electrical connection. So it seems that motherboard based flash could be expected to increase the reliability of VMWare servers.
The down-side to having flash permanently attached to the motherboard is of course the impossibility of moving the boot device to different hardware. In terms of recovering from failure restoring a few gig of flash storage from backup is easy enough. The common debugging option of connecting a hard drive to another machine to fix boot problems would be missed, but I think that the positive aspects of this idea outweigh the negative – and it would of course be an option to not boot from flash.
If anyone knows of a tower server that is reasonably quiet, has ECC RAM and a usable amount of flash storage on the motherboard (2G would be a bare minimum, 4G or 8G would be preferred) then please let me know.
The Sun Fire X4540 server has a compact flash slot, which seems to me like a good solution. The X4540 is not quiet however.
A simple and cheap SATA/IDE->CompactFlash adapter should be sufficient, and those are transparent to the operating system.
The other downside of soldering a part to the motherboard is that it’s tying up capital as long as the motherboard is, and you have to own more of that part because you need some soldered to each kind of compatible motherboard you keep in inventory. If you assemble at the last minute as Dell does, that probably outweighs the extra cost of the connector and assembly time except for very cheap parts that almost everyone wants.
I haven’t read the ESX comment, but I can say that all of our ESX boxes boot from internal mirrored hard disks.
There is a variant called ESXi that is designed to be started from flash so that ESX hosts can be shipped without disks.
chitanh: Thanks for that information about the X4540, it sounds good for server rooms but not so good for my home.
I’ve used IDE->CF adapters, but had some issues. I’ve seen SATA flash storage devices which seem OK, but if you want both hard drives and flash then you lose a port.
Don: That sounds reasonable, combine that with the benefit of being able to install bigger/faster storage and it seems that a CF or SD socket on the motherboard would be a better option.
Paul: You are correct. I had missed the ‘i’ on the end of the name (which was in the comment that I referred to).
I am not sure I understand the context of your requirements.
But, if the server is not storage constrained (that is, you have a free SATA port), why not consider a SATA DOM (disk on module)? Search for Sata DOM or SATADOM.
I have seen 4GB for $100-250, but I am not sure what causes the price range.
Cheers,
JP
We’ve just received two Dell R805 servers with ESXi installed via an on-board SD card. There are to SD slots on a PCI riser, and I imagine you could install any OS on an appropriately-sized card.
I don’t know if the standard build includes the PCI riser with SD slots, but it would only take a quick call to Dell to confirm. I’m unable to recall the cost of having ESXi pre-installed; however, it was not very much as ESXi is now free. Worst-case scenario, you could order it with ESXi & then install the OS of your choosing on the SD card.
Regarding power and cooling, the boxes seem to run very cool & quiet, but we’ve not put them under great load yet. They are AMD-based & don’t use FB-DIMMs, so they are less power-hungry than Intel-based systems (in my experience).
I don’t think there is much harm to just labeling the USB Flash and leaving it connected to the rear of the machine at all times. If possible, try and set up a network bootable approach and have a USB Flash and CD-RW/CD-R as alternates. Then any internal drives are used purely as storage spools and you have a fairly reliable, and manageable system.
the internal usb connector is for VMware esxi, the OS image for vmware is roughly 30-60mb, and is intergrated into a lot of servers from hp/dell and i think IBM. i don’t think nearline storage as flash is very useful for servers hosting an OS like windows. It is much easier and safer to use raided disks in a physical server. how would you deal with RAID using flash? it would introduce a single point of failure that would be a no no for server builds. In a VMware environment it would not matter as the machines will auto failover to a new host if one went down due to a problem with flash.
there is still too short a lifespan for flash to be reliable enough to use as primary OS storage too.
plus, its better to use iscsi or a SAN to provide storage to servers these days. i think its time to move away from internal storage.
I’ve got an array of backup drives ( see this post for details: http://standalone-sysadmin.blogspot.com/2008/05/backup-scheme.html ) attached to a server for short-term point-in-time recovery. I’d be interested in hearing about anything you’ve seen regarding long-term USB storage.
http://www.cyberguys.com/templates/SearchDetail.asp?productID=24839
http://www.cyberguys.com/templates/SearchDetail.asp?productID=24505
The above URLs are for $15 devices that allow connecting a SD card to an IDE bus or a CF card to a SATA bus.
http://etbe.coker.com.au/2008/01/24/ide-dma-and-flash/
I’ve used a card similar to the IDE to SD one before and had some issues with it (documented at the above URL). Also performance was lower than I expected (although adequate for the purpose of the machine).
The CF to SATA device might be OK, but would use a precious SATA port.
Raymond: That’s interesting to know. At the moment I have no plans to buy Dell rack-mount servers so it’s only a matter of curiosity. If Dell starts doing such things in their tower servers then I’ll be interested (I expect to deploy a few of them in the near future).
Les: I have heard some bad reports about the reliability of USB for a long-term connection. Apparently USB is good for what it’s designed for (rugged external connections) but not so good if you want to replace Ethernet or anything else that has semi-permanent connections. Also I’m very concerned about the reliability of external parts, too many idiots gain access to server rooms.
Pants: Solid state storage has vastly different failure modes to moving storage, and thus different methods can be used to protect it. So if you were paranoid you could have the data stored twice on the same flash module (failure of an entire module is a lot less likely than failure of an entire hard disk).
Flash has been used for the BIOS on most motherboards for more than 10 years. It seems that motherboard failure is pretty rare, I’ve got a bunch of broken hard drives from machines I maintain, a heap of machines with broken PSUs, while motherboard problems are very rare. Of the small number of failed motherboards, most of them seem related to cracks from inserting expansion cards. Also for a lot of consumer electronics gear (mobile phones, PDAs, Netbooks, and probably TVs and VCRs) you have flash soldered on to the motherboard. The reliability of such devices seems at least equal to that of PCs.
Matt: Long term storage of USB attached hard disks should be pretty good. Long term storage on cheap USB flash devices is probably a bad idea. I don’t know of any good research on this matter, but I have noticed that on a quiet day at your local electronics store you can ask the staff which brands of USB device tend to be returned and which don’t. With some brands it seems common for them to lose data before the customer loses the receipt.