programming and human factors
by Jeff Atwood
In the last three years, I've gone from carrying a 512 MB USB memory stick to a 16 GB USB memory stick. That's pretty amazing.
According to the storagereview.com archives, hard drives with 16 GB of storage were introduced sometime around the beginning of 1999. Barely 10 years later, we carry around that much on our keychains. Heck, my laptop only has a 32 GB solid state drive, and I manage to scrape by with that. These things are essentially miniature hard drives. I'm starting to wonder why we don't just take our entire computing environment, operating system and all, along with us and boot it up on whatever computer we happen to encounter in the wild.
There is one big problem with this approach, however. USB flash drive performance, even for the best models, is a small fraction of typical hard drive performance.
Modern 2.5" hard drive performance looks something like this:
| HDD Sequential Read | 55 MB/sec |
| HDD Sequential Write | 55 MB/sec |
Mind you, those aren't particularly fantastic numbers, just typical ones. You can get significantly faster hard drives, such as the Western Digital Velociraptor I just bought. Storage Review described the Velociraptor thusly:
single-user scores .. blow away those of every other [hard drive]
I was immediately sold once I read that. It's not cheap, but you get what you pay for, and I'm firmly in the "hard drive performance matters" camp. To quote Ferris Bueller, it is so choice. If you have the means, I highly recommend picking one up.
But what about large USB flash drives? How do they compare to typical hard drive speeds, much less the awe-inspiring Velociraptor? X-Bit Labs recently reviewed three 32 GB USB flash drives:
| Sequential Read | Seqential Write | |
| 32 GB Corsair Flash Voyager | 22 MB/sec | 10 MB/sec |
| 32 GB OCZ Rally 2 | 30 MB/sec | 22 MB/sec |
| 32 GB Patriot Xporter XT | 31 MB/sec | 17 MB/sec |
Not bad by any means, but punishing next to typical hard drive throughput. If you really were booting and running your operating system entirely from a USB flash drive, you'd feel like you had stepped back in time five full years.
Even if you're only planning to use your USB flash drive for the mundane task of storing files, you should care deeply about read and write speeds. Throughput wasn't much of an issue when USB drives were "only" a gigabyte or two. But when we're talking about 8 GB, 16 GB or 32 GB of data, being limited to 10 MB/sec write speeds means 13, 26, and 52 minutes respectively to fill that flash drive up with data. Do you have that kind of time?
The OCZ Rally 2 flash drive appears to be the winner in the Xbit labs roundup, but does the 32 GB size really offer the best bang for your buck? I did a quick spot check of OCZ Rally 2 flash drive prices on Amazon:
| 2 GB | $17 | $8.50/GB |
| 4 GB | $26 | $6.50/GB |
| 8 GB | $37 | $4.62/GB |
| 16 GB | $78 | $4.88/GB |
| 32 GB | $136 | $4.25/GB |
Surprisingly, yes. The 32 GB OCZ Rally2 flash drive offers the best price per gigabyte of storage. I actually didn't do the math before I purchased, thinking that the 16 GB one would be a better deal. Now I wish I had! I just noticed there's a $20 rebate on the 32 GB model, too, which makes it an even more outstanding deal.
I wasn't sure which 16 GB flash drive would be faster -- the Corsair Flash Voyager, or the OCZ Rally 2. So I ended up purchasing both.
I formatted both of the drives with the default NTFS filesystem and did a bit of ad-hoc testing in Vista. You can find the raw ReadyBoost benchmark results if you know where to look in the event viewer.
| 16 GB Corsair Flash Voyager | 16 GB OCZ Rally 2 | |
| 3GB ISO Copy, Read | 26 MB/sec | 26 MB/sec |
| 3GB ISO Copy, Write | 9 MB/sec | 10 MB/sec |
| Readyboost Random Read | 6,426 KB/sec | 6,434 KB/sec |
| Readyboost Random Write | 3,292 KB/sec | 4,695 KB/sec |
In practice, the Rally is noticeably faster at writing, and a smidge faster at reading (not exposed here, but the chddspeed results confirm this). Not the results I expected based on reading the Xbit Labs review, but apparently there's quite a bit of variance for USB flash drives depending on the vagaries of manufacturing and what particular flash memory chips the manufacturer happened to be using that month.
The results are close enough that you may want to pick on ergonomics rather than performance. The Corsair drive has a chunky rubberized coating, which works well on a keychain (less jangly, more durable) but becomes annoying in a pocket or next to a narrow USB slot. I think I prefer the Rally's narrower metallic casing. Since it is a slightly better performer overall, the OCZ Rally 2 series gets my recommendation, if you're in the market.
One caution: if you do plan to use your USB flash drive to run applications or even operating systems, pay close attention to the random read and write speeds. Sequential throughput is a good overall baseline, but it's not the entire performance story. While typical portable storage usage does correlate well with sequential throughput, applications running on a USB flash drive are largely bounded by random access throughput.
You can never have enough storage on your keychain. Now if I can just figure out what else to put on mine...
I'm waiting for the prices to drop a bit
Story of SSD drives in a nutshell. :)
...
Well, yeah, and you can buy $2 wine that will get you just as drunk as the $50 kind, too.
So a $50 wine is worth it, but a $100 wine isn't? :)
Here's a good review of some SSD benchmarking (Samsung, OCZ) versus the Velociraptor:
http://www.anandtech.com/printarticle.aspx?i=3311
Looks like the SSD wins hands down in most cases.
Wikipedia:
"Like all flash memory devices, flash drives can sustain only a limited number of write and erase cycles before failure[20]. Mid-range flash drives under normal conditions will support several hundred thousand cycles[20], although write operations will gradually slow as the device ages. This should be a consideration when using a flash drive to run application software or an operating system."
And this http://en.wikipedia.org/wiki/Solid-state_drive is also a good read.
Mihai on June 3, 2008 2:12 AMI noticed this behavior on a 16 gig USB that I just got. Woe to the person to tries to put a secure storage file on it and use it real-time.
I guess the downside of the USB "standard" is that there are so many layers of abstraction, interrupts and polling that the time left to do actual heavy lifting kind of falls by the wayside. If Firewire was the "standard" you would see much higher speeds. I don't get super speed out of USB hard drives either.
But it is criminal to me that a solid state device can't beat something with as many moving parts as a hard drive.
Chris Chubb on June 3, 2008 3:03 AMAs you point out, the random access speeds are a real killer that can dwarf the transfer rate problems for some applications.
Copy a large file from your hard drive to flash and you see the performance penalty that you suggest. Copy 10,000 small files and it gets really, really ugly.
To give a concrete example, my students are able to run Eclipse off of jump drives without any real issue (single, large executable). Ruby on Rails (30,000+ small files) from a jump drive will run, if you are willing to put up with speed roughly on par with glaciers.
coderprof on June 3, 2008 3:48 AM"I've been bringing my OS around with me for about a year now. Puppy Linux on flash, with a supplemental CD-ROM booter for comps that won't boot straight to USB."
Why didn't I think of that?! I've been suffering from having to burn new CDs for bootable OS's for a while now.
[ICR] on June 3, 2008 3:51 AMSo much information about sizes, speeds, and cost, but little explanation as to WHY you need one of these things. The Internet is available in most places you visit, isn't it?
Josh on June 3, 2008 4:09 AMI just can't see the benifit of buying a small velociraptor hd.
Can just as well buy a Samsung Spinpoint 1tb for less then halve that price who will write/read as fast but is a bit slower for finding data.
Carra on June 3, 2008 4:13 AMCan just as well buy a Samsung Spinpoint 1tb for less then halve that price who will write/read as fast but is a bit slower for finding data
Well, yeah, and you can buy $2 wine that will get you just as drunk as the $50 kind, too.
Furthermore, why pick one over the other? Why not have *both*? I prefer to have 1) a very fast boot drive, and 2) a large data drive. Two spindles confers other advantages as well.
I see my "very fast boot drive" migrating to SSD once they get speeds up and prices down a bit. I think even a modest 64 GB would be fine for me, as long as it's crazy fast and not too comically expensive.
Jeff Atwood on June 3, 2008 4:24 AMI have a 4GB Rally 2, and I'm really happy with it (though I might consider upgrading it to 16 or 32GB). However, I think you should have compared it with the Voyager GT instead of the Voyager.
I'm not 100% sure, but I think the Voyager GT is faster than Rally 2.
Can Berk Gder on June 3, 2008 4:26 AMYour write times are off. Writing a 32GB drive should take twice as long as writing a 16GB drive, which would put it around 52 minutes, assuming 13 and 26 were right.
Mark Tiefenbruck on June 3, 2008 4:35 AMFirst off, nice blog. I check into it every day. I find it entertaining and educational to read.
I wanted to point out something quite weird and frustrating that I have noticed, which would be nice if you checked out into, since you have just mentioned USBs.
I have noticed this quite some time ago, many many months ago. So, you copy a file X onto a USB, and it takes for example 5 minutes. Once it finishes, you copy file B, and it takes maybe 10 minutes. So altogether you get 15 minutes of transfers.
Now situation 2. You start copying file A onto a USB. It tells you 5 minutes left. So you try to copy file B at the same time... Suddenly it goes up to hours. Why is this? Why does USB handle parallel tasks so horribly?
Is it in the architecture?
Is it only me?
Is it only on Windows?
What is going on here?
I managed to replicate this behaviour on Windows Vista, Windows XP, and Ubuntu. I also tried various USB sticks. Is it just that all my sticks are cheap, or can ayone else confirm that this behavious is common to them too please?
Thanks.
With, all these ready boost flash drives, i wonder how fast my VS 2008 builds with non-boosted setup. Is there any tool where i can monitor the performace with my eyes ?
Mehfuz Hossain on June 3, 2008 4:42 AMBeware of cheap flash drives. I get them from vendors from time to time and I've seen a few die after only a few weeks of light-duty use.
Always always always back up anything stored on a flash drive to more durable media.
Dana on June 3, 2008 4:46 AMIf you have the money (or time to wait the price out), then nothing beats *quality* SSDs. 2nd generation is not slow anymore, you don't need to wait for the speed part. With ~100MB/s, it already beats 7200rpm disks and "seek" times are fantastic:
http://www.tomshardware.com/reviews/ssd-memoright,1926.html
I'm waiting for the prices to drop a bit, then it's a pair of SSDs in RAID1 (for better read performance) for all performance-critical stuff (the system itself, VMware images, source code and build dirs), to get rid of these annoying slowdowns when large I/O buffers are sync()ed to disk. That and a pair of Spinpoints (I wouldn't believe Samsung can make good drivers, but they do, and I *can't hear them seeking*) in RAID1 for less-frequently needed data.
Peter on June 3, 2008 4:47 AMDid you get paid by the USB companies to review/push their products?
ilan on June 3, 2008 4:52 AM"I'm starting to wonder why we don't just take our entire computing environment, operating system and all, along with us and boot it up on whatever computer we happen to encounter in the wild."
I already do this ... with an small OS that loads into memory and does as little disk I/O as possible ...
Hard drives are slow this is why Virtual memory should be avoided and why adding more memory speeds up most PC's
My experience with USB sticks suggests that not all USB2 ports are equal either ...
Jaster on June 3, 2008 5:00 AMThe more you fit on your USB drive the more there is too loose when you loose your keys (if you fail to secure the content).
I herd of a case where someone had their CV on their USB pen drive attached to their keys. This meant who ever found the keys also had the persons address!
John on June 3, 2008 5:08 AMI'm waiting for the prices to drop a bit
Story of SSD drives in a nutshell. :) Yes, I expect SSD drives to dominate the laptop space over the next 2-3 years, and certainly the performance/enthusiast market that the WD Raptors currently occupy.
Writing a 32GB drive should take twice as long as writing a 16GB drive, which would put it around 52 minutes
You're absolutely right -- I corrected this error.
So you try to copy file B at the same time
This causes write collisions and contention for both copy processes, so it's not surprising (to me, at least) that the cumulative copy time would be much worse.
With, all these ready boost flash drives
ReadyBoost isn't worth the effort, honestly. Particularly with system memory being ridiculously dirt cheap now. It may help somewhat if you're on a slow system with 1 GB of memory and zero possibility of any memory upgrade, but it's fairly marginal. Upgrade your system memory.
Jeff Atwood on June 3, 2008 5:10 AMMemory sticks are one of the few areas of computer technology that rarely impress me. Most of my work and backups reside in folders containing thousands of files. Zipping/unzipping just to write/read to a memory stick is a real pain in the ass extra step I would think we could have gotten past by now.
PaulG. on June 3, 2008 5:24 AMI just bought a 16 gig corsair to use as dedicated mp3 storage for my car stereo - and you're right, it's noticeably slower then my 4 gig corsair gt (durability and speed have always sold me as far as corsair goes)
matt on June 3, 2008 5:24 AMwhile the speeds of pen drives are less than that of HDDs, why ppl say that pen drive ready boost is faster than hard disk cache?
On an unrelated note:
Your captcha is always orange(as far as i've seen. for about more than a month.) how is that spammers didn't break that yet?
Also it's very important to realize that some drives are better at reading/writing large files while others are better at reading/writing a lot of small files. Most of my files are code so they are small so I went with a drive that was better at small files (Super Talent 4GB).
Mitur Binesderty on June 3, 2008 5:28 AMThe "32 GB Corsair Flash Voyager" link is broken.
I'm not in the hard drive performance matters camp. Hard drive performance is close to irrelevant. I already started some of our products from USB sticks that we on a USB 1.1 hub, so maximum throughput is 12 MBit/s and still, it takes only a couple of seconds to bring them up. If an application depends extremly on HD perfomance to oprate at good speed - it's broken by design, like so often.
E.g. AdiumX, a chat client for MacOS X (supporting AIM, ICQ, Jabber, MSN, Yahoo, Google and many other) saves preferences frequently and this had a negative impact on performance if your prefs got very big. Now you can buy a faster HD to minizie this effect. But what the coders did was instead saving prefs asynchronously in an extra thread in the background and even with the slowest HD, you won't notice anymore that it saves at all.
Even big games can hardly load more texture data from HD than you have memory (RAM) or v-memory (video memory). And if you look at their mimimum memory requirements, you'll see that this is not a lot. Even if you have more memory, they won't load a lot more data than the minimum memory requirements. And if a slow HD means big loading time for games, the game is badly programmed. Instead of loading all at once e.g. it could load dynamically in the background, making the game start very soon and then keep loading while you are already playing. And it could cache more data. Between two levels, not all textures, modells, sounds and so on change. Instead of reloading everything for every level, it should better keep already loaded material in memory.
Saying you need a faster HD for a faster app is like programming an app with horrible performance and relying on a faster CPU to one day fix that. That's not the way to go. There are so little operations where throughput really matters. E.g. if you copy a 4 GB video file from driva A to drive B, that is a case where throughput really matters. However, if you access 10,000 small files (each just a couple of KB), throughput will be almost irrelevant compared to seek time. And here even good HDs only offer about 12 ms, while flash offers seek times in the ns range (milliseconds compared to nanoseconds), as they are usually below 1 ms. Actually, even with only 1/3 of the throughput, in some cases flash memory wins because of the little seek time.
It also depends a lot on OS. On Windows apps typically come with one or a couple of very large resource files (containing tons of data put together in one place). This is because the files are lying around visibly to the user and it just looks cleaner to have little files lying around. Unless the file is very fragmented, throughput might be important here. However, on Mac OS X, apps come in bundles. A bundle is a directory, but doesn't look like one to the user (without going over the context menu or using terminal, there is no way to look into that bundle, it just looks like a single file to the user). Inside the bundle, every piece of data is usually an own file. Some bundles consist of 30,000 files inside. The user won't see the mess anyway. Here seek time can be the important factor, not throughput. And I like the Apple way a lot more. Why? Well, if you just have very little huge files, you need to make sure you can seek within these files at reasonable speed. Finding data, jumping to it and loading it will become your task. On Mac OS X these are individual files. Finding them and jumping to them and loading them is task of the OS (and Apple can speed it up by improving the file system or offering better ATA drivers, and so on). So as developer you will gain performance from other people's work, while in the Windows case MS optizimations will buy you little, as they don't know how you seek within your proprietary archive.
Mecki on June 3, 2008 5:43 AMI do not understand how flash drives can suck when it comes to random access. I thought that this was part of their beauty. The fact that they are memories and not hard-disks, so they don't have to wait for disks to spin and heads to move. Maybe someone can clarify this for me.
Cristian on June 3, 2008 5:46 AMThinking aloud...how about partitioning in a few blocks that aligns with the USB drives flash memories? Then RAID the "disks" for Raid5?
Edward on June 3, 2008 5:47 AMDo not underestimate the special write-characteristics of flash memory - they indeed wear out, it's physically inevitable. With large SSDs it might not be a problem as long as there is a lot of free space left, so the controller can evenly distribute writes over the whole thing. But on fuller "disks" - especially with certain access patterns like write intensive databases - this may cause drop outs much earlier.
I recently learned that DB2e (IBMs database for embedded devices) is "flash-aware", i. e. it does not just flush to disk in the regular "fire-and-forget" way, but tries to accumulate write units that resemble the flash structure and also distributes write accesses inside its data files.
For storage and carrying around stuff, flash pens are really great, however I would not trust them for everyday work.
"I'm starting to wonder why we don't just take our entire computing environment, operating system and all, along with us and boot it up on whatever computer we happen to encounter in the wild"
In fact you can already buy USB sticks with Linux and various apps preinstalled (http://linos.dk/index.php?pid=6 - sorry in Danish, staring at 50€ for 2Gb up to ~85€ for the 8Gb model)
lars hundertwasser on June 3, 2008 5:50 AMGoogle translated in case anyone is interested: http://google.com/translate?u=http%3A%2F%2Flinos.dk%2Findex.php%3Fpid%3D6hl=enie=UTF8sl=datl=en
lars hundertwasser on June 3, 2008 5:53 AMakilan Said:
"Your captcha is always orange(as far as i've seen. for about more than a month.) how is that spammers didn't break that yet?"
It must be the computer your on, or a co-incidence. I'm on Coding Horror often here at the office, and i get a range of things from "apple" to "spammersNO".
Or not.... But I guess it just proves that some anti-spam utility is a lot better than nothing at all.
Shannon on June 3, 2008 5:56 AMI have had Ubuntu Linux installed on a WD Passport that I do plug into use on whatever computer I am using. I had a flash drive I used this way, but performance wasn't that great I had the passport, so why not? It works out pretty well for me.
Dennis on June 3, 2008 5:59 AM@akilan, Jeff has briefly mentioned this is a previous post about CAPTCHA. Basically, it's only used on this site. Go to the following article (CAPTCHA is Dead, Long Live CAPTCHA!), search for orange and you'll be looking at the relevant paragraph:
http://www.codinghorror.com/blog/archives/001067.html
--- Your captcha is always orange(as far as i've seen. for about more than a month.) how is that spammers didn't break that yet?
akilan on June 3, 2008 04:26 AM
"CAPTCHA defeating tools are tailored to very specific inputs; if there's little to no monetary incentive, odds are nobody will bother to customize one for yours. My ridiculously simple "orange" comment form protection is ample evidence of that."
http://www.codinghorror.com/blog/archives/001067.html
:)
Mark on June 3, 2008 6:07 AMI think I'll start carrying a slimmed down version of Linux or something on a USB drive. It gives me a reason to buy one if nothing else. Definitely use encryption on all semi-private data you put on a flash drive though, and preferably include an application to decrypt it, for a few OSes. I can see why people don't bother to though. I wouldn't be too worried about someone getting hold of my CV though, as mentioned above. Lots of people make their CV available on a website about themselves, and if the CV doesn't contain the address, then the domain reigstration details sometimes does. The best way to prevent privacy problems is to not be overly private in my mind.
Grogs on June 3, 2008 6:14 AMBring on the phase change solid state disk could be 500 times faster than current drives.
pete on June 3, 2008 6:29 AM"Well, yeah, and you can buy $2 wine that will get you just as drunk as the $50 kind, too."
Wine prices may not be the best comparison in this case, as a study shows that perception of taste is influenced by knowledge of the price of the wine:
http://www.iht.com/articles/2008/01/15/america/wine.php
However...
"On the other hand, when tasters did not know any price comparisons, they rated the $5 wine as better than any of the others sampled."
Matt Wigdahl on June 3, 2008 6:33 AMThe Corsair's body is rubber. The plastic ones break. For me, longevity is at least as important as xfer speeds.
Maro on June 3, 2008 6:33 AMIt is amusing consumers have been stuck with 10k rpm hard drives for so long and actually think they are fast. I bought a 15k rpm Seagate Cheetah hard drive years ago.
a on June 3, 2008 6:35 AMGrogs - I think the concern was that if the CV on the keychain had the owner's address, then the finder of a lost keychain would know which house and car the keys went with.
Jim on June 3, 2008 6:42 AMThere are already SSDs with a sustained write performance in the 50-100MB/s range, however these are significantly more expensive than comparable hard drives. Expect prices to come down dramatically within 25 months. Word of caution: In real-world tests, these SSDs do not (yet) conserve power compared to hard drives. So "all" you is get far better random seek performance, totally quiet operation and maximum shock resistance for a hefty price tag...
Alexander on June 3, 2008 6:50 AMCan someone explain to me, with these performance characteristics, why there was so much of a big deal with Vista's ability to use a USB flash drive as a cache?
The latency is lower, right? Or something? The numbers are consistent with my experience with flash drives, so even though I know the new tech will be capable of some good number.. .I'm really confused as to why that's a useful feature today.
Mike on June 3, 2008 6:55 AMBased on your data, I can compare sequential read/write.
But what is the typical random read/write for HDDs?
Jonas on June 3, 2008 7:06 AM@Mike, because most people don't know about the speeds of flash drives so Microsfoft were able to say. "Use your flash drive to add more cache". They didn't say "But it's very little difference"
Tony on June 3, 2008 7:07 AMI posted a link to this blog on Clarkhoward.com technology board:
If you've been shopping for a USB drive (aka, thumb drive, flash drive) check out Jeff Atwood's blog at http://www.codinghorror.com/blog/archives/001127.html. "Coding Horror" is mostly a programming blog, but every now and then, Jeff Atwood's his passion for geekery intersects with "bang-for-the-buck"
I've actually heard somewhere that its not worth formatting your usb drive to NTFS and that you should keep it as FAT32 because it recovers better when you pull it from the usb slot. Also did you try your tests on a FAT32 system just to see if there's a difference?
Donny on June 3, 2008 7:26 AMMTRON is scheduled to release an even faster generation of SSDs on June 20. The MSP 7500 series features read speeds of 130MB/sec and write speeds of 120MB/sec, with a maximum capacity of 64gb for the 2.5" form factor and 128gb for the 3.5" form factor. That's a nice boost from the MSP 7000 that Jeremy linked above (rated at 120MB/sec read and 90MB/sec write, maxing out at 32gb in the 2.5" form factor).
I could just barely restrict myself to a 64gb mobile drive, but I'm not willing to cough up $1,872.00. When the 128gb drives are selling for about half that price, I'll finally succumb to the temptation :).
Mike Strobel on June 3, 2008 7:30 AM"I'm starting to wonder why we don't just take our entire computing environment, operating system and all, along with us and boot it up on whatever computer we happen to encounter in the wild."
Some people do:
a href="http://docs.fedoraproject.org/release-notes/f9/en_US/sn-Live.html#sn-USB-Booting"http://docs.fedoraproject.org/release-notes/f9/en_US/sn-Live.html#sn-USB-Booting/a
JML on June 3, 2008 7:37 AMIt seem that Windows 7 will be able to boot up from .vhd image - so that you could have all your applications and data in your pocket and boot from that on any windows 7 pc: http://www.istartedsomething.com/20080523/windows-7-native-support-virtual-hard-disks/
And completely unrelated: a benchmark of various filesystems on same flash drive would be nice to have, as a performance of a high-end flash drive (A-data PD7, 25MB/s) when copying a lot of small files on FAT32 is downright abysmal (hundreds kb/s or even less).
I have an aluminium-case, (physically) bulletproof 16GB corsair. I prefer FAT32 filesystem, because then I can boot an operating system from it. Although it really is slow compared to containing a 200MB lightweight linux.
PalotsB. on June 3, 2008 8:00 AM@Reuben, the difference lies in how you're writing to disk.
In the first example, you're writing sequentially each file to disk.
In the second example, you're writing randomly to disk. It has nothing to do with parallel processing.
The comments about random access times on a flash drive seem a little bit off.
A flash drive should have exactly the same random access read time as sequential access. There's no spinning disk and no time difference no matter where you read. If you see any speed up with sequential access I expect it's from the operating system reading ahead and caching results, a common hard disk optimization.
Random writes on a flash drive will be terrible. Every time you change a byte the following happens:
The block of flash is read into a cache
The block of flash is erased
The block of flash is written, with your new byte
You can imagine that the above process is slow and most applications don't try to optimize around this. Flash devices also have limited life times dictated by the number of times a block can be erased.
Nearly all modern flash drives have a translation layer between the data on the disk and the FAT file system. These try and spread the write load across all the flash blocks. So your sequential reads aren't and the exact effect of writing becomes commercial black magic.
lod on June 3, 2008 8:30 AMInitially I always thought 10MB read speed was very good. Until I saw a review of Corsair, OCZ and Xporter 16GB drive, I straight away bought a Corsair 8GB.
I've tested my Corsair with SLAX, works pretty well. I can't get my hand on OCZ, so can't say anything about it. Compare that to Kingston 1GB that I used to like, Kingston booted into SLAX in 2:30min, while Corsair booted in 1:45min. That's booted into GUI. Not bad actually. I think SLAX's architecture also contribute to the faster bootup time compare to other live linux.
faulty on June 3, 2008 8:45 AMA word of caution about the corsair flash drives. I purchased an 8 gig (I think) drive about a year ago from them. I assumed that the rubber surround would provide some durability. In a matter of a week the usb end of the memory drive broke off. I RMA'd it to newegg and my new one proceeded to do the same thing! I gave up after that as I didn't want to pay shipping to get another defective drive.
They might have fixed the problem by now, but searching through the newegg product reviews for "broke" shows that it still plagues them.
Will Shaver on June 3, 2008 8:53 AMNow that I've dipped my toes into the world of embedded programming, I am struggling to deal with the special problems of flash storage. We have to be careful not to write to flash too often, since we get only a limited number of writes over the life of the device. And we don't dare write on the main application thread, since the write operation can block for tens of seconds.
I'm in no hurry to give up traditional hard drives.
Dave C. on June 3, 2008 9:09 AMI have looked over many numbers relating to flash drive speeds compared to hard drive speeds. On paper, the flash drives are obviously going to be slower, however, the numbers are now in the same ball park so it was time to see how well they actually performed.
All of our development environments (Windows / Visual Studio) run on local virtual machines using mapped drives for storage. The main reasons for this is that it's very easy to setup a new developer and you can "blow" the machine and revert to a fresh installation any time it starts to feel "sluggish".
The majority of developers run these virtual machines on either their primary hdd, or a second internal hdd. The performance differences here is very minimal, as long as you have the memory.
I have also had seen good results in using external USB hard drives. Whilst there is a very minor performance decrease the benefits of being able to pick up your own development enviroment and run it from anywhere far outweigh them.
With USB flash drives at 16 and 32GB now the next step was to try running the virtual hard drives directly from your keychain. The first tests were using the 16GB Corsair Flash Voyager, for performance, price and availability it seemed like a reasonable choice. Unfortunately the reality of it was far from acceptable. The virtual machines ran so badly they were far beyond usable, even just using the drives to transfer the virtual disks from one machine to another too slow to justify.
In conclusion, whilst having 16GB and 32GB flash drives lying around is fantastic for keeping data with you, the access times just aren't anywhere near close enough to run a virtual OS off. However, with the advancements in SSD technology and the promises of USB 3.0 maybe we're not too far away.
Robin
Robin Day on June 3, 2008 9:17 AMJeff,
how you seen this Patriot 150x speed 32GB USB drive? That sounds like a deal. I wonder if it holds up in speed tests.
Or the 64GB USB drive for four grand? (very funny price)
BugFree on June 3, 2008 9:19 AMBugFree: still slow. says right there on the Amazon page:
Data Transfer Rate: 20MBps Read Maximum USB 2.0 - Data Transfer Rate: 15MBps Write USB 2.0
JohnH on June 3, 2008 9:34 AMI'm confused. If flash drives are slower than hard drives, then why does ReadyBoost improve performance? Is it simply because you don't get all the seek latencies for random access?
Adrian on June 3, 2008 9:42 AMAbsolutely true.
Anyway, Windows users don't use to configure their USB storages for the quickest performance. By default, the USB properties sets the optimization for quick extraction.
Changing this parameter, the difference is astonishing.
Personally I've never needed more than 2GB on a flash drive, but that's because I don't use them except for short-term backups or moving things from one computer to another. I love that the 2GB ones are so cheap—they're practically disposable, you don't have to invest any love in the individual stick at all.
Shmork on June 3, 2008 10:14 AMSoftware is like porn, it is better when you have it on the palm of your hand.
Linus Torvalds on June 3, 2008 10:18 AMFor anyone considering running an operating system from a flash drive, there is this article in InformationWeek, "How To Run Linux From A USB Flash Drive"
http://www.informationweek.com/story/showArticle.jhtml?articleID=207800392
To quote the article: "What's more, there are ways to run Linux from a flash drive that don't even require an OS reboot, especially if you're running Windows."
Charles on June 3, 2008 10:46 AMHow do people find the reliability of the larger capacity Flash drives? I've been considering either an 8GB or 16GB, but the number of stories I've read about them crapping out has me hesitant to spend that kind of money.
Evan M. on June 3, 2008 11:21 AMI've been bringing my OS around with me for about a year now. Puppy Linux on flash, with a supplemental CD-ROM booter for comps that won't boot straight to USB.
I even have an emulator in there so I can run it in Windows.
Zoasterboy on June 3, 2008 11:30 AMI'm very happy with my 4GB Kingston pendrive.
Feli on June 3, 2008 11:31 AMHi Jeff,
I've been using OCZ USB drives for a few years now, and there's definitely a large difference. Especially when you compare it to those cheap $10-20 variations!
We also looked at offering our software on a USB key, such as the OCZ, but the performance was noticeable with larger databases. It wasn't atrocious, but it was noticeable. Of course for the average consumer it's probably not that big an issue, we still have many customers using P2 and P3's with under 512MB Ram running XP with antivirus, etc. The OS alone is a crawl and they're willing to accept that.
Something else to note, most USB keys have limited read/writes before they fail, some pretty low. So for example if you're planning to run an application, especially one with a database that gets queried often, be careful and backup often. Biggest tip I can offer.
Stephane Grenier on June 3, 2008 12:25 PM"I'm not in the hard drive performance matters camp" - You got a page file?
The strength of the USB is in random seek/read/write which utterly KILLS your average HD. (well as long as you dont get the cheap flash drives)
Which is the whole Vista ReadyBoost feature. Ready boost will use the usb drive as a cache for your paging. A real geek doesnt need ReadyBoost as a real geek knows how to put the page file onto the USB drive... But it's good for general users.
If you got a 1 or 2 gig laptop with a standard 5k rpm HD, seriously try sticking a GOOD USB drive in and using ReadyBoost. You will notice a difference.
It's really noticable on low memory machines. 'cause Paging happens.
There is a huge hype about SSD drives, and for portable devices I can see the attraction. But for a desktop? Desktop doesn't have the same problem with power-consumption as laptops. SSD is not that fast, it can compete with normal 3.5" 7200 rmp drives, but if we stick them up against 15k rpm SAS drives they fade in comparison.
For running an os, assuming you run your swap/pagetable files on the same hd, it's not the transfer speed that counts, but the seek time. If you need to read 2 files are the same time, you will read 1 block(4k usually) of each file at a time, and you will have to seek in between them to get to the other file. This is specially funny when your drive is fragmented.
Thomas Winsnes on June 3, 2008 12:53 PMThis post came along pretty perfectly. I was just getting ready to look for a new flash drive.
Ryan on June 3, 2008 1:05 PMJeff,
I just thought I should point you to the current king of SSD's:
The MTRON Pro
http://www.king-cart.com/cgi-bin/cart.cgi?store=dvnationproduct_name=MTRON+PRO+2.5+16GB+SATA+SSDexact_match=exact
This thing trumps the Raptors but are still pretty pricey...
They were able to get 860 MB/s in a 4X RAID array. Now that's extreme ;-]
Jeremy on June 3, 2008 1:14 PMOn Security Now #135 they discussed two different types of memory for USB flash drives in the Ironkey interview:
------------------------
DAVID: Oh, yeah. Well, the difference is the single versus dual and the MLC versus SLC. Your main speed comes from MLC versus SLC. So you can get an MLC device with a single chip, it'll write 4 Mbps, maybe 3. And we can write up to 18 to 20 Mbps because it's much faster, more expensive memory, and because it's dual channel.
LEO: Is the read faster, as well?
DAVID: Yeah, it's like 30 Mbps or more on the bigger ones.
---------------------------
Are there any other USB drives using dual channel SLC memory?
Ryan Collins on June 3, 2008 1:26 PMJeff,
Do you know if the storage technology in USB drives is the same as in SSDs, such as those that ship in the Apple Air?
Craig Boland on June 3, 2008 1:27 PMThis is totally discounting el-cheapo USB drives, of course. I remember winning a free USB stick (I think it was a Microstore one?) from the University that has an absolutely _abysmal_ read/write speed.
Copying my textures over to the disk (roughly 500, each around 1MB) took well over an hour on the disk. And thats not even counting Vista's caching, which refused to let me eject the disk for another ten or so minutes.
Don't buy the $10 2GB USB drives. Bad, bad idea!
Scott MacDonald on June 3, 2008 1:36 PMI'm all for bigger, faster hard drives - but there is a tipping point looming out there.
How much data can I afford to have attached to a small device that I can drop in a puddle? or leave behind in my hotel room?
I'm sure there are lots of good practices to keep these devices attached to items you aren't likely to forget - but a USB drive is yet another small, expensive easy-to-lose item. Add in the ever-increasing capacity for storage and you have the makings of a device that will encourage you to store more and more stuff - and then it will fall down a sewer grate.
There's undeniable appeal to small and portable - but stand alone portable storage has risks that networked storage doesn't have.
murph on June 3, 2008 1:43 PMA while back, I came across an interview with Linus Torvalds, in which he said something like, "file systems on non-rotating media will be different". This led me to quote him on a thread on comp.databases.theory. I mused that such storage, combined with multi-core, multi-processor machines would lead to the resurrection of the relational (fully normalized) database. Since such a database is by definition without redundant data, its data footprint is at least an order of magnitude smaller than the xml flat file stuff now in vogue.
Anyway, Joe Celko chimed in, in agreement and said his soon to be published book ("Thinking in Sets", get it) discussed such.
Texas Memory Systems has been building SSD machines for a while. Industrial strength SSDs with MC/MP cpus. Oracle claims to be doing work with just such. It'll be interesting.
BuggyFunBunny on June 4, 2008 2:13 AMSanDisk's U3 is about taking your system everywhere on a USB memory
robi on June 4, 2008 2:36 AMSuperTalent DH series deserves kudos. Its probably as fast as our second place contender, yet costs less than any of these.
rektide on June 4, 2008 2:56 AMOn the subject of USB key speeds...
Anyone has any idea why Flash keys (in general) seems to have difficulty writing as fast as they can when downloading something off a HTTP(S) server?
i.e. why do, when I try to download a 500MB file directly to my USB key using HTTPS, it takes 25-30 minutes (300KB/s), but when I download the same file, from the same server, to my local HDD, it takes 2 minutes (4MB/s), and copying the same file from my local HDD to the USB key takes about 2 minutes too... It's as if there was something that throttled the download speed when downloading directly to a USB key...
Also, I tried downloading the same file to a USB HDD, and I get speed similar to downloading to a local HDD...
(Only tested on Windows; not sure if Linux / Mac has similar problems...)
Jeff: Good article, as usu, but a couple of observations to go w/ the rest:
1) Formatting a UFD w/ NTFS (or ext3, xfs, etc) is generally regarded as bad practice. Why ? The aforementioned are all journaling filesystems and write more than just your data to the drive (for fault tolerance.) Virtually every time I've read about this, these file systems are reported to significantly shorten the life of a UFD. What's your take on the subj ?
2) Get a UFD that has a retractable USB iface. While they're plastic, my SanDisk Cruisers have held up pretty well (knock on wood !) after six months of daily residence in my pocket. And I carry them EVERY time I leave the house. With a retractable iface, that's one less thing to lose (the iface cap) and pos shorten the life of the UFD.
Don't have much to add on the subj of speed as I don't expect much, given the nature of how UFDs are used. That is, we plan to plug these devices into many diff systems and has already been pointed out, not all USBs hubs are the same. Hell, it's not uncommon to plug in via a USB 1 root hub, esp if you're doing repairs and maintenance on other folks PCs.
But they're absolutely amazing for carrying around a bootable, useable OS, like BartPE, Knoppix, or Slax !
The Jetman on June 4, 2008 5:31 AMJust wanted to agree and emphasize Thomas Winsnes comments regarding SAS. Current PATA or SATA desktop hard drives at 7500 or even 10000 RPM are not comparable to SAS (Serial Attached SCSI) drives or even older U320 and U160 SCSI drives at 10000 RPM and especially at 15000 RPM.
Seek times on a good SCSI drive are a fraction of PATA or SATA drives and the SCSI architecture allows for concurrent read/write operations and other optimizations to maximize throughput and ensure data integrity.
If you really want optimal performance on your desktop, get a good SAS controller and drive. You won't believe the difference... seriously. Another benefit is that the SCSI drives are built to last as they’re intended for servers in a commercial environment. Sure they cost more than the consumer drives but you get what you pay for... and are far less than the current SSD drive prices.
I've been running my web server on an Adaptec 29160 controller and a Fujitsu 7200RPM SCSI drive for over 8 years without a single issue. The server is running Windows Server 2003 Standard on dual Athlon MP 1.2GHz processors and it's still my fastest desktop system. Much more responsive than my Intel Core 2 Duo and Athlon X2 multi-core systems running on SATA 300 drives… even with 2-4 times the system memory and at much higher processor speeds.
But the article was still helpful as I'm looking for a fast USB thumbdrive to use for ReadyBoost in Vista. However, I'll definitely be going with an SAS controller/drive(s) in my next system build.
For geek factor alone, I'm surprised nobody has mentioned the KingMax SuperStick. It's so small that I carry it around in an empty Compact Flash holder (where it rattles around). It's strong and works a treat and is guaranteed to fit in any USB slot (mostly because it's smaller than the USB slot in the first place).
I bought the 1 Gig one a few years back for 14 quid. For capacity, it doesn't matter; There's enough space in 1G to store normal files, and I have an IPod for anything beefy like transporting photos around.
The SuperStick is a wonderful device. I see all these chunky rubbish usb sticks and laugh. Why would anyone want to carry those behemoths around?
Flash speeds are painfully slow. I wasn't an early adopter, and I too expected better random access times. Flash maybe slow, but it should beat out spinning discs, even low power ones in ipods, etc. What gives?
http://www.kingmax.com/products/ud/ss.htm
It seems odd, Jeff, to say something like "think of how long it'll take to fill at that speed!" and then say, in closing, "Now if I can just figure out what else to put on mine..."
Some people might be filling and emptying their entire flash drive very often (they really ought to buy a tiny 1.6 or 2.5" drive enclosure, seriously) - but I don' think that's most of us.
Most of us, in my experience, will simply use a 16G flash drive exactly like a 2G one... except it'll take 8 times as long to get to the point of "what do I delete to make enough room for more stuff?". (Or even longer, if we need "X Megs free to put an image on it for work", since the X remains constant rather than proportional to the total.)
Bruce: Amazing how spending hundreds of dollars on a dedicated controller makes something faster than a random motherboard controller, isn't it?
(Though your 8 year old drive is long past the end of its manufacturer's claims of its duty cycle. Fujitsu, like everyone else I've ever looked at, says their product life for a drive is five years. I'd start thinking real hard about pre-emptive replacement if uptime matters.)
Sigivald on June 4, 2008 12:24 PMSpace + os:
Mocka
Qemu (try qemupuppy for example, bootable or virtual machine)
But event with fastest hdd (or ssd), VS2008 from time to time just stops responding. CPU is idle, hdd doing nothing, but VS has some kind of System.Threading.Thread.Sleep(5000) :) Even with all the patches.
I think they're releasing new sw versions with eye-candy stuff without harvesting all raw cpu/hdd power, just adding new stuff to old code base. And there is all kinds of developers.. bad ones too
hhrvoje on June 4, 2008 1:20 PM"USB flash drive performance, even for the best models, is a small fraction of typical hard drive performance."
Why was the above bolded, as though it were news, and not common knowledge?
George Jempty on June 4, 2008 1:38 PMHey, how much you get paid for this??
Shasts on June 5, 2008 2:32 AMWow...they have 32GB USB Drives now? I'm really lagging behind with only 2GB...
Leo on June 5, 2008 3:35 AMNice post Jeff,
Just wondering what partitioning tool you used to format the drive as NTFS. I'm only able to format as FAT32 in XP for whatever reason. I should keep the drive (OCZ ATV 32GB) compatible with Macs, though, right? Trying to decide which direction I want to go on that end.
Much appreciated!
Chris Stewart on June 5, 2008 7:07 AMThat flash voyager isn't very durable. It looks like it is because of that rubber casing, but the usb flange thing and connecter is just soldered onto a circuit board with no reinforcement besides the rubber housing. Rubber isn't rigid at all, so basically there is no reinforcement. My flash voyager broke within a month... but I use it a lot installing software at work. The Super Talent Pico-C is by far the best drive I've ever owned. It's simple compact design makes it really durable. I wouldn't attach it to my keys, i don't think it's that durable. But it's so small I could put it in my wallet, but I attach it to a clip on my work Id badge. Oh, and your blog is the best!
Ben on June 5, 2008 8:30 AMI have a dream, that by the end of next year, I will be able to buy bootable 128Gb USB 3.0 flash drives which give performance on a par with at least a laptop HDD. I can buy a single laptop with little or no onboard storage, and have multiple system installs: work environment; dev environment; gaming environment; CTPs and betas; probably some other things I haven't even thought of. I'll be able to hibernate one session, switch drives and resume another in a matter of seconds.
Mark on June 6, 2008 2:32 AMOne thing you didn't mention is the fact that USB Flash keys tend to die if you keep them in your pocket.
I am in the habit of keeping a USB key in my pocket at all times. They are sooo convenient. However, I never store anything of value on them without a backup because the USB keys only last about 6 months before failure. THAT is the reason that I only use the cheapest ones that I can find--preferably free.
Sometimes it is the case that falls apart and other times it is the memory itself that fails. I would hate to spend good money on a 16GB memory key that will be worth $0 before you know it.
Dominoes on June 6, 2008 2:43 AMWhy is it that whenever a blogger make a post about comparing different products, some snarky asshole always comments and asks how much said blogger is getting paid for it?
I mean, you're not allowed to have positive opinions about products you have used unless the manufacturer has given you money to do so, right?
Adam on June 6, 2008 10:46 AMPlease, someone give me at least one example of 32gb-64gb of data that you would _need_ to carry around with you.
I'd love a 64gb USB stick too, because I'm a huge geek, but I'd never use that much. It's all hand waving reasoning to have one after that.
Steve-O on June 9, 2008 8:24 AMMy thoughts on Multi-use:
After losing my one and only USB flash drive (which I used all of 2 or 3 times), I picked up a 16 GB SD card for my camera packaged with a USB card reader. No, it's not as convenient as a flash drive, it requires carrying around two things instead of just one, but for people who don't use flash drives very often but do want to have the ability to carry large amounts of data on something small (for those trips to Kinkos to have a 50 page document filled with uncompressed pictures printed), it works quite well. Of course, its nice to have capacity for 5000 pictures for those long laptop-free vacations.
ddiepo on June 9, 2008 1:48 PMThe Voyager GT is definitely faster than the standard Voyager. I have 2 x 8GB GT's for running virtual machines, and they do a nice job. I also have a 120GB Western Digital Passport portable drive (5400rpm) and I would say that the 2 compared to each other offer about the same performance when running VMware.
Stu on June 16, 2008 9:27 AMI remember the first time i noticed this. I had upgraded from 256mb flash drive to a 2gb drive. I did a lot of development right off that drive at school and at home. I noticed it took really long to merge my project directory even though it was USB 2.0. it turned out that the chip itself was significantly slower than my old chip. it was disappointing because i actually really liked that flash drive other wise. it ended up breaking a couple of years later, screwing me over on a couple of things i was storing there.
Tim on June 18, 2008 10:40 AMStrange, I would expect Flash memory to be insanely faster than hard drives for random access. No physical parts to move during seeking.
Nicolas on June 24, 2008 1:09 PMI have a newegg scraper that automatically calculates the GB/$ for hard drives, flash cards, and usb flash drives @ http://forre.st/storage
Forrest Voight on August 1, 2008 11:34 AMsoo perfect!
inna lopez on August 7, 2008 7:30 AMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |