324 disks takes up most of a row. When we had to do service maintenance in Canyon Park (fun with NT 3.51 and 4.0, before the days of Terminal Server) I'd walk over to TerraServer and admire it's enormity.

>324 disks takes up most of a row.

Not any more. Using high density disk enclosures, you can fit that many disks in a third of a rack, and if each disk held 4.5TB you'd have almost 1.5PB of storage!

What an astonishing change in only 15 years!

Just under 20 years ago when I got started in this racket, 30G was consider a huge database, cutting edge stuff, requiring experts and special techniques. Nowadays a serious DBA doesn't blink at 30T.

I wonder if 324 on Lustre would have more throughput than a 1 4TB disk?

They would be leaps and bounds faster. Especially comparing random IOPS.

Let's just assume they were using a Seagate Cheetah 9LP. That's a 9GB, 10k RPM, 1MB cache UW SCSI disk that was widely available in 1998, for a whopping... $1200 each. $390,000 worth of disks... wow. That could buy 1300 * 4TB SAS drives today with a total raw storage of 5.2PB!

324 * 9LP's would have around 5.5GB/sec of total read/write throughput. That easily beats a single 4TB disk with around 200MB/sec read/write performance and terrible IOPS in comparison to even a single 9LP.

SSDs compare much more favorably here. 10 x 1TB SSDs for ~$6500 could easily reach 5.5GB/sec throughput and triple the capacity of the 324 x 9GB drives.

You're comparing off-the-shelf disks to enterprise-class storage.

"Enterprise" 4TB disks don't have orders of magnitude more IOPs than consumer versions.

But they are more expensive, so for comparison to be right one needs to take that under consideration.

Definitely... and _way_ more IOPS.

You can't just compare storage on the capacity, you also have to account for speed. Even these days, 149GB 15k RPM SAS drives are popular for non-SSD usage. Long way from 4TB, but they outperform them any day when it comes to semi-random usage.

I wonder how much of that usage came from people doing the MSDN tutorials. Those were the golden days of MSDN - I remember learning about factoradics and permutations from my 10 CD MSDN library. Microsoft was doing great stuff back then.

I am thinking to do a back-of-the-envelope orders-of-the-magnitude analysis for a setup similar to this for an Amazon AWS EC2/S3/RDS/foo/bar/baz based system.

Moore's law is about semiconductor transistor count -- why would it have anything to do with magnetic storage?