Drives, interfaces, and capacity planning

Why this matters

The previous lesson chose how disks are combined. This one chooses the disks: which kind, on which interface, and how many terabytes to buy so the server does not run out of space next year or run too slowly this year.

The two classic mistakes point in opposite directions. One is buying capacity without performance: plenty of terabytes on slow disks under a database that needs fast random access. The other is running out of space because nobody measured how quickly it was being used. Both are planning failures, and both are covered here.

The lesson

Hard disks, SATA and SAS SSDs, and NVMe, and where each belongs

Hard disk drives (HDDs) store data on spinning platters read by moving heads. They are the cheapest storage per terabyte and are good at long sequential reads and writes, but every random access means moving the head and waiting for the platter, so they manage only around a hundred to a couple of hundred random operations per second. Server HDDs spin at 7,200, 10,000 or 15,000 revolutions per minute, with faster drives giving better random performance at lower capacities.

Solid-state drives (SSDs) have no moving parts. Their random performance is many times that of any hard disk, and their latency is far lower.

The interface decides how the drive connects and how fast it can talk:

• SATA is the common, inexpensive interface, at 6 Gb/s. It suits lower-cost and less demanding storage.
• SAS (Serial Attached SCSI) is the enterprise interface, at 12 Gb/s and, on newer equipment, 24 Gb/s. SAS drives are built for continuous duty, and many have two ports, so a drive can be reached through two controllers or paths and survive a failure of either. A SAS controller also accepts SATA drives, but a SATA controller cannot run SAS drives.
• NVMe drives connect directly to the PCI Express bus instead of going through a disk controller, which removes a bottleneck and gives the highest performance and lowest latency of the three. In servers they come as U.2 and newer EDSFF drives in hot-swappable front bays, and as M.2 modules, often used for boot.

Drives also come in two physical sizes: 3.5-inch (large form factor), which holds the highest-capacity hard disks, and 2.5-inch (small form factor), the usual size for SSDs and for servers that want many drive bays.

As a rule of thumb: hard disks for bulk capacity, backups and archives; SAS or SATA SSDs for general server and virtual machine storage; NVMe for databases, busy virtual machine hosts and any workload where latency is the bottleneck.

IOPS against throughput, and sizing storage to the workload

Storage performance has two main measures, and workloads care about them very differently.

• IOPS, input/output operations per second, counts how many separate reads and writes the storage can complete. Databases, virtual machines and mail servers make many small, random requests, so IOPS is what limits them.
• Throughput, measured in megabytes or gigabytes per second, is how much data moves. Backups, video and large file copies read and write big, sequential blocks, so throughput limits them.

A third measure, latency, is how long each individual request takes, and for interactive applications it is often what users actually feel.

To size storage, describe the workload first: the mix of reads and writes, whether access is random or sequential, and the typical request size. Then remember the RAID write penalty from the previous lesson. An array's usable write IOPS is its raw IOPS divided by the penalty, so a parity array of hard disks can look adequate on paper and struggle badly with a write-heavy load.

Raw capacity against usable capacity once RAID and formatting take their share

The number on the drive label is not the space you will have. Several things come off the top.

• RAID overhead. Parity or mirroring consumes whole disks, as the capacity table in the previous lesson showed.
• Hot spares. A spare holds no data until it is needed.
• Decimal against binary units. Drive makers use decimal units, where a terabyte is 1,000,000,000,000 bytes. Operating systems usually report in binary units, where a tebibyte (TiB) is 1,099,511,627,776 bytes. So a 4 TB drive appears as roughly 3.64 TiB, often still labelled "TB" by the operating system, and the gap grows with capacity.
• File system overhead and reserved space for metadata.
• Snapshots and free-space reserves that you should plan to keep.

Plan from usable capacity, never from the sum of the labels.

Estimating growth, and the threshold at which to order more

Capacity planning is measuring how fast storage fills and acting before it is full.

Record usage regularly and look at the trend, not just today's figure. If a volume grew by 200 GB a month for the last six months, you can estimate when it will fill. Then work backwards from the lead time to add capacity: ordering, delivery, installation and data migration all take time, so the decision point is well before the volume is full.

Set thresholds that trigger action early, commonly a warning around 70 to 80 per cent full and a critical alert above that. Running storage close to full also causes problems of its own: many file systems slow down as free space runs out, and a full volume can stop applications, databases and logging outright.

Watch for the things that grow unexpectedly: logs, snapshots that were never deleted, temporary files, and backups written to the same storage.

Reading a vendor data sheet for endurance and warranty

A drive's data sheet lists capacity, interface, form factor and, for hard disks, rotational speed. It also lists performance figures, which are measured under ideal conditions and should be read as upper limits rather than promises.

For SSDs, the most important figure is endurance. Flash cells wear out after a limited number of writes, and vendors state how much writing a drive is rated for:

• DWPD, drive writes per day: how many times the drive's full capacity can be written every day across the warranty period.
• TBW, terabytes written: the total amount of data that can be written over the drive's life.

Enterprise SSDs are sold by endurance class: read-intensive drives rated around one drive write per day, mixed-use drives around three, and write-intensive drives higher still. Match the class to the workload. A read-intensive drive under a heavy write load, such as a database log or a caching tier, can wear out long before its warranty ends.

Finally check the warranty length and terms, and the vendor's reliability figures, MTBF (mean time between failures) or AFR (annualised failure rate), remembering that these describe large populations of drives rather than when any one drive will fail.