20TB OptiNAND CMR vs 20TB SMR comparison
Find out how technologically different HDDs of the same capacity stack up against each other
Who is who?
1. Western Digital Ultrastar DC HC 560 CMR
Western Digital just launched their 20 TB mechanical hard drive, with 2.2 terabytes per platter and onboard flash memory utilizing OptiNAND technology.
2. Western Digital Ultrastar DC HC 650 SMR
SMR is an acronym for “Shingled Magnetic Recording,” an important technology utilized to increase capacity and enable lower cost per TB in hard disk drives. Read more here about SMR.
Western Digital OptiNAND
WD OptiNAND is a platform that combines iNAND, controller, and firmware as an SoC delivered on the PCB of a hard drive. The iNAND is a 64-layer dynamic flash (BICS3) package that manages the hard drive’s metadata. In short, while the capacity of the drive tends to grow, the IOPS per capacity metric goes down and some innovations are needed to mitigate the performance issue.
The OptiNAND technology makes perfect sense to solve the performance issue and keep TB cost low, for larger capacities in the future dual-actuator models will be introduced to boost performance across the greater number of platters.
Testing Methodology Details
The benchmarking process was done for various kinds of workloads: sequential reads and writes, random reads and writes; various block sizes: 4k, 64k, 1m, 32m. The size of the file is 50G, 8 threads per HDD, 10 minutes for each kind of test (8 drives was tested simultaneously to simulate the real-life load for the software and the hardware stack) –
see the server specification and script as follows:
The server specification:
2U DIAWAY Tartu Server
CPU: AMD EPYC 7313 16-Core Processor
RAM: 8 x 32GB DDR4 Samsung 2933Mhz
Kernel: Linux btrfs 5.16.0-0.rc1.20211115git8ab774587903.14.btrfs.fc36.x86
BTRFS: btrfs-progs v5.15
FIO: fio-3.28
Benchmarking Numbers Insights
We see the full data rate for the more sequential workloads (1MB Write and 32MB Write) = 256MiBs (actual) versus 257 MiB/s (typical as specified).
Even 4kB (linear) writes come close to the theoretical data rate limit, speaking for the quality of the HDD internal command queue, handling > 65,500 commands per second.
We also see 157-160 random read IOPS at 4kB and 64kB = 100% above the mechanical full random seek rate (ca. 80 IOPS) as expected through command re-ordering.
The random write rate of the SMR HDD at 64kB and 1MB block size is higher as the result of external buffering and command re-ordering which is also not surprising and speaks for the quality of the BTRFS software layer due to native SMR support implementation.
It’s a bit puzzling that the linear reads stay behind the linear writes (258 MiBs versus 260 MiBs) which may have to do with buffer segmentation, read-ahead strategy, and the occasional sector miss after a head or cylinder switch. Since that phenomenon appears to be present in both HDD generations, it must be something systematic, speaking against read signal- or servo quality-related causes.
Highly likely it was caused by BTRFS layer write optimization. Mind that there is a sync command for POSIX FS that forces writing the data to HDD, otherwise, the data is written most conveniently for the software (e.g. when FS decides when and how is better to write the data to HDD). That's why writes can be relatively high compared to reads. In general, I expect numbers to be correlated with HDD physics, but there is an interim FS layer that affects numbers a lot, and that is especially interesting to compare real SMR implementation vs PMR in an apple-to-apple manner.
Conclusion
20TB SMR pro’s
At the moment 20TB SMR has the best $/TB while the performance penalty is relatively low on average – may even make no difference or SMR can be faster for some specific use cases like a cold backup (WORM = write once, read many).
20TB SMR contra’s
The SMR adoption effort/investment only makes sense at a significant scale (let’s say starting from 1000 HDDs).
20TB CMR pro’s
No special adoption is needed for the highest density, performance.
20TB CMR contra’s
The price may be suboptimal in case the density is not a priority. 18TB PMR has a better TB cost if SMR is not an option for some reason.
For large-scale deployments, SMR is a very good trade-off between the performance, cost of TB, and technology adoption effort. Many hyperscalers including Wasabi.com and Dropbox.com leveraged the SMR benefits on their scale.
At the same time, we are prepping the tests for Min.io and LizardFS that will be based on BTRFS to highlight the real-life applications' applicability for SMR technology.