Non-Motherboard NVMe Installation

[Dave Labrecque]

You know, these days we are often presented with numbers with a whole lot of zeros. It's hard to put them into a framework we can grasp. I just thought of a way to (hopefully) put the scope of NVMe throughput into focus.

Those of us, who are old enough, remember what an encyclopedia, in paper volumes, looks like. What a volume weighs in the hand. We visited the library to use their volumes to research our school reports. How long would it take you to read the entire set of encyclopedia, do you think?

Well, Microsoft's Encyclopedia Encarta 2005 contains about 26 million words, comprised of about 40 million characters (according to: https://en.wikipedia.org/wiki/Wikipe...ze_comparisons). As straight text, each alphabet character can be represented by 1 byte. That new Samsung ver. 4 NVMe drive can read those words in their entirety, the entire encyclopedia, 175 times every second (seven billion / 40 million). NVMe is capable of reading an entire encyclopedia 175 times every second. That's how fast it is.

Some may point out that these days each character is almost always graphical, and is made of dots so that it can have lots of extra things associated with it like font and type size and color - and all those things increase the storage size of each. And an encyclopedia has pictures and charts that are not comprised of letters of the alphabet. And that's all true. But, if you rendered just the words of Encyclopedia Encarta 2005 in a text editor - each character would take exactly 1 byte, and there are 40 million of them (give or take). Work with me here.

Truly impressive, but not very real-world, I think. In reality it's lots of smaller files. How long would it take to read 175 files that each contained the Encyclopedia text from this NMVe drive? I think that changes the picture quite a bit.

Sorry. I'm just a big, wet blanket.

[John Ludlow]

Truly impressive, but not very real-world, I think. In reality it's lots of smaller files. How long would it take to read 175 files that each contained the Encyclopedia text from this NMVe drive? I think that changes the picture quite a bit.

Sorry. I'm just a big, wet blanket.

Well... that is beyond my practical knowledge, so I don't know for sure. But, I am sure it's not like the situation on a Winchester drive. One of the big reasons that small files are slower on a spinning disk drive, is that the beginning of each file has to be located physically on the disk. An un-fragmented disk drive will have a lot of its file's constituent sectors lined up, in order, next to each other. But the drive still has to locate the beginning of each file. So, the read head moves to the correct cylinder and then waits for the beginning of the file to spin around to it. But when it finally arrives, it just reads sector after sector as they pass beneath the read head. Large files have more contiguous sectors per each 'seek' than small files do. The drive has to locate the beginning of each file by moving its read head and waiting for the first sector to spin around to it more often. And that takes time.


But, SSDs are just RAM. Each virtual 'sector' has an absolute address - and they're all equivalent to each other from the drive's perspective. There's no advantage, that I can imagine, to all a files memory locations being physically near each other. As a result, I don't see why small files would cause it to run slower. And also, from the perspective of 'beginning of file seeks' there would only be 175 of them distributed over seven billion characters read anyway - which is a pretty small ratio. Recall, you're identifying an encyclopedia as a small file. But, again, I haven't tested to know for sure - and maybe you have.


Still... what if, in the very worst case, it takes 2 seconds to read 175 encyclopedias? That still puts the relative speed of an NVMe SSD into 'touchable' perspective, doesn't it? They're frighteningly fast.

[Dave Labrecque]

Well... that is beyond my practical knowledge, so I don't know for sure. But, I am sure it's not like the situation on a Winchester drive. One of the big reasons that small files are slower on a spinning disk drive, is that the beginning of each file has to be located physically on the disk. An un-fragmented disk drive will have a lot of its file's constituent sectors lined up, in order, next to each other. But the drive still has to locate the beginning of each file. So, the read head moves to the correct cylinder and then waits for the beginning of the file to spin around to it. But when it finally arrives, it just reads sector after sector as they pass beneath the read head. Large files have more contiguous sectors per each 'seek' than small files do. The drive has to locate the beginning of each file by moving its read head and waiting for the first sector to spin around to it more often. And that takes time.


But, SSDs are just RAM. Each virtual 'sector' has an absolute address - and they're all equivalent to each other from the drive's perspective. There's no advantage, that I can imagine, to all a files memory locations being physically near each other. As a result, I don't see why small files would cause it to run slower. And also, from the perspective of 'beginning of file seeks' there would only be 175 of them distributed over seven billion characters read anyway - which is a pretty small ratio. Recall, you're identifying an encyclopedia as a small file. But, again, I haven't tested to know for sure - and maybe you have.


Still... what if, in the very worst case, it takes 2 seconds to read 175 encyclopedias? That still puts the relative speed of an NVMe SSD into 'touchable' perspective, doesn't it? They're frighteningly fast.

Yeah, I was taking us off your point, which is that they're just crazy fast, regardless of scenario. And it's true that 175 encyclopedias are still 175 pretty good-sized files.

I'm just acting out from my recent experience, I think, of having bought two M.2 NVMe drives (1/2 and 1 TB Samsung 970 EVO Plus) that had really decent bang-for-the-buck benchmark scores. But when transferring data to them (from, admittedly, a not-as-fast, older Samsung SATA SSD) or even between them, I watched the throughput sit down in double-digit MB/sec territory with only occasional peaks into the 2 and 3 GB/sec zones—due to the large number of smaller files on my (probably typical) system. It was a real lesson in real-world performance and how important the "other" number is when looking at drive benchmarks.

So, sorry for my poopy diaper.

[John Ludlow]

Yeah, I was taking us off your point, which is that they're just crazy fast, regardless of scenario. And it's true that 175 encyclopedias are still 175 pretty good-sized files.

I'm just acting out from my recent experience, I think, of having bought two M.2 NVMe drives (1/2 and 1 TB Samsung 970 EVO Plus) that had really decent bang-for-the-buck benchmark scores. But when transferring data to them (from, admittedly, a not-as-fast, older Samsung SATA SSD) or even between them, I watched the throughput sit down in double-digit MB/sec territory with only occasional peaks into the 2 and 3 GB/sec zones due to the large number of smaller files on my (probably typical) system. It was a real lesson in real-world performance and how important the "other" number is when looking at drive benchmarks.

So, sorry for my poopy diaper.

No, no - you're good. I have a couple of 970s too. They're great, I think - although they're ver. 3 drives. It does sound like you know whereof you speak. But, consider that while transferring from one to the other, from a read operation standpoint - the write at the other drive is a limiting factor. The read drive has to periodically stop to let the write drive catch up, otherwise it would overflow the buffer in between them. Although the write specs from a 970 are nothing to scoff at either. So it still doesn't explain double-digit read figures. I don't understand why that would happen.

How were you monitoring throughput?

[cgrafx]

Yeah, I was taking us off your point, which is that they're just crazy fast, regardless of scenario. And it's true that 175 encyclopedias are still 175 pretty good-sized files.

I'm just acting out from my recent experience, I think, of having bought two M.2 NVMe drives (1/2 and 1 TB Samsung 970 EVO Plus) that had really decent bang-for-the-buck benchmark scores. But when transferring data to them (from, admittedly, a not-as-fast, older Samsung SATA SSD) or even between them, I watched the throughput sit down in double-digit MB/sec territory with only occasional peaks into the 2 and 3 GB/sec zones***8212;due to the large number of smaller files on my (probably typical) system. It was a real lesson in real-world performance and how important the "other" number is when looking at drive benchmarks.

So, sorry for my poopy diaper.

You can't transfer data faster than your slowest connect device (if your trying to transfer data to or from a slower device than that is going to be your max data rate)

The SATA SSD is going to severely limit the data transfer rates to the M.2 drive. This is true for other data transfer circumstances as well.

We have 500Mbit fiber internet but my max transfer speed down or up is going to be limited by the remote connection. If the person or system I'm connected to has a slow internet connection, than that will be the max data rate that can be transferred.

[Dave Labrecque]

No, no - you're good. I have a couple of 970s too. They're great, I think - although they're ver. 3 drives. It does sound like you know whereof you speak. But, consider that while transferring from one to the other, from a read operation standpoint - the write at the other drive is a limiting factor. The read drive has to periodically stop to let the write drive catch up, otherwise it would overflow the buffer in between them. Although the write specs from a 970 are nothing to scoff at either. So it still doesn't explain double-digit read figures. I don't understand why that would happen.

How were you monitoring throughput?

I think I was either looking at the throughput readout in the Windows File Explorer file transfer progress indicator and/or the disk read/write displays in the Task Manager performance tab.

I get that the writing drive bottlenecks the throughput, but IIRC, the write and read speeds are actually not too different. Like only a few percent different. (When is there NOT a writing drive on the receiving end? Has me wondering how one could ever realize true read speed potentials.)

I figured it was just the lots-of-little-files factor that bogged it down so. I'd much prefer if there's something I can tweak to vastly improve things. But then I don't do a whole lot of mutli-GB-size transfers. Only when I'm moving data to a new computer. Or maybe when I'm archiving projects to an external drive. But then it's a SATA spinner, so...

My nightly online backup is at the mercy of my 17th-century DSL connection (which lately spends most of it's life under 1 Mbps)! I do have a nightly boot disc image from one EVO to the other that happens, so I suppose that could be fun to watch go faster, but it's kind of inconsequential, happening in the background, and usually after I'm done with the computer for the day.

Gigabit fiber Internet coming in the spring to our town, though! Can't freakin' wait!

[Dave Labrecque]

You can't transfer data faster than your slowest connect device (if your trying to transfer data to or from a slower device than that is going to be your max data rate)

The SATA SSD is going to severely limit the data transfer rates to the M.2 drive. This is true for other data transfer circumstances as well.

We have 500Mbit fiber internet but my max transfer speed down or up is going to be limited by the remote connection. If the person or system I'm connected to has a slow internet connection, than that will be the max data rate that can be transferred.

To be clear, you're saying that the slowest drive involved in the transfer will bottleneck the throughput, right? Not that, for example, my SATA USB spinner existing on the same computer will bog down a transfer between my two NVMe drives, I hope.

Even between those two SSDs I have these crazy low transfer speeds when it's doing the little files. To be sure I wasn't misremembering, I just did a test. I copied C:\Program Files (x86) to a newly created temp folder on my D: drive. Both NVMe drives were around 60% full. I stopped it about halfway through the transfer, and to that point it varied between about 2 and 200 MB/sec, but seemed to average around 40 MB/sec or so. That's, what, around 320 Mbps?

[cgrafx]

Yes, the slowest device involved in the transfer.

[cgrafx]

I get that the writing drive bottlenecks the throughput, but IIRC, the write and read speeds are actually not too different. Like only a few percent different. (When is there NOT a writing drive on the receiving end? Has me wondering how one could ever realize true read speed potentials.)!

Very easily when editing video, and multi-camera 4k/6k/8k footage requires specialized hardware to deal with the bandwidth.

30 minutes of even 4k footage can be 300-400 gigs of data depending on the recording format.

[Dave Labrecque]

Very easily when editing video, and multi-camera 4k/6k/8k footage requires specialized hardware to deal with the bandwidth.

30 minutes of even 4k footage can be 300-400 gigs of data depending on the recording format.

Ah. HD video. Of course.