Saturday, June 15, 2013

Internet "speed tests" generally unhelpful

So let's say you are having internet issues that look like traffic drops.  The ISP's technician runs a speed test and shows you that your connection speed is ok.  But you know from your previous runs that the results are unstable.  A reason why is that several popular speed test sites measure speed by receiving files (download speed) and sending files (upload speed).  Some examples include http://www.speedtest.net/, http://www.speakeasy.net/speedtest/, http://speedtest.comcast.net/, http://www.bandwidthplace.com/, and http://whatismyipaddress.com/speed-test.  The issue with this procedure is that they also end up measuring file I/O and, particularly, how your browser does file I/O, and how your operating system caches file I/O.

Yes yes I know your SSD drive is fast, and yes yes your hardware RAID card is awesome.  But why should there be file[system] I/O at all when doing a network test?  Here's the net effect I have observed on OS X with Firefox, on a fast machine with plenty of RAM to cache disk reads.

First, there is the issue of where Firefox puts the files, which is the browser's cache.  If your cache is full, then Firefox will have to make space in the browser's cache as it's writing the file during the speed test.  The cleanup operation will cause random file I/O in the cache's hash bucketed directory structure to find the files, and then more random file I/O to delete the files.  OS X doesn't like caching a lot of disk writes while deleting files, so this means file I/O checkpoints.  If your file system is POSIX, that also means a number of atime updates as well.  Finally, I assume something will have to be done in the database that's holding which URLs correspond to what files.  And what if the cache is full of tiny (less than 4kb) files, as it usually is?  Then all this file I/O will happen every 4kb or less during your speed test measured in megabytes per second.  Oh, and did I mention there has to be file I/O to write the file to the browser's cache too?

Well ok you could at least clear the cache before beginning.  Try deleting 250k files out of a 1gb browser cache on OS X.  With an SSD drive, that can take a significant fraction of a minute (top speed on the order of 2k IOPS, usually less than 1k IOPS).  With a regular hard drive, that will cost you half an hour or more (top speed on the order of 300 IOPS, and a lot of audible seeking --- as if the OS was flushing disk buffers continuously).

But even if you flush the cache so the browser can write without deleting anything, that's not a guarantee of a proper test because OS X also buffers writes until the buffer fills, and then further I/O is effectively blocked until the buffers are flushed to disk.  You can see this effect by using the activity monitor app during the test.  The writes will flatline at zero most of the time, and at some point the buffers will be flushed (at hopefully much higher speeds than the speed test itself is receiving data).  While the flushing happens, most speed tests seem to get stuck and their fancy graphics don't update, etc.

I've yet to find a browser based speed test that simply allocates memory to do these things.  I'm not saying none exists.  Speed tests such as the above are not useful because of the problems described above.

Generally, the issue with the Firefox cache induced file I/O can be improved dramatically by creating an ExFAT disk image and mounting it where Firefox expects the hash bucketed directory structure to be.  Clearing the browser's cache is vastly improved with this approach.  Whereas it can take half an hour or more with a regular drive to delete 250k files and 1gb of cached data, the disk image approach can do the same thing in a few seconds.  Effectively, you get SSD performance out of your mechanical drive simply because you coax software to do a better job of file I/O.  The only hiccup will occur when OS X decides to flush disk buffers, at which point the machine might seem stuck for a couple seconds.

Can we please fix these underlying issues so that we can get proper performance out of SSD drives too?

Update: a Comcast technician suggests http://testmy.net, which doesn't look like it's writing stuff to the browser's cache.  Nice!

No comments: