Lag Testing on a Budget

Keeping input and display latency to a minimum is very important when playing any kind of vintage video game which relies to some extent on reflexes.  There are some methods which can test display lag of any display, like the Time Sleuth or the Leo Bodnar Display Lag Testers.  Other methods may require running the same software on two consoles at the same time or connecting one console to two displays via splitters and adapters.  Testing controller latency often requires wiring up an LED or shooting video of a screen and button pressing at a very high frame rate.  These methods tend to be expensive, but what if we consider an approach that is likely to be inexpensive and perhaps cost you nothing?

When testing lag it is best to control for all variables other than the one thing you are comparing.  If you are testing displays, use the same kind of system and controller.  If you are testing controllers, use the same display and system.  If testing emulators, use the same PC to run them and controller for input.  

My method relies on the Manual Lag test of the 240p Test Suite by Artemio Urbina.  This test is available for most of the popular consoles : Sega Genesis (Mega Drive), Sega CD (Mega CD), PC Engine/Turbografx-16, Super CD-ROM2/CD-ROM2, NES/Famicom, Super Nintendo/Super Famicom, Sega Dreamcast, Nintendo Wii and GameCube, PlayStation, Game Boy/Game Boy Color (144p) and Game Boy Advance (160p).  The manual lag test displays a moving reticle which you must press a button when the moving reticle is exactly over the static reticle.  The movement reticle can move horizontally or vertically, play a sound when the reticles meet or not, and move a fixed or a random distance from the static reticle.

When testing lag, you should focus on the question "what is good enough for you?".  Manual lag testing is important because it measures not just lag but your own "built-in" lag.  If you play with wireless controllers or HDMI LCDs or emulators, you can expect some lag to be involved.  Start adding more of these things and the lag will increase, these things tend to stack up.  Certain combinations can be superior to other combinations, so mixing and matching displays, controllers and game-playing hardware may be your ticket to low-latency.  

The ideal lag begins, if you can manage it, with a CRT, wired controllers and original consoles.  One cannot measure lag in a vacuum, in order to be a useful score for retro video games, lag must be measured relative to something else.  This is important to establish a base control for what kind of lag you can expect when playing with ideal conditions.  Everything that does not function like the originals should be measured relative to this measurement.  

Run the test at least three times with the reticle moving horizontally and three times vertically.  Stagger the tests so that a vertical test follows a horizonal test or vice versa.  Keep the reticle movement on manual.  Although perhaps the audio would be best if it emitted a tone randomly, as games do not always give you audio cues, with the random movement on you should not get too stuck in a pattern.  All these conditions are designed to simulate real-life play as much as possible.  Games are not always obliging with predictable rhtyhms or patterns.  You must hit the moving reticle exactly when or after it passes the static reticle ten times, pressing the button too early will not count.  

Write down your results for each test and keep the time between each run at a minimum.  Your lag results are likely to be better once you have fully woken up than when you are falling asleep or suffering from a lack of sleep.  Use the average of the three or more horizontal/vertical runs to give a rough idea of what the control lag average is for you and your hardware.  Your lag is likely to be longer as you sit further away from the screen than if you sit closer to it.  Reduce or eliminate distractions, avoid extreme levels of room brightness, turn on your game mode if your modern display has one.  

Next, run the tests again under the same conditions but try to only change one thing, whether it be a controller, a console or a display.  Again you should start the new test ideally soon after you perform the control test.  If you are testing a software emulator's lag, you may not be able to use the same controller as you can on an original console or you may have to use a converter.  You generally can use 8bitdo Bluetooth controllers,  them on original consoles with Retro Receivers and on emulators via the PC or the Rasp Pi's built-in Bluetooth stack.  

In my case, I wanted to get an idea of the lag differences between a wired NES controller, a 2.4G Wireless NES controller and a Bluetooth controller.  So I grabbed a NES-004 with good membranes, an 8bitdo N30 2.4G Wireless Controller and an 8bitdo SN30 Bluetooth with NES Retro Receiver.  I did not have an N30 Bluetooth but the NES Retro Receiver will not return more than eight button states anyway with current firmware.  I used my AV Famicom and composite cables, no mods here, and one of my favorite 15KHz CRTs.  I made sure the firmware was updated for the N30 (v5.01) and SN30 (v6.14) controllers and the Retro Receiver (v1.39) to their most current versions available as of this blog post.    

Here are the results :

These results confirmed my own subjective experience, namely that the wired controller is slightly less laggy than the 2.4G controller and the 2.4G controller is somewhat less laggy than the Bluetooth controller.  One thing you may notice is that the Bluetooth controller has double the runs than the other controllers.  I wanted to see if more runs would help get the lag down, but the second three runs were only .1 frames on average less laggy than the first three runs.  Also, while the wired and 2.4G have a range of 0.5 frames between their lowest and highest frames, the Bluetooth has significantly more of a range, 1.4 frames, than the others.  You may want to consider using something with less lag than a Bluetooth controller if you are trying to beat Mike Tyson in Punch-Out!!.

I should note that when I had either the 2.4G receiver or the Retro Receiver in the system, there was only one wireless receiver connected to the console.  Putting a second receiver may cause an increase in the lag scores as you add more, similar wireless signals to the mix.  The NES may be especially prone to interference where the controller ports are almost touching each other.  I did not write them down, but I received similar numbers from the SN30 and the M30 2.4G controllers, 8bitdo did an excellent job with keeping latency to a minimum with these controllers.  

Because the button presses before the reticles meet do not count, you are likely to get many early results during your testing.  This is actually a good thing if you can keep ahead of the reticle more often than not.  With very laggy setups the test will be over very quickly as you will almost constantly be late. 

If you are going to do the stopwatch style test, one easy way to run it is to use a console which has two outputs.  The front loading NES has RF and composite video outputs and both can be driven at the same time.  The full-size SNES and the Sega Genesis Model 1 and 2 also have RF and Composite/RGB output (SNES also has S-Video).   The PlayStation 1 SCPH-1001 has separate RCA AV output jacks in addition to the Sony Multi-I/O output, and the GameCube can output Composite or S-Video alongside Component Video from the digital port (except the DOL-101 and only at 240p/480i).  The Turbo Grafx 16 only does RF out of the box but AV can be added through inexpensive adapters.  Also, for those consoles which feature more than one type of video output on an AV port, there are often inexpensive third-party cables which can offer both composite and S-video output on a single cable.  These cables may not be high quality, but for purposes of testing lag, they should do.

The only thing you really need for this test is a way to run the 240p Test Suite.  So at best you may need to buy a flash cart or install a mod chip or optical drive emulator. But if you are a regular reader of this blog, you're probably all set in that department :)