bsnes Frame Advance Input Latency

I wrote an article recently about optimizing input latency in emulators, available here.
This led to a comment about the perceived input latency in Mega Man X using the frame advance functionality in bsnes. In investigating this issue on my Twitch channel yesterday, I discovered there were two separate issues. These issues only affected frame advance, and not latency in standard gameplay, but all the same, I'd like to do a post-mortem on these here today.

Steps to Reproduce

Load Mega Man X and get in-game. Pause the emulator. Now hold down the fire key, and keep tapping frame advance. See how many frames it takes Mega Man to shoot.

This was taking four frames, whereas in Snes9X it was taking two.

Issue 1: pause to frame advance transitioning

The first issue is that when bsnes was paused and the frame advance key was pressed, the emulator transitioned from pause mode to frame advance mode, but did not actually run a frame. So in effect, nothing happened at all. The user would ordinarily expect a paused emulator to advance one frame on the first key press.

This change corrected the issue.

This reduced the input latency from four frames to three frames, but we still weren't there yet.

Issue 2: cooperative threading and paused emulation

The second issue is much more involved, and is the reason for this blog post.

bsnes is designed using a cooperative threading model. What this means is that unlike a traditional state machine emulator like Snes9X, bsnes can run each emulated processor out of order, synchronizing components only when needed.

bsnes further handles input using just-in-time input polling, as per the input latency article linked at the beginning of this post.

Mega Man X, like most games, polls controller input during the start of the NMI (non-maskable interrupt) period, which is at scanline 225 in non-overscan (NTSC) display mode.

The PPU (video) thread in bsnes would exit once it reached scanline 225, which is done for the sake of the libretro port, which does not use just-in-time polling.

However, because of the design of bsnes, the CPU thread was already executing part of scanline 225 before it needed to synchronize the PPU thread. This is because the CPU was not drawing graphics in the vertical blanking period of the display, and hence had little reason to talk to the PPU.

What was happening here was that Mega Man X immediately started to poll gamepad inputs, and managed to poll two buttons, Mega Man X's fire and jump buttons, before synchronizing the PPU, which caused the scheduler to then exit.

Post-Mortem

So if bsnes has just-in-time polling, why did this result in another frame delay?

It's because of frame advance: by pausing the emulator, it was stopped around halfway through the 225th scanline on the CPU side, and the fire button had already been just-in-time polled and latched (cached) for later use. The CPU already read back the button and had it in its emulated state.

By pressing the fire button now while the emulator was paused, it was already too late: the input was already read. And so when frame advance was hit, the fire button read back as not being pressed.

Workaround

The workaround is here.

Essentially, I took the frame event (scheduler exit) code out of the PPU core, and moved it into the CPU core, where it technically doesn't belong. Now, when the CPU thread hits scanline 225, we perform the frame event here. However, if the CPU is currently executing ahead of the PPU, that would be bad news and the final scanline of the screen wouldn't be fully rendered yet. And so to prevent that, I forcefully synchronize the PPU to the CPU, which guarantees the PPU is at scanline 225 or later when the CPU thread sends a frame event to the GUI.

We don't have to worry about the  case where the PPU might end up ahead of scanline 225, because the PPU does not poll inputs, the CPU does. As long as the CPU has only just started scanline 225, we know that no NMI code has run yet that might have polled inputs.

As a result, when the emulator is paused, we haven't polled the fire button yet. When the user holds the button down and hits frame advance, the emulator will immediately read back the now-pressed fire button, and the input latency in Mega Man X drops to two frames.

Why Two Frames Still?

If there's no latency, why are there still two frames of lag? This is because the internal Mega Man X game has its own input processing delays. It takes the game two frames to acknowledge the fire button press, update the sprites, and begin to draw the fire animation.

There is a technique known as run-ahead which is far too complicated to explain here, but it essentially uses save states to remove the internal processing lag frames, which can reduce the wait to appear instanteous. Pretty cool stuff.

Why This Doesn't Affect Normal Gameplay

It should be obvious why the first issue had no effect on input latency: the GUI never transitions between paused to frame advance mode during normal gameplay, the game is always running.

But the second issue is a bit more nuanced. The reason that doesn't affect in-game latency is due to bsnes' just-in-time input polling: bsnes does not need to perform a frame event to poll the real hardware and latch the fire button's state, it happens immediately when the game reads it.

Thus, bsnes doesn't care if the PPU is in the middle of scanline 224, or if it has caught up to scanline 225. bsnes doens't care if the CPU is halfway through scanline 225 or not. When the CPU code tries to poll the fire button during NMI, the GUI is notified and polls real hardware right away, and the game latches the current state of the fire button, irrespective of current frame having been rendered onscreen or not already.

Again, this is not the case in RetroArch due to the frontend polling controllers only between frame events, and so RetroArch would see the extra one frame of lag.

And the reason it was also seen in frame advance mode in bsnes is because time was stopped when the emulator was polled. This situation is impossible when playing a game normally: time doesn't stop, giving you unlimited time to then hold down the fire button and then start pressing frame advance.

If you were to be holding down the fire button before pausing the emulator, the fire button would have been latched as set, and frame advance would have shown two frames of lag.

When actually playing a game, time is not stopped, and so you do not have the opportunity to press the fire button at your leisure.

This is a Hack

I would like to stress that this workaround is a hack: we are relying on the video game to poll inputs immediately after the NMI event, and to do so, I have to put the video frame event code into the CPU core where it really doesn't belong.

Even though I cannot name any games that do this, there is nothing technically stopping a game from polling inputs at any other time: a game could very easily set an IRQ for scanline 224 and start polling inputs there, and the lag frame would reappear in both bsnes and RetroArch. Even with the Snes9X port of RetroArch it would appear.

Conclusion: frame advance is not an effective emulator latency testing method

Ultimately, frame advance was not designed to test input latency. In bsnes, it's an almost brand-new feature recently introduced that hadn't really been optimized for input responsiveness. It was an unfortunate oversight, and it's worth using a hack to optimize for the general case (true 99.9+% of the time) where games poll inputs during NMI.

But this stands in stark contrast to bsnes during regular gameplay, where I've spent years on the issue of input latency. Again, this issue only affected the frame advance functionality.

Hopefully this all made sense. Thanks for reading!

Comments

Popular posts from this blog

Hello, I'm byuu

Summers and Bug-Catching