Thursday, April 9, 2020

The NES and Famicom Accurate Cartridge Information Database

NES and Famicom emulation has been around for over twenty-five years.  In that time, the internal hardware has become very well documented.  NES and Famicom cartridges, on the other hand, have had a parallel journey of discovery during this time, but emulators and flash carts and FPGA devices have not always been up to date with current developments.  The core games which people enjoy with NES emulation, namely those licensed and approved by Nintendo and unlicensed games released during the NES' lifespan, sometimes suffer in emulation due not to bad dumps but a wrong information in their file header.  The header indicates what kind of hardware the game uses, but if the information in the header is wrong, out of date or missing, the game will not play or play correctly.  In this blog article I will explain how headers work, why they are necessary, the need for accurate information in them and how they have evolved over time.  Then I will describe and link to my database which contains the most accurate and up to date information for the NES and Famicom ROMs most people care about.




A Brief Overview of NES Cartridge Hardware

The NES and Famicom w designed to handle cartridges which contained two ROM chips.  The first, called the Program ROM (PRG-ROM) contained all the code, data and music for the game.  The PRG-ROM was connected to the CPU bus and the CPU could normally address 32KiB of PRG-ROM at a time.  The second, called the Character ROM (CHR-ROM) contained the graphical tiles used by the game to display its graphics.  CHR-ROM is connected to the PPU bus and the PPU can normally address 8KiB of CHR-ROM at a time.  At the time the Famicom was released, the memory system could support up to 32KiB of PRG-ROM and 8KiB of CHR-ROM.  (1KiB = 1024 bytes)  That was considered "the limit" and the Famicom Disk System was introduced to help overcome those memory limits.

Cartridges had many advantages over the disks, and cartridge development continued to overcome the memory limitations.  The NES never received the disk system, so by 1985 the 32KiB/8KiB limit was a straitjacket for larger games and greater innovation.  Memory mapping schemes were introduced to allow the NES to transfer or bankswitch portions of ROM to its CPU and PPU.  At first the schemes tended to rely on off-the-shelf logic chips to allow for simple PRG-ROM or CHR-ROM bankswitching (or both).  Some of these methods allowed a game to use only a larger PRG-ROM chip whose code would instruct the CPU to copy graphics data from it into an 8KiB Character RAM chip (CHR-RAM) connected to the PPU bus.

By 1987, the NES and Famicom were sufficiently profitable for companies to develop specialized ASIC chips, like the Nintendo MMC and Konami VRC series of chips, to address far more memory than ever thought possible in 1983.  With these advanced memory mapper chips came the ability to add extra RAM, called PRG-RAM, for the CPU to use.  The NES only has 2KiB of RAM for its CPU to store data, variables and its stack, so some games needed an extra 8KiB for their data.  Some games like the Legend of Zelda added a battery to keep the PRG-RAM contents saved while the game was not being played, allowing for saving games without passwords.

Similarly, the graphics capabilities of the NES were improved by the advanced memory mapper chips.  Each screen is made up of background tiles and the data required to store the tile map requires 1KiB.  The PPU can potentially select from four unique screens, but only has 2KiB of internal Video RAM (V-RAM), so it is in effect limited to two unique screens with the other two screens "mirrored".  The mirroring arrangement of the V-RAM can be arranged by cartridges to optimize the V-RAM addressing for horizontal scrolling or vertical scrolling, and in early cartridges the arrangement was hardwired on the cartridge PCB.  More advanced mappers allow the mirroring to be switched by the CPU instantaneously, allowing for easy scrolling in both directions.

Advanced memory mappers can allow for other features, such as cycle or scanline counters to allow for timing for graphic effects and menu bars, the ability to show more tiles or more colors for the tiles, and with the Famicom to add extra audio and mix them in with the internal audio.  There are dozens of memory mapping schemes Nintendo and its Japanese third-party licensees used from 1983-1994.  Then unlicensed game manufacturers used their own methods and later homebrew authors devised their own boards and methods, leading to hundreds of different memory mapping schemes, "mappers" for short.

Why Headers are Necessary for ROMs

When you dump a cartridge's ROM(s), you read all the data off the ROM chip(s) and store it in a binary file on a computer.  Without reverse engineering the game code, those ROM dumps do not tell you anything about the hardware inside the cartridge.  Early dumping hardware for NES and Famicom cartridges was often home-made or used copiers that copied ROM dumps to Famicom Disks System disks.  In the mid-to-late 1990s, a flood of dumps for NES games appeared.  At first, games tended to have a split ROM format with to separate files named like "SMB.PRG" and "SMB.CHR"  An emulator like Pasofami would use a third file to describe the hardware.

During those early days, the community was just figuring out how many mappers there really were.  Another early emulator, iNES by Marat Fayzullin, used a ROM format which copied the PRG and CHR data into one file and added a 16-byte header to describe the game's hardware.  The iNES format was somewhat limited, it could indicate the amount of PRG-ROM, the amount of CHR-ROM, the mirroring type, whether there is battery backed RAM and if the game is intended for the Vs System.  Originally it only used four bits to determine the mapper, but it was quickly bumped up to eight bits when the dumpers found out that the diversity of NES mappers was far greater than was originally thought.  So the official iNES 1.0 specification allows for 256 mappers to be defined and designated and was widely supported in popular emulators like Nesticle and was pretty much acknowledged as "the standard" by the year 2000.

Without a correct header, a ROM may not function correctly or at all.  Many ROMs have been distributed over the decades with inaccurate or junk data in their headers, even if the data contained within the PRG and CHR is good.  Eventually ROM archiving tools like GoodNES and later No-Intro were released to help distinguish files with original, uncorrupted PRG and CHR data from hacked, pirated and corrupted releases.  Those ROM archiving tools do not determine whether the header contains correct information or bad information.

By the mid-2000s, it was widely recognized that the iNES format was just too limited to describe the vastness of the cartridge hardware that had been released and was still being released.  The 256 mappers designated by the iNES 1.0 standard was rapidly being exhausted as new hardware designs and incompatible hardware variants were being discovered.  Emulators could use file hashes which would compare a ROM's hash with a hash in a database to figure out how to run the game.  But hashes are only as good as the emulator is updated to support them.  A translation patch or a hack to a game will break the hash and a system that wholly relies on hashes.  A flash cartridge may not have the computing power to compare hashes in an expedient manner.

While the UNIF specification was one way to describe hardware, it never quite caught on for various reasons.  NES 2.0 was a header format created by kevtris in 2006 which was backwards compatible with iNES 1.0 but used unused bits and bytes in the header field to define more information about the cartridges.  Kevtris's specification allowed the header to designate (officially) if it was a PlayChoice-10 or Vs. System game, the ability to define up to 4096 mappers, the ability to designate submappers (16 per mapper), to support larger sizes of PRG-ROM and CHR-ROM, to define the amount of battery backed RAM and non battery backed RAM a cartridge has, whether the game requires an NTSC or PAL PPU and the copy protection method on Vs. System games.

Later developers from the NESDev community (lidnariq, tepples, rainwarrior, NewRisingSun) expanded on the NES 2.0 format to note miscellaneous ROMs a cartridge may have which cannot be characterized as PRG-ROM or CHR-ROM, to define unusual, small and large ROM sizes in an alternate manner, to indicate that the ROM requires the capabilities of an enhanced NES-like clone such as the VT-0x series, and to tell the emulator what expansion device it uses.

My Solution to the Problem of Wrong Headers

Despite the enormous amount of work devoted by the community in the past quarter century, I still regularly read of people who cannot get game x working on their emulator or their EverDrive because they are using a bad ROM, even one in the No-Intro set (which only defines "good ROMs").  The people behind the no-intro project are indifferent to headers, so a set you may find on the internet is not guaranteed to include an accurate header or even any header.  Emulators will almost certainly will not recognize a NES ROM without a header.  I cannot program a tool that will turn bad headers into good ones, but I can provide the information needed to inform everyone what the correct header should be, at least for the ROMs that 98% of the community cares about.

My focus has always been on "the licensed set".  My spreadsheet includes all known good dumps of games licensed or authorized by Nintendo to be released in North America, Japan, South Korea, Australia, Brazil and Western Europe.  It also includes unlicensed games known to have been released in those regions of the world from 1986-1995 (and perhaps later).  It does not include Taiwan games not known to have a release in those countries, pirate multi carts, Chinese "originals", homebrew, hacks or translations.  It also does not include releases on the Virtual Consoles, Nintendo Switch or NES Classic unless those releases have some unique improvement to them.  It does include official re-releases with some improvement to them.  It does not include unplayably broken games or even official hacks which only work on emulators.

The idea is that if you encounter a ROM which is not working, you can reference the header and CRC32 information in the spreadsheet and use a recent emulator with NES 2.0 editing abilities like Mesen, NintendulatorNRS to adjust the header parameters to what they should be.  You can determine the file size of any game as well, just add the PRG-ROM and CHR-ROM (if any) values together, then multiply by 1024 and then add 16 bytes for the header (and 8,224 bytes for PlayChoice-10 ROMs).

Why use my spreadsheet?  I have checked my assignments with the best sources available, the NES Cartridge Database, the official Nintendo factory spreadsheet, the NESDev wiki and with discussions with the most knowledgeable experts in the NES emulation scene.  I have spent countless hours testing games, trying to track down board shots, fixing errors and otherwise making sure everything is where it should be.

The iNES 1.0 format is hopelessly dated, it cannot hope to accurately describe cartridge hardware.  NES 2.0 can describe cartridge hardware accurately, and not only what a cartridge contains but what a cartridge does not contain.  Not all games need NES 2.0 headers, most games in the licensed set will work with 1.0 headers, but having 2.0 headers for all makes for a consistent approach to the problem.  Some cartridges will not work if PRG-RAM is assumed, as is the case with iNES 1.0.  With a proper database, there is no reason to use file hashing to figure out how a game works.  It also can serve as a decent substitute for the NES Cartridge Database, which is hosted on an individual's computer and has a habit of going down at the most inconvenient times.

Remember that if you convert a iNES 1.0 header to a NES 2.0 or adjust a NES 2.0 header, an emulator, flash cart or FPGA console may not assume anything about a NES 2.0 ROM.  You may get garbled graphics, probably because you failed to indicate that a CHR-ROM-less game had 8KiB of CHR-RAM.  Your game may crash because you did not set the right amount of PRG-RAM or may show weird glitches when scrolling because the mirroring bit is opposite of what it should be.

Key to the Spreadsheet

Game Name - Self-explanatory, the No-intro names are used unless the ROM is not in the No-intro set.  ROMs not in the No-Intro Set will use a GoodNES name if available.  PlayChoice-10 names are taken from GoodNES and Vs. System names are taken from MAME.  The sheet for Japan also has the Japanese names.  I segregate ROMs into several categories.  The first lists the standard list of games you could actually have bought in a store during the system's lifespan.  If a game has more than one version or revision, the newest revision is listed here.  Then I list all older versions of games in the next section.  Finally I put "special games" in their own section.  This section includes updated re-releases (during the console's lifespan), official multi-carts, competition and prize cartridges and other unusual variants.

PRG-ROM - The amount of PRG-ROM a game has in kilobytes (kibibytes).  Galaxian uses only an 8KiB PRG-ROM and requires NES 2.0's alternate size designation system to have its size defined properly.  The Vs. System uses some unusual PRG-ROM sizes because most of its games came as a varying number bundle of 8KiB EPROMs that fit into sockets on an arcade PCB.

CHR-ROM - The amount of CHR-ROM a game has in kilobytes (kibibytes), if any.  A few games use CHR-ROM and CHR-RAM, but most games use one or the other.

Misc ROM - The number of miscellaneous ROMs a game has.  In my Database, this is relevant only for PlayChoice-10 ROMs, which always have three miscellaneous ROMs: an 8KiB Instruction ROM, a 16 byte PROM and a 16 byte Readout.

Mapper - The mapper number and submapper (if any) the game uses.  NES 2.0 is required for submappers and for mappers numbers greater than 256.

Mirror + Flag - Indicates whether the game uses Horizontal mirroring (H), Vertical mirroring (V) and whether the game has non-volatile, usually battery backed, memory (B).  Many mappers can adjust the mirroring on the fly or use one-screen mirroring and do not care how the mirroring flag is set.  A few ROMs, except for the Vs. System, use four screen mirroring (4).

PPU - Determines what PPUs a game is intended use : NTSC (2C02), PAL (2C07), PlayChoice-10 (2C03B), Vs. System (2C03B, 2C04-1, 2, 3, 4, 2C05-1, 2, 3, 4).  Vs. System games will show bizarre colors if the wrong PPU is selected and those games which use 2C05s won't start at all.

PRG-RAM - The amount of non-battery backed RAM the game has.  Does not include any RAM contained in a mapper chip.

PRG-RAM B - The amount of battery backed RAM or EEPROM the game has.  Sizes in kilobytes indicates SRAM is used, sizes in bytes indicates EEPROM is used.  Also not include RAM contained in a mapper chip., even if battery backed.

CHR-RAM - The amount of CHR-RAM the game has.  Rarely more than 8KiB, but recent releases like Holy Diver Collector's Edition and Metal Storm Collector's Edition have a large amount of CHR-RAM because their hardware copies graphics data from the large PRG-ROM chip to the CHR-RAM chip and then disable the Write Enable on the CHR-RAM, making it function like CHR-ROM as those games originally had.

CHR-RAM B - The amount of battery-backed CHR-RAM the game has.  In this Database, battery-backed CHR-RAM is only used by Racermate Challenge, which uses part of its CHR-RAM to save lap times and such.

Console - Indicates whether a normal console, a PlayChoice-10 or one of the Vs. System configurations is intended.  For the Vs. System, this will indicate if the Vs. Dual System is required (two monitors and two Vs System CPUs and PPUs communicating with each other) or if the game uses a special kind of protection.

Controller Type - Indicates if an expansion controller is supported.  Typical controllers include the Zapper, the Power Pad (with selections for each side), NES four player adapter and their Japanese counterparts.  This is mainly useful for emulators.

CRC32 - Gives the CRC32 hash of the combined PRG-ROM and CHR-ROM (if any).  This is NOT the CRC32 of the file, the header is not included in the hashing.  PlayChoice-10 CRC32s include the content of the miscellaneous ROMs.

Outstanding Issues - If there is more data to dump from a game, such as is the case with the Famicom games which use speech chips, then this will be noted.  Also, I have noted one instance where further knowledge of a cartridge's hardware is required

BIOS - This sheet is very basic, these files are typically not used as headered NES ROMs even though most could conceivably run as NES ROMs but would be fairly useless.  I have only included BIOSes and such that would have been available in countries where the NES or Famicom was officially released.  The Study Box cassette loader was assigned Mapper 186 but is useless without the emulation of the cassette loading hardware.  Similarly, the Sharp Famicom Titler program uses Mapper 1 but without emulation of the drawing pad and extra buttons of the Titler console, its functionality is constrained.

What is Included?

I ended up defining my goals when I made my pack as including : 

1. all games released in the 72-pin format during the calendar years 1985-1995, 
2. all 60-pin games released in Japan from 1983-1995,
3. all 72-pin games released by a company which released games before 1995,
4. all Famicom Disk System games released between 1986-1992,
5. all licensed games for the Vs. System and PlayChoice-10, 
6. all known revisions, variations, prototypes, betas, sample games, expansion packs of the above and test programs released between 1983-1995,
7. all authorized multi-cartridges, contest games and educational software made by a company that falls into any one of the categories above
8. any game re-released in either a physical cartridge or digital or extractable ROM format after 1995 from or with the permission of a company or its lawful successor and offers some improvement or aesthetic difference compared to the original game
9. no obvious piracy either with single releases or multicarts or hacks, with certain exceptions

Conclusion

My database is not comprehensive, nor is it intended to be comprehensive.  It is intended as a guide to help people get their games working.  But even within this limited database no emulator, no flash cartridge and no FPGA console can run everything contained perfectly.  These assignments are based on most accurate accurate and up to date information available, so anything else has a hard task to show it is more accurate.

Bold claims of "I can run everything" are made
With only the popular games tested and played.
Until your creation can run all the files herein,
Your claims to accuracy are looking weak . . . and thin.

Here is the link to the database, I hope it helps.

5 comments:

  1. Nice work, I don't do nes development, but I was thinking, is it possible to code some heuristics to scan PRG, and maybe derivate some data from it?, as possible memory addresses access? or something like that? This don't imply that I think your DB is not ok, but I was wondering if this was possible in some way?
    regards!

    ReplyDelete
  2. Typo:
    CHR-ROM - The amount of PRG-ROM

    ReplyDelete
  3. Database URL points to a file that has been deleted

    ReplyDelete