In Support of Headered NES ROMs

General No-Intro related discussions.
Post Reply
Buddybenj
Posts: 121
Joined: 29 Jan 2015 23:00

In Support of Headered NES ROMs

Post by Buddybenj »

Headers have long been seen as contrary to the goal of preservation, as it adds data to the ROM file that isn't on the original ROM. (I use header to refer to external headers, not internal headers, which are part of the ROM and should be preserved.) In many cases this is true; we have no purpose in preserving this data. No-Intro has therefore taken the position of removing headers on NES ROMs. I have recently begun to start thinking of this as a mistake (based on my limited knowledge of how NES cartridges work), so I want to document and discuss how to best store NES ROMs. Although I defend the use of headers, I am not defending a particular header format.

First, there are three tenets of No-Intro that, as I understand them, are relevant to this.
1. One cartridge, one ROM file. We do not split ROMs.
2. The ROM file should be sufficient to describe everything about the cartridge's data. One should not have to rely on external data about the particular cartridge to understand or reconstruct the ROM data as it is read by the console. Similarly, if two cartridges correspond to the same ROM file, then their data as it is read by the console must be identical.
3. Extraneous data not on the cartridge that's not necessary for the preservation of the cartridge's data should not be included in the ROM file.

With that said, I'll now consider several proposed formats for NES ROMs.

Split PRG and CHR ROMs: This is a blatant violation of tenet 1.

Concatenated PRG and CHR ROMs: This is what we have now. The problem is that it violates tenet 2. The size of the PRG and CHR ROMs is lost when they are concatenated; the way the data is organized on the cartridge is, which is essential to understanding the data, is lost. People have suggested that if headerless ROMs were to be used by an emulator, an external database would have to be used, containing the information that would otherwise be in the header. But all the data necessary to describe the data n the cartridge should be self-contained in the ROM file. This is like preserving books by stripping them of their cover and table of contents; now you just have a bunch of loose pages.

Split PRG and CHR ROMs together in an archive (such as a renamed ZIP or tarball file): This solves tenet 1 and 2, but tenet 3 is violated. If a header is bad, then this is even worse because now there is much more than 16 bytes of metadata and headers as part of the ZIP or tarball format (and, of course, none of this is part of the original data). Furthermore, these files can produce different hashes even with the same ROM files (a ZIP file can have different compression algorithms and a tarball can have different date modified parameters). We could fix this with a standard that specifies what compression algorithms, date modified parameters, etc. are to be used, but then the convenient appeal of using a renamed ZIP or tarball file is lost.

Concatenated PRG and CHR ROMs with a header: This certainly satisfies tenet 1 and 2, but does it satisfy 3? I argue that it does, because the NES headers contain essential data that should be preserved, unlike the external headers on, for example, SNES ROMs. We shouldn't do away with headers just because the data is not part of the ROM itself, but is rather a representation of how the data is organized on the cartridge.

Any way of organizing the data into a single file will surely be arbitrary. Although such arbitrariness can be standardized (e.g., iNES and NES 2.0), we have no reason to preserve this arbitrariness. So how can I reconcile this with my support for headered ROMs? Although I think headered ROMs are a good way to store the data, I do not think the headered file is what should be datted. My solution is for the DAT to store the hash of both the PRG and CHR. Then, one may store these however one likes, and the DAT would only verify that the PRG and CHR files have the correct hash, regardless of how one chooses to store them.
KingMike
Posts: 696
Joined: 22 Sep 2012 16:36

Re: In Support of Headered NES ROMs

Post by KingMike »

Isn't storing the hashes of both the PRG and CHR basically the same amount of usefulness as storing the hashes of a headerless ROM?
Buddybenj
Posts: 121
Joined: 29 Jan 2015 23:00

Re: In Support of Headered NES ROMs

Post by Buddybenj »

Not necessarily. If I have a headered ROM, but my header is wrong (e.g., the size of the PRG or CHR ROMs could be wrong), then the ROM will verify as good because the concatenation of the PRG and CHR ROMs still matches (this has actually happened to me). But if hashes of both the PRG and CHR ROMs were used to verify the ROM, they wouldn't match. I could also have a headerless ROM that matches the hash, but I wouldn't know where the PRG ROM ends and the CHR ROM starts; I wouldn't have enough information to understand the ROM.

Edit: The more I think about it, the more I like the idea of splitting the PRG and CHR ROMs (even if it goes against No-Intro tradition). But, regardless of which way you lean, hashing the PRG and CHR ROMs separately would be a step in the right direction and compatible with any format you like.
Screwtape
Posts: 34
Joined: 29 Aug 2017 08:46

Re: In Support of Headered NES ROMs

Post by Screwtape »

Buddybenj wrote:Any way of organizing the data into a single file will surely be arbitrary. Although such arbitrariness can be standardized (e.g., iNES and NES 2.0), we have no reason to preserve this arbitrariness. So how can I reconcile this with my support for headered ROMs? Although I think headered ROMs are a good way to store the data, I do not think the headered file is what should be datted. My solution is for the DAT to store the hash of both the PRG and CHR. Then, one may store these however one likes, and the DAT would only verify that the PRG and CHR files have the correct hash, regardless of how one chooses to store them.
That breaks your tenet #2, though. Recording the board layout a game used is just as important as recording the contents of its ROMs, at least for consoles like the NES (and SNES!) that used such diverse boards.

Exactly how you record the header information is a separate debate: you could use NES2.0, but it's a weird format for compatibility reasons, and it has a bunch of fields you might have to explicitly zero out to get reproducible hashes. You could use a Nestopia-style XML fragment, but then you have to legislate things like white-space and character-sets to get reproducible hashes.

> The more I think about it, the more I like the idea of splitting the PRG and CHR ROMs (even if it goes against No-Intro tradition).

I'm guessing that the whole "one cartridge, one ROM file" tenet is a reaction to MAME's "a separate file for every chip" policy, which makes sense for arcade cabinets but less sense for mass-produced goods like the console games No-Intro focuses on. However, "one cartridge, one ROM file" kind of assumes that all the data in a cartridge is accessible as one broad arena; you just scoop all the bytes out of the cartridge in order, and lay them out on disk in the same order, and you have a clean dump. This is true for modern cart-based consoles like the GBA and NDS where No-Intro got its start, but it's not true for older consoles that used more complex memory layouts. The most obvious example is the NES, with its separate PRG and CHR buses, but it's also relevant for things like SNES co-processor firmware. I think a better policy would be "one address bus, one ROM file"
Buddybenj
Posts: 121
Joined: 29 Jan 2015 23:00

Re: In Support of Headered NES ROMs

Post by Buddybenj »

OK, so knowing the size of the PRG and CHR isn't enough to run the ROM? I was unsure if that was the case, but I suppose it makes sense that the mapper needs to be known too. I'm still unsure how the NES "detects" the mapper though (I need to read more about this).

Unsplit ROMs are more convenient for organization than split ROMs IMO. I support splitting ROMs for FDS disks, because the console can only read one side at a time, so it doesn't make sense to include both sides in one file. But for NES ROMs, managing multiple files for one game seems like a pain (I know the goal is preservation, not convenience though). I suppose we could have a third file that points to the PRG and CRG files and includes mapper info, which an emulator would run. But is that any better than just concatenating the three files together to make a headered ROM?
Screwtape
Posts: 34
Joined: 29 Aug 2017 08:46

Re: In Support of Headered NES ROMs

Post by Screwtape »

> I'm still unsure how the NES "detects" the mapper though (I need to read more about this).

For a super-basic example: the NES supports drawing a background made of tiles, and lets you scroll the viewport anywhere in an area two screens wide by two screens high (a total of 4 screens' worth of tile data). HOWEVER, the base NES unit only includes enough tile RAM fo store 2 screens' worth of tile data, and games can choose whether the 2 screens are stacked vertically (making vertical scrolling easy) or horizontally (making horizontal scrolling easy). Specifically, games can choose by physically connecting particular pins on the NES cartridge connector. If you try to emulate a NES game without knowing how it expects mirroring to be set up, you'll get gibberish on the screen, but it's not information that exists anywhere in PRG or CHR ROM, it's the actual wiring of the circuit board itself.

iNES has a special field in the header to record whether the game uses horizontal or vertical mapping.

Later NES games included mapper chips that could control how the mirroring was wired up, while the game was running. This is how Metroid works: whenever you go through a door, the game pokes Metroid's mapper chip, and it toggles from horizontal to vertical scrolling (or back).

It's not entirely clear what the iNES horizontal/vertical field should be set to, when that feature is controlled by the mapper.

> Unsplit ROMs are more convenient for organization than split ROMs IMO.

The fundamental problem is "how do we store multiple byte streams together, so that we can reliably pull them apart when we need to", and this problem has been solved many, many times:

- Directories. Every OS supports them, every file-manager can deal with them. The downside is that most OSs don't support directory extensions the way they support file extensions, so you can't name a directory *.nes and double-click it to open it in your favourite emulator.
- General-purpose archive formats (Zip, tar, 7z, RAR, etc.) Very widely supported, and you can give it a custom extension so it the OS won't automatically treat it like a generic archive (see also: .jar, .docx, etc.) Unfortunately, archive formats tend to support compression which generally has non-deterministic output, which means you can't identify such archives by hash (but see also TorrentZip, MAMEZip)
- Custom archive formats (iNES, RIFF). You can make a format that supports exactly the features you want with none of the mess, but of course you'll need to write special tools to pack and unpack them.

From a preservation point of view, all of the above are pretty much equivalent. From a convenience point of view, they're very different, and any given option might be more or less convenient depending on whether you're talking about "convenient to create and update" versus "convenient to use with existing emulators".
KingMike
Posts: 696
Joined: 22 Sep 2012 16:36

Re: In Support of Headered NES ROMs

Post by KingMike »

Buddybenj wrote:I'm still unsure how the NES "detects" the mapper though (I need to read more about this).
The actual NES hardware itself doesn't even know mappers are there (I think the first game with a mapper was made in 1985, two years after the Famicom was released. And I'd want to say it was a fairly simple CHR swap.)
It's like if you were reading a book and halfway through somebody somehow replaced the pages without you noticing but you just keep right on reading it anyways.

I think TurboGrafx was the first console designed with mappers in mind (as it even built one into the console's standard hardware).
FitzRoy
Posts: 40
Joined: 30 Aug 2017 09:10

Re: In Support of Headered NES ROMs

Post by FitzRoy »

Buddybenj wrote:1. One cartridge, one ROM file. We do not split ROMs.
A cartridge is made of pieces soldered together. If the pieces that contain data are dumped and appended together, it's not closer or farther than say, an archive with those pieces as files inside it. But it does obscure the beginnings and ends of each, and makes the emulator use ordering rules vs filename rules. The rules difference is a wash (emulators will always need a way of distinguishing PRG from CHR), but the need to add values deciphering "where PRG ends and CHR begins" doesn't exist with separation. An actual cartridge certainly doesn't concern itself with joinage points.

Other busted parts of this logic:

1. Why is SRAM written to its own separate "zelda.sav" file instead of at the end of the "zelda.nes" file? After all, the ram chip was just as much a part of the "cartridge" as the rom chips were.
2. If file = cartridge, then emulator = console and should be blind to anything in a cartridge. Yet all NES emulators contain mapper chip and board logic.

When looking at a cartridge, it's important to differentiate between rom data and electronics. I'd say that any attempt to include electronics tags or files with rom data is misguided because of how volatile it is and how much it needs to evolve. It is much, much harder to update roms in circulation than emulators. This is why the ines header format couldn't even effectuate its own 2.0 revision.
Buddybenj wrote:an external database would have to be used, containing the information that would otherwise be in the header. But all the data necessary to describe the data n the cartridge should be self-contained in the ROM file. This is like preserving books by stripping them of their cover and table of contents; now you just have a bunch of loose pages.
This isn't the best analogy. Books don't have gazillions of custom machine underpinnings necessary for legibility. If they did, it would certainly be better to put that in the reader program. Otherwise, there's a much higher chance that the presentation will suffer based on where you downloaded the book. At the end of the day, it's the integrity of the presentation that matters most, not where the pieces ultimately come together.
KingMike
Posts: 696
Joined: 22 Sep 2012 16:36

Re: In Support of Headered NES ROMs

Post by KingMike »

FitzRoy wrote: 1. Why is SRAM written to its own separate "zelda.sav" file instead of at the end of the "zelda.nes" file? After all, the ram chip was just as much a part of the "cartridge" as the rom chips were.
Admittedly one difference is that the SRAM is a writable medium whereas the ROM is not.
Buddybenj
Posts: 121
Joined: 29 Jan 2015 23:00

Re: In Support of Headered NES ROMs

Post by Buddybenj »

FitzRoy wrote:2. If file = cartridge, then emulator = console and should be blind to anything in a cartridge. Yet all NES emulators contain mapper chip and board logic.
The file does not equal the cartridge; the file simply corresponds to the data particular to that cartridge. So it does not follow that the emulator equals the console. It's fine for the emulator to emulate the mapper ASIC (and I think that's how it should be). But since the file corresponds to the data particular to the cartridge, the file must indicate the mapper to the emulator.
Post Reply