NES ROM format - the final decision

[BigFred]

Now that most games have been found, dumped and many been verified the biggest run to solve the horrible NES-chaos is almost over. We still have one problem - the header problem. I guess many people are fed up dealing with messy crc or even size-changes disturbing the process of accurate verification and leading to unplayable games because of wrong settings with lots of different random-headers floating around the internet.

The question is whether we should put the header into a seperate file (could be scanned by rom-managers) or create a database which can be used by emulators and serves as a global reference for changes. Second question is if the PRG and CHR should be seperate roms similar to mame or merged like now. Everything else like the structure of the database for instance will be discussed afterwards.

[ElBarto]

I've talk about that about 2 or 3 years ago ...
The PRG and CHR roms HAVE to be separated, the problem is emulators.
I think nestopia could handle separated roms but I don't know about the others.

The others problem is if someone doesn't "distribute" nes roms separated none of the emulators authors would support this "kind" of roms.

[xuom2]

the problem is emulators.

we are not here to play roms

[NGEfreak]

All required header information should be put into one external database file (like NEStopia's XML database). This database can then be created by DOM. Also, to support the spread of pure headerless ROM images, the CMP NES header support should be removed from the datfile.

I don't see why the PRG and CHR have to be seperated. The principle of 1 media == 1 image == 1 file is much more trouble-free to implement in a database than having multiple ROM images per media. Multiple ROM images just lead to another n:m relation that could be easily avoided.

[Connie]

I agree with NGEfreak because regarding emulation, nestopia is quite capable of setting up/editing an iNES header.

However, to maintain the "No-Intro headerless ROM standard", where would this leave other sets utilising the 'header skip' function such as Atari - Lynx and Nintendo - FDS?

[BigFred]

I think we all agree headers have to go. So this step will be done for next dat. If we use seperated PRG/CHR or not we can decide later on. If we go for split ROMs we can introduce a tool to do this for us.

The more important matter is handling the extra-information:
I also think an xml-database is the best solution for our needs which contains checksum infos as well as all the required information currently covered by headers. This database has to be accessed by emulators and should definitely not be built-in. Now we have to find a structure satisfying all needs and convince emulator-authors of this. We might also need to introduce a new extension for the headerless files.

I am aware Nestopia handles NES XML-files. I'm not sure how this is supposed to work but it doesn't appear to be the same approach we are aiming for. Anyone willing to explain how it works?

An other option is to distribute the rom-files with a seperate XML containing all extra-info. But I like the approach of a global database more.

[BigFred]

Ok, so it looks like Nestopia uses an XML-database co-developed by bootgod - exactly what we are looking for. Unfortunately the database has been built into the .exe and can only be viewed in the source-code package. The second problem is that Nestopia still does not load headerless files. It uses the database to overwrite the ines-info internally. We need to contact the developers to fix these issues. We definitely need a new file-extension for headerless dumps which can then easily be accessed by emulators without confusion.

[Connie]

For the purposes of emulation, an option could be to create a 'generic' 16 byte header that could be used in conjunction with

Code: Select all

copy /b header.bin+pgr.bin+chr.bin romname.nes

which would at least allow users to load and then fix the rom header in for example Nestopia.
In fact, a much simpler and better sollution:
considering ~2500 dumps,
2500*16 bytes = 40,000 byes = ~40kB
Couldn't we just supply a <40kB zip file of headers with the dat?

Otherwise, I think that documenting the header structure is going to be a necessary step to account for dumps that arn't yet part of any existing 'database', be it web sites or emulators. I'm not sure how far you'll get with having emulators re-written and integrated databases updated.

[ElBarto]

You can't supply a generic header cause the mapper type and CHR/PRG size are in it.

I just wrote a (crappy) simple tool to separate Header/PRG/CHR from an ines rom.
I've calculated also the number of the roms that are the same (for PRG and CHR)
There are 8 same PRG roms and 358 same CHR roms.
You can found the result and the tool here : http://www.megadrive.org/~elbarto/NES/ (I've no windows to compile it, if someone could)

I haven't look yet at the Nestopia db format, could it use separated PRG/CHR roms ?

[BigFred]

From what bootgod told me the new XML-format in Nestopia uses a zip containing said XML as well as seperated PRG and CHR. The zip may contain several versions of the same game and they share ROMs they have in common. Unfortunately this means we'd need to dat hundreds of single XMLs with random crcs. What do you think about this?

My idea would be that Nestopia or whatever emu uses a priority-system. If an XML is present in the zip it will use this. If not, it checks the global XML-db. This way you could easily release hacks and translations of a game without flooding the global db with hacks.

Thank you for the PRG/CHR comparison. Super Mario Bros. 3 (Japan) is indeed a fake and will be removed from next dat. Which dat did you use anyway? Mahjong (look at PRG-list) should have been removed in this release bacause it's also fake.

[NGEfreak]

A central database is better. Additions, corrections or even a revision of the database format can be much easier distributed.

A priority system would be the best. I think a good order would be:
1. XML database in archive or working directory
2. global XML database
3. copier header
4. hardcoded fallback database

However, to maintain the "No-Intro headerless ROM standard", where would this leave other sets utilising the 'header skip' function such as Atari - Lynx and Nintendo - FDS?

They just need to implement a similar system.

We might also need to introduce a new extension for the headerless files.

No, a new extensions is unnecessary. Why does it matter what extension a file has? It should be allowed to name them with any extension or even no extension at all. Wildcard support ftw!

[BigFred]

bootgod gave me some interesting insight:

Nestopia does implement a priority system just like you mentioned. It has it's internal database with the minimum details of getting the game running. You can load an external XML from my DB, which will take priority over any duplicate hashes. Then finally if you load a ROM in this format, it's XML will take priority over all others.

Now that's just what we want. How the loading of an external XML is supposed to work has to be clearified. Don't know if it is documented somewhere. And as mentioned before an option to load a headerless file needs to be implemented. Otherwise we'd still have to stick with ines-headeres files or we'd be forced to use single XML files for each game since Nestopia only allows the loading of the XML which is part of bootgod's new format.

[BigFred]

It seems loading an external db is simple. Options - database. That's it. The structure of this database can be viewed from the internal database accessible by downloading the sourcecode package. Generally it looks like this:

Header info:

<?xml version="1.0" encoding="UTF-8"?>
<database version="1.0" conformance="loose">

Game entry:

<game>
<cartridge system="NES-PAL" dump="ok" crc="001388B3" sha1="4BCD36C05FCAF45C74001257C65AFB7EC5FA53D7">
<board type="NES-TLROM" mapper="4">
<prg size="256k" />
<chr size="128k" />
<chip type="MMC3C" />
</board>
</cartridge>
</game>
<game>

I'm informed these XML files can all be created from bootgod's db and the nessecary software will be up within few weeks.

bootgod is undecided about the loading header/XML-less files. We need some good and convincing arguments

[Connie]

If we have access to this xml database, then surely it's as simple as writing some code which could recreate the iNES header?

These files could theoretically be generated by a "header-o-matic" (c)No-Intro, or simpler still, a temp archive could store the 16byte files and be 'compiled' into a zip archive based on the dat settings used during creation of the dat.

Yes, it means we're 'stuck' with the iNES format, but why make it difficult? The current set is primarily based on this format anyway. Since the No-Intro goal here is just about how we dat the ROM data, why are we bothered about headerless NES dumps for implementation into emulators? Those interested purely in emulation arn't going to be stripping away their NES ROM headers anyway and are therefore unlikely to ever use a dat format along the lines of:

Code: Select all

game (
	name "8 Eyes (U)"
	description "8 Eyes (U)"
	rom ( name "PRG-ROM.bin" size 131072 crc CB475567 md5 E4C062BE220D5B00C82371D8F7B9679E sha1 AF33F817CDD3C5B3D2860CA320AA3B8C4BB4E1E5 )
	rom ( name "CHR-ROM.bin" size 131072 crc DF52CED7 md5 DE996B2E54516DFFFC52CBCB4EC3FAF0 sha1 5EFAA966D02290EA1E791C31F6947864899BC342 )
)

[FitzRoy]

We were recently discussing something similar with bsnes because the author doesn't like constructing hypothetical board mappings from the rom's internal header (which NES games don't have). The best solution I could think of was indeed a single database file in the emulator directory. It would be a simple CSV (comma separated values) file with each entry being "name(ignored)","size","sha256checksum","pcbserial". The emulator would scan the loaded rom's size and checksum, scan for a matching entry in the CSV file, and use the entry's documented mapping. Easy to read file, easy to create and make changes to, avoids the mess of having thousands of individual headers maintained on the rom end.

For the NES, a lot of boards did not have serials, so we would have to assign them prison numbers like iNES did.