Atari 7800/Lynx headers

[Psychofox11]

This may apply to other systems I'm not aware of as well.

I'd like any feedback on this, and believe it should ultimately result in a standard and wiki entry, but wanted to open this up to thoughts. Discussions have been had on Discord but have been a bit scattered.

For Atari 7800 and Lynx, we currently do not dat headered versions. It should seem easy enough - just add a headered option like NES, right? Here's the issue and why a standard is needed though. Both of these use larger headers that contain text fields like game name, and other 'you can type anything you want here' fields. Someone using different spacing or capitalization even would end up with different hashes.

While many emulators/flash carts recommend no-intro verified dumps, Lynx and 7800 have to essentially warn against them and recommend something else. Keeping headered info will help keep the no-intro hashes relevant as a go-to place for verifying dumps before emulation etc.

For a long time, headers were fairly interchangeable between games, especially of the same size, but aftermarket games have added new complexities to them, needing more specific ones to function. On 7800 this can involve knowing where game data starts, what address a sound expansion uses, bank-switching method, etc.

My proposal: Dat headered versions with the most recent header version (included as file info in hex as well like NES) with any byte (or bit) that is not essential to emulation set to 0. No title, no info etc. - just emulator/flash cart compatibility, and anything not needed for that to work, should be 0.
I'd also love to hear from anyone more familiar with these headers, as creating them can be a bit tricky, I've found especially on 7800 (though not quite as complex as NES).

So, like on NES, if a dev shares a digital with a wrong header, or iNes header, we update it to a proper 2.0 header before datting. The same would apply here. Make sure header is the proper version, blank out any non-essentials. To illustrate, I recently datted headerless versions of two digitals that were datted with headers originally on 7800, both to keep them in line with the rest of the 7800 dat, and because of potential inconsistency. In this case, the 'game title' field was simply 'my game' - no one else making a header is likely to arrive at those same characters. Make it 0s!

While I could probably eventually take on putting together a guide on using all 00's, I don't have the knowledge to ensure I'm making good headers for these otherwise, so they'd likely start as blanked out versions of the headers from other sets (which seem to be easily found for original retail), along with people that know what they are doing making headers for aftermarkets.

In an ideal world, we'd come up with a new header version minus all that, but since emulators/flash carts already have buy-in on this format, it seems this would be easier than trying to force a new standard (though if we did, emulators picking it up would probably happen).

Anyone object? Does this seem like the way to go?

[smesgr]

don't get it. Why want to mess with the headers - does aren't dumped by any dumper I know of. Starting messing with those make it impossible to use hashes to verify file integrity.
If any site, emulator or whatever want to ensure against no-intro they should strip the header and compare the actual dump.

for my point of view: the database should have what it is on the medium. If a flash contains header informations - like for the Gameboy Advance - than the dump should have it too. If a masked ROM or EPROM doesn't contain the information - the dump shouldn't have it.
If the information to generate a file with header should be available at no-intro. Additional information for the cart could be given along side the actual dump or both like encrypted vs unencrypted for DS.

[Psychofox11]

I'm just suggesting we do what we do for NES - datting headers and headered roms helps keep us relevant as a source as many use those, and recoring proper headers is important. This is the only place to find good NES 2.0 headers for many ROMs, especially those distributed with iNES headers.
Several systems record multiple formats, and headered/headerless is the standard for NES. 7800 and Lynx are no different. We just need a standard to go by as we go by NES 2.0, and I'm suggesting one.

I'm not really arguing about whether headered roms should be recorded - that seems decided with NES. They also provide useful information about the ROM (mapper, save size, addresses of certain functions, controller type, etc.). It's also useful to have the file hashes with the header for verification purposes, as that's what many will be using.

Of course I want to record what is on the chip specifically and that's why I'd never suggest NOT recording unheadered first and foremost. You can always ignore the headered version when you export things for your own use, but I think it would be useful to add them, and I don't see convincing the site to remove headered NES, which I wouldn't want.

So, I don't think anyone would be against it - I'm just proposing a standard so that we end up with matching hashes since the header contains fields that aren't used to provide data about how the game runs.

[smesgr]

I'm just suggesting we do what we do for NES - datting headers and headered

you mean headerless and headered? thats fine for me. Only Headered I'm strongly against. I still say headered especially on the 7800 is useless for comparison, because it contains description string and joystick configuration which is up to the 'creator'. It even depends on the header generator 7800header of the 7800basic toolset for example fills the unused part of the description text with spaces while other tools fills those with 0x00. Thus defining a standard here doesn't make it comparable with tools or users which doesn't follow you specified standard.

[Psychofox11]

It's similar to NES - many devs still distribute with old iNES headers. We update them to our standard - NES 2.0 - in order to dat. If you download a kickstarter demo you'll often find its hashes don't match the version datted here due to that. So it would require our own standard (or preferably pointing to an existing one we'll use - perhaps one of the tools you mentioned). Many people also have interleaved TG-16 ROMs that run fine, but we want them deinterleaved as we consider that proper. There's a lot of things that require a standard, but I think having recorded headers would be important. They also include additional info required for preservation as mentioned, that is needed to run them correctly on emulation or flash cart systems.

So just like with some others, no, ours wouldn't match every copy out there - but they'd match ones with proper 'no-intro' headers, and people would have instructions on creating a proper header with us. We DO record headered Lynx right now and with no standard, I agree that it is pointless. We just need to define what bits should always be 0 when contributing here, or which tool should be used.

I also think there's too many formats on Atari Jaguar now. I think at most we should have 'full chip' and 'full chip plus header needed to interpret the data correctly'. The other formats can typically be made with conversion tools. I haven't looked into Jag and how its headers, if it uses them, work, but I don't see the need for us to record so many formats.

I'd never suggest getting rid of headerless, and I'm a big fan of full-chip preservation on original retails (less important to me when a 1MB aftermarket is on a 64MB chip full of FF).

[Psychofox11]

I was thinking, we don't even need a standard, however we should dat the headers along with the file info like NES. Then whoever posts a header there first, as long as it makes sense and is somewhat consistent (either blanked out or proper title/manufacturer) could be the official 'no-intro' header. We just need to add them to the file info like NES. They are a bit bigger and uglier, but it would be the only way to get consistent hashes.

I did message omonim to ask what approach he took for the headered Lynx files datted so I can try to match mine.