[SMD] Header size vs. ROM size and the purpose of overdumping?

General No-Intro related discussions.
Post Reply
Vigi
Dumper
Posts: 177
Joined: 18 Jun 2008 10:16

[SMD] Header size vs. ROM size and the purpose of overdumping?

Post by Vigi »

Hello,

today I submitted a dump in dat-o-matic that was "trimmed" to 1MB, even though it was stored on a 4MB ROM chip.
I thought it made complete sense to trim it, because it was stored on a 4MB chip on a generic aftermarket PCB without a mapper called "DragonDrive", which is commonly used for homebrew and aftermarket stuff: https://www.dragonbox.de/en/homebrew-pr ... -with-sram
When checking the ROM, the data ends exactly at 1MB, with the next 3MB only filled with 0xFF bytes.
The "ROM address range" stored inside the ROM header indicates a 1MB size.
On top of that, the version of the game that was released on Switch and PS Vita was also a 1MB file.

Before long, I ended up in a somewhat heated discussion with Hiccup, who insisted on getting the hashes for the "complete" 4MB ROM dump.

This is prompting me to start a new discussion about the exact purpose of dumping the full ROM range / chip contents as opposed to the correct address range that is defined in the ROM header.

For example, let's take "Batman Forever (World).md". The dat-o-matic is riddled with "bad" dumps that were 3MB in size: https://datomatic.no-intro.org/index.ph ... =32&n=0170
When comparing the "bad" dumps to the new "good" one, the new dump has 1MB of 0xFF bytes added.
The ROM header indicates that 3MB is the correct size.

So I'd like to know:
- What is the benefit of adding empty (0xFF) padding/data to a dump that isn't addressed by the game itself, and has no preservation value?
- Why not preserve/document unused data and the chip capacity, in case it mismatches the correct ROM size, by making a comment about it in the dump entry, rather than appending this useless data to the dump?

When checking the full current SMD set, out of the 2554 ROMs, 2128 have the size that is reported in the ROM header.
426 have a different filesize than is indicated in the ROM. The majority of these are (Unl), (Beta), (Proto) or (Aftermarket).

Attached an Excel sheet that shows the differences.
You do not have the required permissions to view the files attached to this post.
sCZther
Datter
Posts: 109
Joined: 16 Jun 2014 21:09

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by sCZther »

Not sure how common it would be with SMD, but usually there could be additional "garbage" data in the rest of the chip. 0xFF of course isn't interesting, but especially with pre-release roms there could be unoverwritten stuff (that's how we got some beta Zelda 64 content this year).
relax
High Council
Posts: 893
Joined: 27 May 2008 17:52

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by relax »

It's a discussion whether to preserve the games or preserve the media. Neither perspective is wrong. No-Intro's goal, starting with physical games, has been to preserve the media, not just the games. Meaning the full ROM chip size. Like redump.org don't scrubb their optical disc images.

It can be discussed what is the best practice. In some cases, like with the magnetic disks found in FDS, I think preserving the disks is futile, as the dumps can't be reproduced.

Many of SMD ROMs in your list are beta ROMs coming from developer backup archives. I don't see a reason why we should modify these.
omonim2007
Datter
Posts: 365
Joined: 20 Jul 2016 12:20

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by omonim2007 »

I'll add some information to the last comment.

Let's first define the concept of a dump itself (since we are using this concept). A dump is a digital copy of a physical data storage (cartridge, memory card and so on). So it is a piece of data that is identical in all respects to the parent device from which it has been copied. Therefore the information is stored in the database in the form as it written on the physical chip.

We cannot consider the dump to be correct (equal to what is in the physical chip) if we remove empty information from it.

An exception can be made only for games that were released first on some types of PCBs (for example, on two chips, summering 786 KB), and then on others (one chip, 1 MB). There are quite a few such examples (SMD, SNES and so on). However, if the game was published on media of a larger volume, then we save a digital copy of this version only (in this case, it does not matter at all how much in % game data take up on the media).
Vigi
Dumper
Posts: 177
Joined: 18 Jun 2008 10:16

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by Vigi »

omonim2007 wrote: 13 Apr 2021 05:27 I'll add some information to the last comment.

Let's first define the concept of a dump itself (since we are using this concept). A dump is a digital copy of a physical data storage (cartridge, memory card and so on). So it is a piece of data that is identical in all respects to the parent device from which it has been copied. Therefore the information is stored in the database in the form as it written on the physical chip.

We cannot consider the dump to be correct (equal to what is in the physical chip) if we remove empty information from it.

An exception can be made only for games that were released first on some types of PCBs (for example, on two chips, summering 786 KB), and then on others (one chip, 1 MB). There are quite a few such examples (SMD, SNES and so on). However, if the game was published on media of a larger volume, then we save a digital copy of this version only (in this case, it does not matter at all how much in % game data take up on the media).
But what do you do for those exceptions? Add both dumps separately and mark one of them as (Alt)?

As I pointed out to Hiccup, I think the project took a turn towards chip preservation instead of game preservation only in recent years, after MAME and byuu started preserving cartridges this way. No-Intro back in the early days always used to trim dumps and leave out any unneeded data.
Now, because the vast majority of the dumps have no documented chip sizes, you have a lot of trimmed dumps probably that are not dumped according to your new definition.
Another thing is that you're not really preserving the storage medium because the dumps are all combined. So the current set is somewhat of a frankenstein set composed of old trimmed dumps as the base and and any "bad" dumps according to your new definition, replaced in case MAME, byuu or someone else redumped them and the old dump turned out to be trimmed. This leaves me to wonder how everybody feels about this question:

- Why not preserve/document unused data and the chip capacity, in case it mismatches the correct ROM size, by making a comment about it in the dump entry, rather than appending this useless data to the dump?

I think this solution would give the proper balance between clean dumps with their true intended size and would still preserve, albeit in text, the original storage medium.
rwebster
Posts: 25
Joined: 01 Mar 2021 18:15

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by rwebster »

Not sure if worth thinking about but I thought it was relevant.

ROMs rereleased in digital form are sometimes trimmed. This is the case with a few on the Mega Drive Mini. Specifically Wonder Boy in Monster World off the top of my head.
omonim2007
Datter
Posts: 365
Joined: 20 Jul 2016 12:20

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by omonim2007 »

But what do you do for those exceptions? Add both dumps separately and mark one of them as (Alt)?
No, the first one (the smaller one) becomes the main source, and the other one is marking as an overdump, so nothing special.
This leaves me to wonder how everybody feels about this question:

- Why not preserve/document unused data and the chip capacity, in case it mismatches the correct ROM size, by making a comment about it in the dump entry, rather than appending this useless data to the dump?
But what is "the correct ROM size"? ROM = Read Only Memory, which relates directly to the physical medium.

There is no problem for emulators to play games in their real size according to the size of the memory chip(s), and trimmed versions plays well too. But in doing our work, we save the information as it really is. Why are you so concerned about this question?
User avatar
dreimer
Posts: 154
Joined: 14 Nov 2015 13:26

Re: [SMD] Header size vs. ROM size and the purpose of overdumping?

Post by dreimer »

I think the main reasoning here is simple and explained quickly. As already mentioned above especially on Beta carts there already were crazy things found after the ROM itself as the cart is not wiped completely on reflash sometimes. So trimming by yourself is not the way to go. Sure this might not be a old beta cart, but the Dragondrive can be seen similar to one as it's flashable by the ROM maker or whoever else. Dump the whole thing and let the datter analyse it. Sure he will find billion FF in the end, but then we all are happy. I for myself always dumped what was on there or in VC titles or whatever release form we have nowadays. Of course I stumbled upon the typical "Yeah your dump does not fit the db checksum as it's overdumped.", but better that way than missing interesting preserveable stuff later on. The guys here know what they are doing, so just do as you are asked to and all is fine. Some extra information like "Yo, might be an overdump as the header says so" is always welcome of course.
Post Reply