[FDS] The state of FDS dats

[prominos]

I have recently attempted to verify my FDS dumps against the FDS dat/DoM and I have to say it's a pain in the butt. I am wondering if there's a better way to dat FDS dump?

Let's be realists in 2023 chances are if you buy an FDS disk you're going to get a used disk (unless you're willing to pay a huge amount of money) that could have been rewritten multiple times so goodbye verification.

Don't get me wrong I think it's absolutely a good idea to dat sealed dumps, however, I am wondering if two sets could be created for FDS (2 different dat files)

• Set 1: Sealed FDS dump: The cream of the crop unmodified disks.

• Set 2: Normalized FDS dump: Used disks, the dump is post processed through a tool to "Normalize" it essentially nullifying variable fields in the header and resetting/removing save files.

I've read the dumping guide and I have to say the section about verification leaves a sour taste in my mouth:

Verification with FDS games is difficult because the hashes never match up due to the way FDS files work and are created. FDS headers contain a large amount of information that can be different from dump to dump despite the game being identical. Rewrite dates, or dates they were created along with the fact that disks get modified when read for the first time in some cases can make verification quite a chore. However to be sure it's best to submit them via the No-Intro forums and see if someone there can take a closer look at the actual game files to ensure they look correct.

I think there should be an automated way to verify dumps without involving the actions of a human being. Plus I don't think anyone likes to dive into hex dump to verify a thousand dumps manually. So that's were my idea of set 2 comes in.

Personally I want to verify two things:

• Is the actual game data/program correct? (no corruption) I don't really care about the metadata in the headers unless it affects the way the game is played or loaded

• Will the game run in an emulator. If it can run in an emulator then I'm fine with it.

Here are some thoughts about how a "Normalized" FDS dump could be achieved. I'll be referencing these sites to discuss the different fields in the headers:

• FDS file header: This is 16 bytes added to the top of the FDS file by the dump tool (FDSStick) it's similar to the iNES header for NES dumps. It's much simpler tho since the only information it adds is the number of sides on the disk: https://www.nesdev.org/wiki/FDS_file_format

• FDS Disk Format: This is the actual format on the disk created by Nintendo. It contains a large amount of information that can vary from disks to disks and thus mess up verification: https://www.nesdev.org/wiki/FDS_disk_format

Here's how I propose to normalize an FDS dump:

• The FDS header can safely be removed if it is present in the first 16 bytes of the dump. The rom in the DoM right now are headerless

• Manufacturing date (3 bytes): Can be different from disk to disk, reset to 0

• Offset $27 of block 1 (5 bytes): This is listed as "Unknown Speculative: some kind of game information representation?" In some pages of the DoM in the comments I've seen these bytes labelled "Disk ID" these seem to be different from disk to disk so I suggest resetting to 0

• Rewritten Disk Date (3 bytes): This is the date the disk was last rewritten at a Nintendo kiosk. If never rewritten it should be the same ad manufacturing date. This can obviously change and should be reset to 0

• Disk Writer serial number (2 bytes): When a disk is rewritten at a kiosk the writer puts its id number here. When never rewritten it's equal to 0xFFFF. I think it should be reset to 0xFFFF

• Disk rewrite count (1 byte): The number of time a disk was rewritten at a kiosk. Incremented by one each time it is rewritten. Should be reset to 0

About games that save:

• All the above is perfectly fine for game that don't keep a save file on disk but some games keep a save file.

• Unfortunately there's no standard file name for save file, although they are often a variation of SAVE, SAV, SAVEDATA, etc.

• From the FDS files I observed, it seems like the save file is always the last file on SIDE A and/or SIDE B of the disk

I see 2 strategies here:

• The data area of the save file can be reset to all 0's or all 0xFF's

• The save file can be deleted altogether. That means the File amount block (block 2) must be decremented by one, the area where the save data is stored (block 4) must be reset to all 0's or all 0xFF's and the File header block (block 3) that contains file name and size must be deleted for the save file

Garbage data:

It seems like sometimes at the end of side A and/or B there are garbage bytes that looks like this

Code: Select all

DB B6 6D DB B6 6D DB B6 6D DB B6 6D DB B6 6D DB B6

this can be removed or it can stay there. I'm not sure what the best course of action is?

I have already written code to parse FDS file that can be found here https://gitlab.com/VincentEmond/readfds. I can modify this code to provide an easy way to normalize an FDS dump but I'd like to have the input of FDS savvy people and other FDS dumper/datters before doing anything.

I can also provide help in updating the FDS dumping guide in the wiki.

So the discussion is open what do you think?

PS: The only reliable way I've found to validate my disks right now is this website: https://pony.velvet.jp/fcdisk/fdsstdylst.html (in japanese) it contains the internal listing of files with CRC32. Unfortunately the dats were not helpful to me at all.

[arromdee]

I don't know how practical this is, but how about instead of normalizing the dump itself, writing a checksum program that checksums the dump as if it's had 0 in the normalized sections even when it doesn't.

That way you discourage people from actually modifying the existing dumps in order to normalize them.

[Hiccup]

I don't know how practical this is, but how about instead of normalizing the dump itself, writing a checksum program that checksums the dump as if it's had 0 in the normalized sections even when it doesn't.

That way you discourage people from actually modifying the existing dumps in order to normalize them.

It might be okay for people to modify them, as long as the program made backups or stored the removed data in a "sidecar" file.
People would probably want their files to match the dat, after all.

[relax]

We're discussing two things here.

Firstly the sealed disk vs. used disk issue. This is actually not a big problem anymore, the state of the FDS dat is very good, zero licensed FDS disks in the dat are marked as bad. Can't guarantee the situation is perfect, but I would say it's close to. Dozens of disks that were marked as bad because they were not in pristine condition, have been redumped from sealed disks. Thanks in particular to Yuki and HardcoreHubz on this forum, search for "sealed" and you will see some of their contributions. Then we have Gponys, all his dumps marked green are from pristine disks. https://pony.velvet.jp/fcdisk/fdsstdylst.html. Also the leaked Nintendo master archive confirmed a lot, although there were a few odd disks that looked modified and should probably be restored (in the QD-set, not the FDS-set. viewtopic.php?p=25400#p25400)

//

Secondly the the normalization issue. I don't think we should replace the existing set from pristine disks. However, I don't mind an additional normalized set, but I don't think its as important as I thought before the master archive leaked.

Regarding arromdee's approach:

I don't know how practical this is, but how about instead of normalizing the dump itself, writing a checksum program that checksums the dump as if it's had 0 in the normalized sections even when it doesn't.

That way you discourage people from actually modifying the existing dumps in order to normalize them.

There is such a program, FDS Study. It matches the checksum of the individual files on the disk against a database. So you get an "OK" even if the date and other header variables of your file differ. This consept could be improved on.
https://pony.velvet.jp/fcdisk/fmcmdskw11.html

Regarding prominos 's approach:

How to determine the normalized state? I'll quote myself from an internal post back in May 2020. I haven't looked at it since, so some info could be outdated.

To modify/restore ROMs is often controversial, but sometimes necessary. We have done it for Satellaview to restore the ROMs to their pristine condition. For Switch it's standard to replace the unique cart certificate with FF bytes. FDS is similar to Switch, but there are more fields that vary from disk to disk.

For FDS I see two perspectives. One is to preserve the info as stored on the disks, another is to preserve the ROM masters from the disk writer kiosks and factories used to create the more or less random info on the disks.

To create a master archive of FDS, we need to blank out fields that vary from disk to disk. And we need to add the info that's removed from the FDS-format but present on the disks and on the ROM masters, meaning CRCs. A perfect format for preservation is .qd, while for lecacy/emulation .fds can be kept for a while.

Some could argue for preserving disk gaps, but I don't see a good reason for that, as these seems to be random. And in the ROM master-perspective, the gaps are not relevant.

Looking at the headers of QD-files from Virtual Console etc., the following system could be used. Have I missed any fields?

Offset $1F 3bytes Manufacturing date -> FFs *
Offset $27 1byte Unknown -> 00s
Offset $28 3bytes Disk serial? -> FFs
Offset $2B 1byte Unknown -> 00s
Offset $2C 3bytes Rewritten date -> FFs
Offset $30 1byte Unknown -> FFs
Offset $31 3bytes Disk writer kiosk serial -> FFs
Offset $36 1byte Price -> 00s

*Note: Most QDs from VC that I have checked have a manufacturing date, fewer have a rewritten date. Metroid rev 3 from VC has all FFs. When they have a date, it seems to be early, perhaps it's the earliest possible date for each title and revision?

For the QD-format CRCs must be added. The first CRC is at offset $38 (2bytes) It's not always correct for all files in QDs from VC, but that's probably bacause Nintendo have modified some files for re-release on VC without updating the CRCs. Files that we know that are unmodified, like
Famicom Tantei Club Part II (Kouhen) from Wii VC can be losslessly converted from QD to FDS to QD with fdstool.

It seems that a collector has got a disk writer kiosk complete with NES ROM masters carts. So there's a small we'll find out about the format for these ROM masters.
https://twitter.com/kazzykazycom/status ... 0388642818

[prominos]

I had no idea most of them were dumped from pristine condition. That is actually very helpful to know, that is exactly why I was looking for input about this. Also thanks for telling me about FDS Study. My code actually output something almost identical to that tool because I was comparing the output of my script to the output of this FDS Study tool (without knowing). Looks like it can be found here: https://pony.velvet.jp/fcdisk/fmcmdskw11.html and I assume you already know that but I am posting it just so I can remember.

I wonder what the best way would be to compare second hand disks in an automated way. Preferably by using a Rom manager like RomVault. I know it's preferable to keep everything as close as retail as possible but in this case it's next to impossible to get a match in these circumstances. I mean, I don't really mind comparing manually using the pony velvet database and FDS Study but I like to see little green Rom chips in RomVault

[KingMike]

I am only now noticing this, sorry.

As to save files, I do know enough of the FDS to know if the software can actually create files (creating a save file if doesn't exist) or if could only modify exiting files (games would have shipped with a prewritten "blank" file).
However, I have heard that save files need to be the last file on a disk side. I have heard that from a forum post from a user who reported they played with a graphics modding tool back in the day and it could/would corrupt disks when overwritting files that weren't the last one.

Also, the file count is kind of meaningless. Some games would write a file beyond the stated count, as a simple copy-protection check. Like the file would be a single byte of actual data, so the game could load it and then read that byte to check if it was loaded. The devs would expect copying software to strictly follow the file count and not check for additional data. I know Deep Dungeon II: Yuushi no Monosho was one. In making a translation patch, I had to update my disk tool to support that. If it detected it was a pirated copy, after loading a saved game, it would freeze on a message telling a user like "The game is available for 3400 yen.", obviously suggesting them to buy an original copy.