FDS "cleaned garbage"

[xuom2]

Just reporting that scorp wrote me he doesn't like the "garbage cleaning" of:

https://datomatic.no-intro.org/index.ph ... =31&n=0003

dead_screem started adding modifications
but instead of that in DAT is not modification, but first present trusted dump he marked as forced
with different CRC
and he marked first dump (sealed disk) as bad dump. but presence of garbage data is normal, as FDS writes in sectors IIRC, so file on FDD takes less space than it uses in sectors. Dump should still be trusted.
the datter just nulled them out for no reason (that's okay I guess, they do not matter anything), but also marked trusted dump as bad, which is wrong.

i am asking to revert 21A1AD6E back to be a trusted version.

actually there are a lot of dumps he marked as bad
i do not know what is the current policy about it, but I do not think it is correct, as they were made from existing trusted dump
it does not become bad if it is precise copy of the diskette

[dead_screem]

Just reporting that scorp wrote me about this:

https://datomatic.no-intro.org/index.ph ... =31&n=0003

He doesn't like the garbage cleaning.

The garbage bytes included are caused by a final block of type 05. What? 05? That's not a valid block type. the block is full of $DBB66D repeating until the apparent end of disk. it doesn't appear to have a checksum.
a bug in fdsstick software is causing it to include just 17 bytes of the much larger 05 block. these exact same garbage bytes are included in almost all Hardcorehubz dumps. 05DBB66DDBB66DDBB66DDBB66DDBB66DDBB6. gponys dumps don't have them, because of the era disk copy method was used and not fdsstick.

If you look in the bin, you can see this final invalid 05 block is in fact MUCH larger than what fdsstick puts in the fds. in ai senshi nicol side a, the entire block is 0x305a bytes plus a final $0b byte, apple town side a the 05 block is 0x37f0 bytes plus 2 more bytes of $3600.

If we decide that this invalid 05 block (which doesnt even appear to have a checksum) is supposed to be there, and try to include it in full, in the fds dump... Let's take ai senshi nicol side a. from the end of the last valid file until end of disk is 0x2eee bytes, but this invalid 05 block is (including the 05 block start code byte) until end of bin is 0x305c bytes! that exceeds the size limit of a single disk side and goes into the disk post gap.

This invalid block 05 shouldn't be in the .fds. infact it can't, it's impossible. This block can only be in the bin dump. fdsstick including first 17 bytes of it in .fds dumps is a bug. of the era disk copy methods "correctly" ignore this block 05. Unless someone can prove the games try to read from this 05 block as some form of copy protection (afaik they don't) there is no point in trying to force it into the .fds or .qd image formats.

I know nothing of programming, and know only some of the disk format, but unless he actually knows anything however small about the qd format, or fc disk game programming, I won't change my edits.
(for the record )

[dead_screem]

Firstly, in the case of ai senshi nicol (and all other disks in the dat), all dumps were datted as having the same hash. However, they are not the same. some have a different date code. and all have different "unknown header bytes" which I now believe to be a disk serial number. But aside these few bytes they are the same. What I did was add the correct hashes for all dumps, mark the ones not to be datted as "excluded", and marked one dump as "forced". In this case The dump marked "forced" was one of gponys unmodified dumps.
Hardcorehubz dump was flagged bad, and I added a copy of Hardcorehubz dump with the garbage data removed as a Trusted modification and flagged it "excluded".

To clairify, Yes. this data is infact on the disk. However it has an invalid block type of 05, so usual disk copy methods correctly ignore block type 05. fdsstick seems to be confused by it (maybe?).
There are 4 block types. 01 the disk header, 02 is number of files on disk, the rest of disk will be an 03 block followed by a 04 block. and again an 03 then 04 for every file on disk. 03 is file descriptor block, contains file id, file name and size and other info. 04 is the file data block, it contains the actual data of the file specified in previous 03 block.

At the end of certain disks is a data block with a id of 05. Lets take a look.

https://ibb.co/CtbsXhW
Here is a photo of ai senshi nicol side a bin dump at near the end of disk image, you can clearly see data blocks and gaps. Let's look at the address I highlighted e1eb. You can see a byte, $80. This byte is only found in bin/raw dumps. It's a control byte that means basically, end of gap/data block start. after the $80 is the block type id code. Here its $03 which is a file descriptor. You can see various data after that including file number $09, file name "SAVE_DT0", file size 0000A0 and the checksum at the end 3A5E. (you can see other bytes after that, which seem to be other control bytes I don't know what they are. these are also not in the .fds/.qd but only in bin.
Then you see a gap of zeros followed by $8004, the start of the file data block after the 04 is 0x0a (ten) zeros. followed by the checksum 95f0 followed by four "x" the four "x" would seem to mean end of file data block. You can notice above at the beginning of photo, that file data block ends with four "F" why the difference? I don't know.

Now after the next gap of zeros is $8005. data block 05 is apparently invalid (so it seems). You can see the data in this block is nonsense, its just dbb66d repeated over and over. There is not even a checksum for this block. it goes on for much longer than in the photo, until address 13365 (end of bin file).

https://ibb.co/99RKsrY
Here is the .fds dumped by hardcorehubz using fdsstick. You can see the usual stuf about the save file but with the checksum and extra control bytes ommitted. The 05 data block is included, but only 17 bytes of it for some reason. It was much larger in the bin. These 17 bytes included in .fds dump by fdsstick is a mistake/bug.

https://ibb.co/GPrc26Z
Here is a dump by gponys. This dump doesn't use fdsstick. But an older disk copy tool.
The 05 block is completely not present. Which is correct.


https://ibb.co/DgvqTRZ
Even if you say "even though the data is garbage, it's on the disk! It must be included!" Well, just look at this.
This photo shows copying the entire 05 data block from the bin image and pasting it in the .fds. .fds files have a max file size of 65500 per side which ends on address ffdb. Notice the area I highlighted. Including the entire 05 data block would extend the image beyond the max size of 65500 until 65866 bytes!
If this was a .qd file which includes the block checksums, has a max image size of 65536, this would extend to 65902. This is why including this block in the .fds or .qd is a thing which is impossible. The physical disk size is actually larger than the user format area of 65500, there is a pregap of zeros before the main data and a post gap of (usually) zeros after the main data. This 05 data block extends out of the user format area and into the post gap area. This means this garbage data block can only be in the raw bin image dump. And cant be in the .fds or .qd dumps.


Hope that explains it.

[dead_screem]

but presence of garbage data is normal, as FDS writes in sectors IIRC, so file on FDD takes less space than it uses in sectors. Dump should still be trusted.

This is wrong.
FDS DD is not a normal random access disk drive. It does not use "sectors". It is infact much closer in behavior to a cassette tape drive. The R/W head on fds drive is similar to a cassette tape head and the data on fds disk is read / written completely linear. That is, a spiral stored on the disk.

afaik the fdsstick RAW image seems to be low level analog data. As you woud expect analog data with a cassette tape head.

[scorp256]

but presence of garbage data is normal, as FDS writes in sectors IIRC, so file on FDD takes less space than it uses in sectors. Dump should still be trusted.

This is wrong.
FDS DD is not a normal random access disk drive. It does not use "sectors". It is infact much closer in behavior to a cassette tape drive. The R/W head on fds drive is similar to a cassette tape head and the data on fds disk is read / written completely linear. That is, a spiral stored on the disk.

afaik the fdsstick RAW image seems to be low level analog data. As you woud expect analog data with a cassette tape head.

I am speaking about inner structure of the synthetic FDS image, not about raw diskette. Synthetic image have to follow FDS format, so no gaps or CRCs or markers or stuff like that, as there is only 65500 bytes per side.
I know how usual magnetic floppy works and FDS works the same, and yes, it is very similar to a cassette tape, as it uses same basic principles. As you know (I think) each time the drive read data, it reads it differently (means it could start before or after, not in exact same place each time it reads), thus why there is need of these markers which mark start of data track. Thus why you can see various wrong data bytes read after markers - these are so-called floating bytes. Anyway, I tell sectors, you can tell block, whatever the correct naming is.

So, you think that these 05 sector data should not be there, because you think it should be ignored by FDS format completely. Do I get it right? If so, why there should be 00 instead? I mean, these FDS are valid even with this additional data, they are not invalid in terms of format, right? Every emulator out there will check and load them just fine, at least. Why we should remove them? You showed yourself that part of this data was present in raw file, so I can say that "well, due to FDS format imperfection it was impossible to fit it properly, so something is better than nothing", as this data was indeed in the raw dump.

[dead_screem]

I am speaking about inner structure of the synthetic FDS image, not about raw diskette. Synthetic image have to follow FDS format, so no gaps or CRCs or markers or stuff like that, as there is only 65500 bytes per side.
I know how usual magnetic floppy works and FDS works the same,

No it doesn't.
A normal floppy disk works similar to a hard disk drive. FDS/QD is similar to cassette tape.

As you Anyway, I tell sectors, you can tell block, whatever the correct naming is.

Sector is a word that is used for the physical format of hard disk and normal floppy disk. i.e. Cylinders, heads, tracks. sectors. FDS doesn't have such things in physical format.

As for the FDS/QD disk internal format, referring to the data seperated by gaps, you should say block or chunk or something like that, "sector" is 100% wrong.

So, you think that these 05 sector data should not be there, because you think it should be ignored by FDS format completely. Do I get it right? If so, why there should be 00 instead? I mean, these FDS are valid even with this additional data, they are not invalid in terms of format, right? Every emulator out there will check and load them just fine, at least. Why we should remove them? You showed yourself that part of this data was present in raw file.

If you are still are asking this, than you did not read my post completely and understand it. posting.php?mode=quote&f=6&p=23137#pr23125
I fully explained it in that post, if you cant understand, that's not my problem.

[scorp256]

No it doesn't.
A normal floppy disk works similar to a hard disk drive. FDS/QD is similar to cassette tape.

Actually it does. Did you checked raw reading of some early devices, like ZX Spectrum Beta Disc interface? Very similar.

Sector is a word that is used for the physical format of hard disk and normal floppy disk. i.e. Cylinders, heads, tracks. sectors. FDS doesn't have such things in physical format.

As for the FDS/QD disk internal format, referring to the data seperated by gaps, you should say block or chunk or something like that, "sector" is 100% wrong.

Okay.

If you are still are asking this, than you did not read my post completely and understand it. posting.php?mode=quote&f=6&p=23137#pr23125
I fully explained it in that post, if you cant understand, that's not my problem.

Actually I read that and I hope I understood it. Your example does not explain for me questions I asked. You just tell "so if we put all this data properly in full it will make the disc invalid". Currently it in on FDS in cropped form, but disc is valid. So yes, I do not understand, why cut version of actual data is worse than no data at all.

[dreimer]

If I get things right (hopefully) The problem is that some dumping tools read that 05 sector data and some ignore it completely as the game itself does not need and read it anyways as it contains filler garbage and not more. So in theory it should be irrelevant if that data is there or not and as the dumps work fine it seems to be the case. If that makes one or the other correct now... other piece of paper. IMO we should put what we can into the FDS format, only partially possible or not, it would be more... "correct" that way. Of course a updated FDS format would be the best idea, but good luck doing such a big step... Remember the cases in the NDS database? The data missing in all dumps was irrelevant, too as it could not even be read by the NDS or game at all. But still.... all games got redumped and marked bad until that happened. Similar here. If all FDS disks have these 05 sector data, we should add the minimal variant of it to all (or make a new more correct format^^) and not remove it everywhere. Only my opinion, hopefully I got the posts above right.

[xuom2]

What scorp256 is suggesting is, considering the fact you won't get the same crc from 2 different disks of same game, to set in database any dump as "good" and forcing DOM to exclude what doesn't follow the ruleset...

... and ruleset is the second part of the question. If leaving garbage is more a clean-nointro way to keep original things or not, like fixing an overdump.

On my side I am happy we are using DOM in the correct way, adding data (instead of replacing) and commenting the differences between dumps.

[dead_screem]

Actually it does. Did you checked raw reading of some early devices, like ZX Spectrum Beta Disc interface? Very similar.

by normal I mean standard 3 1/2" , 5 1/4" and 8" floppys, these behave similar to hard disk. QD/FDS and anything else similar is not normal floppy disk.

Actually I read that and I hope I understood it. Your example does not explain for me questions I asked. You just tell "so if we put all this data properly in full it will make the disc invalid". Currently it in on FDS in cropped form, but disc is valid. So yes, I do not understand, why cut version of actual data is worse than no data at all.

I'll explain again than. Hardcorehubz disk dump using fdsstick software, that contains the 05 block, that block in the dump is only 17 bytes (18 if you count the 05 byte). The total size of the 05 block in the bin dump is 12379 bytes (12380 counting 05 byte)! Even though the data is garbage, how is including only 17 bytes of it correct? Thats a purely random number. Why fdsstick software is doing this I don't know. Is it a bug? or a deliberate choice of loopy?
Regardless of that, If you really want the 05 data block included, you should include all of it, not just first 17 bytes.
Except you cant. As I already mentioned in previous post there is not enough free space on the disk, ai senshi nicol side a there is only 12014 bytes left. Thats not enough. That's because this data block extends out of the user format area (65500 bytes) and into the post gap area of disk.
Sure, you could paste the complete block into the .fds and trim excess bytes to correct 65500 size limit, but why? Its still only partial data.

Which is why removing that incomplete and invalid data block from .fds is correct imo .fds file format is supposed to be internal disk format, this data is invalid so should not be included in it. This data is basically the same as 00 byte gaps at end of disk, except for some reason some disks have this garbage data at end of disk when there is usually 00s. This garbage data should be only allowed in bin dumps.

It should also be mentioned that this block is never the same size, it just seems to go until end of physical disk (it would seem).
And yes, fdsstick is only tool software which includes (some of) this garbage data. Other tools ignore it, which is correct imo.

[scorp256]

I'll explain again than. Hardcorehubz disk dump using fdsstick software, that contains the 05 block, that block in the dump is only 17 bytes (18 if you count the 05 byte). The total size of the 05 block in the bin dump is 12379 bytes (12380 counting 05 byte)! Even though the data is garbage, how is including only 17 bytes of it correct? Thats a purely random number. Why fdsstick software is doing this I don't know. Is it a bug? or a deliberate choice of loopy?
Regardless of that, If you really want the 05 data block included, you should include all of it, not just first 17 bytes.
Except you cant. As I already mentioned in previous post there is not enough free space on the disk, ai senshi nicol side a there is only 12014 bytes left. Thats not enough. That's because this data block extends out of the user format area (65500 bytes) and into the post gap area of disk.
Sure, you could paste the complete block into the .fds and trim excess bytes to correct 65500 size limit, but why? Its still only partial data.

Which is why removing that incomplete and invalid data block from .fds is correct imo .fds file format is supposed to be internal disk format, this data is invalid so should not be included in it. This data is basically the same as 00 byte gaps at end of disk, except for some reason some disks have this garbage data at end of disk when there is usually 00s. This garbage data should be only allowed in bin dumps.

It should also be mentioned that this block is never the same size, it just seems to go until end of physical disk (it would seem).
And yes, fdsstick is only tool software which includes (some of) this garbage data. Other tools ignore it, which is correct imo.

So your point is that as this is incomplete garbage data it should be nulled. Do we have such policy? I just thought it is as xuom2 told, so to keep more data rather than clear it out. Also my previous question about marking this as bad dump stays. fdsstick does this, we both know that no matter how you fill the free space disc will be still valid - so why nulled version is "more correct" than the partially filled version?

So my point is that I do not think you should mark this dump as "bad" in the database, but I agree that as there is alternative dump without this data it is correct to mark it as one to go into .dat

[dead_screem]

So your point is that as this is incomplete garbage data it should be nulled. Do we have such policy? I just thought it is as xuom2 told, so to keep more data rather than clear it out. Also my previous question about marking this as bad dump stays. fdsstick does this, we both know that no matter how you fill the free space disc will be still valid - so why nulled version is "more correct" than the partially filled version?

So my point is that I do not think you should mark this dump as "bad" in the database, but I agree that as there is alternative dump without this data it is correct to mark it as one to go into .dat

You still not understanding. Infact, I don't think you are willing to understand. You just think you are right. So, this is the last time I'm trying to have you understand.

To sum it all up, not only is the block incomplete, its incomplete because of fdsstick error, that means bad. In addition, its not a valid block, so is not part of the fds/qd internal format (IMO), despite looking like it. .fds should be only valid internal format blocks, nothing else allowed. So it's double bad. This invalid block should be only allowed in the full bin dump. If it turns out games could possibly access the data, i.e that the bios rom doesn't block access to this invalid data... then this may need rethinking.

I am right, you are wong. Deal with it.

[scorp256]

You still not understanding. Infact, I don't think you are willing to understand. You just think you are right. So, this is the last time I'm trying to have you understand.

I got you from your first post. No, I do not think I am right, I just want to see something more concrete, rather than your unchangeable opinion in different posts.

You know, firmware docs or something, which will prove, that it is impossible to load block #5 on actual hardware. Or impossible to have it in actual .fds file.

To sum it all up, not only is the block incomplete, its incomplete because of fdsstick error, that means bad.

I think different, I think that other dumping tools dump only content, which is used by files, not the whole disk content. And as there is no file pointing to this data it is not dumped by other dumping tools. FDSStick seems tries to dump also unused content, but fails to do it properly.

In addition, its not a valid block, so is not part of the fds/qd internal format (IMO), despite looking like it.

By fds you mean actual system, not the file format, right? As fds file format does not specify what blocks could be there, it is just derived from the data on actual fds disk. So to tell if it is used or not we need to see what is in the chip of disk system and in its code disassembly. If you have such documentation, it would be very interesting to read that.

.fds should be only valid internal format blocks, nothing else allowed. So it's double bad.

But it is not. If you read here https://wiki.nesdev.com/w/index.php/FDS_file_format it is told that it uses actual disk format without gaps and crcs. They do not specify in .fds format what types of block are valid.

This invalid block should be only allowed in the full bin dump. If it turns out games could possibly access the data, i.e that the bios rom doesn't block access to this invalid data... then this may need rethinking.

And we again came to the beginning of the argument. UPD see below, it seems possible to access the data, bios does indeed allow any type of block.

I am right, you are wong. Deal with it.

Yeah.

Anyway, I see we're going in circles, it is my deduction against your deduction. Without concrete docs telling "this block is impossible to read on actual hardware, as it is not in firmware access at all" there is not much to discuss.

UPD: Actually it seems possible to read block with type #5 on actual hardware. From what I read here https://nesdev.com/FDS%20technical%20reference.txt it is told that you can read files which are not in catalog and it was used in copy-protection. And as I understand you technically can call CheckBlkType directly, you can pass block type (any number) as parameter and load with rom routines instead of normal file loading routine. I do not see any check for block type in rom disassembly, only for error text return, but who cares about error text, when you use rom load.

[dead_screem]

https://wiki.nesdev.com/w/index.php/FDS_file_format
"After the last file block, fill a side with all 0 so that exactly 65500 bytes is reached. "
a file block is 03 and 04 blocks. 05 is not file block.


https://wiki.nesdev.com/w/index.php/FDS_disk_format
"Each disk side must be structured into block as follows :
1, 2, 3, 4, 3, 4, ...., 3, 4
The 3, 4 pattern should be repeated once per file present on the disk.
Blocks type 1 and 2 represent information about the disk and how many files it has, and each block type 3 and 4 pair represents a single file."

UPD: Actually it seems possible to read block with type #5 on actual hardware. From what I read here https://nesdev.com/FDS%20technical%20reference.txt it is told that you can read files which are not in catalog and it was used in copy-protection. And as I understand you technically can call CheckBlkType directly, you can pass block type (any number) as parameter and load with rom routines instead of normal file loading routine. I do not see any check for block type in rom disassembly, only for error text return, but who cares about error text, when you use rom load.

that is just your guess.


I've said my stance, and I'm sticking to it. 05 block is (seemingly) an invalid or unused type. The block has no valid crc data. So, the bios should return a crc error (or other error) if you try to access it. And thus, it is not part of the .fds internal format, and should be treated the same as gap data, i.e not included.
And, here is a problem. Since 05 has no crc data and .fds doesn't store this data, when you load a .fds with the invalid 05 block into an emulator (or fdsstick) or write it to a physical disk... all data on .fds is assumed to be good, so crcs will be generated for all blocks. So that 05 block will become "good" by having a good crc added. This is wrong! So you can see, .fds format is only for valid data blocks with good crc. Adding this block into .fds makes invalid disk image in the end, because upon loading the image, crcs get added for all blocks in image, including that 05 block which doesn't have any normally.

Even if it is possible to access this invalid block somehow without triggering a crc error, (the crc calculation is done by ram adapter automatically not on FC CPU). It still can only be in .bin format dump. Because putting it in .fds means it always gets a good crc. so an .fds with this block is always a bad dump.

[scorp256]

https://wiki.nesdev.com/w/index.php/FDS_file_format
"After the last file block, fill a side with all 0 so that exactly 65500 bytes is reached. "
a file block is 03 and 04 blocks. 05 is not file block.

This could be interpreted "after last block used", as I think author did not know there could be other block types. Also the person who wrote it in wiki most likely was not the format author.

https://wiki.nesdev.com/w/index.php/FDS_disk_format
"Each disk side must be structured into block as follows :
1, 2, 3, 4, 3, 4, ...., 3, 4
The 3, 4 pattern should be repeated once per file present on the disk.
Blocks type 1 and 2 represent information about the disk and how many files it has, and each block type 3 and 4 pair represents a single file."

This is a description of standard diskette format. We are speaking about non-standard.

that is just your guess.

Not a guess, it is in the code, you can check yourself. But if you insist, I can try to find a development kit for FDS and code you a test, if you are willing to take a look if it works in real hardware (Actually joking, I do not really want to learn how to code valid programs in FDS assembly just for the sake of this lenghty argument).

I've said my stance, and I'm sticking to it. 05 block is (seemingly) an invalid or unused type. The block has no valid crc data. So, the bios should return a crc error (or other error) if you try to access it.

It will return crc error. But it does not mean you cannot read it, as you can simply ignore this error, as it will return it to your own code (see XferDone). Actually this is how protection in old days worked, you read data which typical copiers cannot read.

And thus, it is not part of the .fds internal format, and should be treated the same as gap data, i.e not included.
And, here is a problem. Since 05 has no crc data and .fds doesn't store this data, when you load a .fds with the invalid 05 block into an emulator (or fdsstick) or write it to a physical disk... all data on .fds is assumed to be good, so crcs will be generated for all blocks. So that 05 block will become "good" by having a good crc added. This is wrong! So you can see, .fds format is only for valid data blocks with good crc. Adding this block into .fds makes invalid disk image in the end, because upon loading the image, crcs get added for all blocks in image, including that 05 block which doesn't have any normally.

Even if it is possible to access this invalid block somehow without triggering a crc error, (the crc calculation is done by ram adapter automatically not on FC CPU). It still can only be in .bin format dump. Because putting it in .fds means it always gets a good crc. so an .fds with this block is always a bad dump.

For bad CRC I told before, yes, you can load these blocks.

Okay, you got a point here. Yes, if we assume that fds could hold only blocks which have always valid CRC (which never told anywhere explicitly, but still only right way is to keep it so), then there cannot be invalid blocks with bad crc and this block is clearly with bad crc. I did not looked at this problem from this angle. You was right, I was wrong