DAT-o-MATIC xml support in N64 dumper for easier processing

[wolfre]

Hello everyone

As suggested by hiccup over here, I'm currently working on supporting dat-o-matic xml in my N64 dumper / info generator.
The idea is to make processing easier compared to the regular info text format.
Here are my sources I've looked through:
- No-Intro Wiki: https://wiki.no-intro.org/index.php?tit ... ittinginfo
- Example xml: https://datomatic.no-intro.org/stuff/ex ... custom.xml
- Parent/Clone BigEndian xml: https://datomatic.no-intro.org/index.ph ... =24&op=xml
- Standard DAT BigEndian: https://datomatic.no-intro.org/index.ph ... 4&op=daily

Here's a comparison between the info text and my current xml variant (For PAL Wave Race 64, as of commit dfa58661):

Code: Select all

Dumping tool and version: Some Tool 1.0.0
Dumper: Mr. Dump
Affiliation: No-Intro
Dump creation date: 2020-12-01
Dump release date: 2020-12-02
Link(s): https://example.com

Title: Wave Race 64
Region: Europe
Edition: Retail
Languages: En,De
Language Select: yes
WikiData ID: Q3142278

ROM Region: Europe
ROM Revision: 0x00 (1.0)
ROM Serial: NWRP

Original format: BigEndian

BigEndian:
  File: Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De).z64
  Size: 8388608
  CRC32: FB289893
  MD5: 310659115E9939F219A783ABDD456CE9
  SHA-1: C20AA97DC25AEC7A9B9C6889286BBD216FC7CAA4
  SHA-256: 8897DA0CB6E286659AC4D72C5737F6FE97A51BCDFC6E2328AE7DFC0715BF1761
ByteSwapped:
  File: Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De).v64
  Size: 8388608
  CRC32: D74117B0
  MD5: BAFF32EE50896E62DC713E23D1A589DA
  SHA-1: 7FFDD44890B1A00E0C13868A18B482FBF8C752F7
  SHA-256: 4C07E69B289777A3ECC3719A1970BA4287737BE03BB79536CCCCE52EB7049EA0

Media Serial 1: NUS-NWRP-EUR
Media Stamp: 19
PCB Serial: NUS-01A-01
ROM Chip Serial 1: NUS-NWRP-0
Box Serial: P-NWRP-NNOE
Box Barcode: 045496870034

Code: Select all

<?xml version="1.0"?>
<datafile xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <header />
  <game name="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De)">
    <archive name="Wave Race 64" namealt="" region="Europe" languages="En,De" showlang="2" version="" devstatus="" additional="" special1="" special2="" gameid="" clone="" regionalparent="" mergeof="" datternote="" stickynote="" />
    <flags bios="0" licensed="1" pirate="0" physical="1" complete="1" nodump="0" public="1" dat="1" />
    <source>
      <details section="" rominfo="" dumpdate="2020-12-01" originalformat="BigEndian" knowndumpdate="1" releasedate="2020-12-02" knownreleasedate="1" dumper="Mr. Dump" project="No-Intro" tool="Some Tool 1.0.0" origin="" comment1="Retail" comment2="" link1="https://example.com" link2="" region="Europe" mediatitle="Wave Race 64" />
      <serials mediaserial1="NUS-NWRP-EUR" mediaserial2="" pcbserial="NUS-01A-01" romchipserial1="NUS-NWRP-0" romchipserial2="" lockoutserial="" savechipserial="" chipserial="" boxserial="P-NWRP-NNOE" mediastamp="19" boxbarcode="045496870034" digitalserial1="" digitalserial2="" />
      <rom forcename="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De).z64" emptydir="0" extension="z64" item="" date="2020-12-01" format="BigEndian" version="0x00 (1.0)" utype="" size="8388608" crc="fb289893" md5="310659115e9939f219a783abdd456ce9" sha1="c20aa97dc25aec7a9b9c6889286bbd216fc7caa4" sha256="8897da0cb6e286659ac4d72c5737f6fe97a51bcdfc6e2328ae7dfc0715bf1761" serial="NWRP" bad="0" unique="0" mergename="" />
    </source>
  </game>
  <game name="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De)">
    <archive name="Wave Race 64" namealt="" region="Europe" languages="En,De" showlang="2" version="" devstatus="" additional="" special1="" special2="" gameid="" clone="" regionalparent="" mergeof="" datternote="" stickynote="" />
    <flags bios="0" licensed="1" pirate="0" physical="1" complete="1" nodump="0" public="1" dat="1" />
    <source>
      <details section="" rominfo="" dumpdate="2020-12-01" originalformat="BigEndian" knowndumpdate="1" releasedate="2020-12-02" knownreleasedate="1" dumper="Mr. Dump" project="No-Intro" tool="Some Tool 1.0.0" origin="" comment1="Retail" comment2="" link1="https://example.com" link2="" region="Europe" mediatitle="Wave Race 64" />
      <serials mediaserial1="NUS-NWRP-EUR" mediaserial2="" pcbserial="NUS-01A-01" romchipserial1="NUS-NWRP-0" romchipserial2="" lockoutserial="" savechipserial="" chipserial="" boxserial="P-NWRP-NNOE" mediastamp="19" boxbarcode="045496870034" digitalserial1="" digitalserial2="" />
      <rom forcename="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De).v64" emptydir="0" extension="v64" item="" date="2020-12-01" format="ByteSwapped" version="0x00 (1.0)" utype="" size="8388608" crc="d74117b0" md5="baff32ee50896e62dc713e23d1a589da" sha1="7ffdd44890b1a00e0c13868a18b482fbf8c752f7" sha256="4c07e69b289777a3ecc3719a1970ba4287737be03bb79536cccce52eb7049ea0" serial="NWRP" bad="0" unique="0" mergename="" />
    </source>
  </game>
</datafile>

I wasn't sure on a couple of things, here's a list of questions:
- There's <game><source> and <game><release>, I went with <source>, the DAT format suggests it should be <release> instead?
- Is it better to leave out unfilled / empty fields?
- I chose two <game> entries for big endian and swapped, would have two <source> (<release>) or <rom> been better?
- Should the <flags> be omitted, its a bit pointless for a redump?
- In <serials> there are field "chipserial" and "digitalserial1", what are those / are they relevant for N64?

Any comments are welcome, feel free to post a corrected version, and I'll diff it
Hope it helps to process data with less manual work

Regards
Rene

[Hiccup]

- There's <game><source> and <game><release>, I went with <source>, the DAT format suggests it should be <release> instead?

release is a special type of source for scene releases. it contains things like nfo name, dirname and release group.

- There's <game><source> and <game><release>, I went with <source>, the DAT format suggests it should be <release> instead?
- Is it better to leave out unfilled / empty fields?

the custom xml import page (i guess only datters have access to it) says only include the fields you need. although i'm not sure if that's wise - maybe it'd be better to be explicit, as the defaults aren't really documented properly. Its probably better to leave them blank for now.

- I chose two <game> entries for big endian and swapped, would have two <source> (<release>) or <rom> been better?

different formats that originate from a single dump go under a single source and single archive - so two rom is better

- In <serials> there are field "chipserial" and "digitalserial1", what are those / are they relevant for N64?

chip serial is deprecated ("rom chip 1 serial(s)" and "rom chip 2 serial(s)" replaces it and digitalserial is for digital dats (wiiware etc)

I tried importing your XML, and it worked, but I did notice some other things that weren't right:
- "adult" was set to yes even though you didn't specify this - i'll tell xuom2 about this (apparent) bug
- you put "Retail" in comment1. as retail is the default, you can leave that out. If its from a dev cart or kiosk demo cart, you can put "Dev cart" or "Kiosk demo cart" in origin
- You can leave mediatitle blank - that's sort of deprecated and isn't really needed.
- You can leave release date/release date known blank, if this is just a redump, as the ROM is already out there. And even if its a new dump, you can leave it ambiguous if you want (leave out the fields - unless you've made a big public release like a news entry on Hidden Palace for a prototype or something, in that case the release date should be specified and and url for that release page should be put in link1.
- Leave the file date blank - that's basically deprecated
- Leave native filename (forcename) blank - under most circumstances, the filenames should be auto-generated by datomatic, based on the archive fields. If you put something in forcename, that doesn't happen.
- File version and serial are fine (make sure they are parsed from the ROM itself, not manually entered by the user), but "0x00 (1.0)" should just be "0".
- File "unique" should be set to the default "1" - this is a new field for MAME style merging. its not used in any dats yet (and i don't fully understand it).
- For redumps (as opposed to new dumps) you can leave out everything from the archive field, other than "id". then it can be imported using the "Append only sources and files" option in the "submit custom xml" form.

I've asked xuom2 about bulk-importing photos along with the XML (uploading a zip alongside and referring to each file by filename in the xml?), but he hasn't done that yet.

Btw, you should also read these (linked on "Datting Guide" on the wiki):
https://wiki.no-intro.org/index.php?tit ... Convention
https://wiki.no-intro.org/index.php?tit ... Convention

I'm going to start to document the "internal database names" (which you can see in the html of the edit view in datomatic) and the "custom xml" names (apparently they are different) for each field, on those pages, plus make a page for the file fields.

[wolfre]

Thanks for your quick and detailed response!

release is a special type of source for scene releases. it contains things like nfo name, dirname and release group.

Ok I stick with <source> then

... says only include the fields you need.

Non needed fields are omitted now

so two rom is better

Its a single <source> with two <rom> now

chip serial is deprecated [...] digitalserial is for digital dats (wiiware etc)

Great both are ommited then in my case

"Retail" in comment1

Removed and if its something like "Gold Cart" (e.g. OOT), this goes to comment2 now (according to https://wiki.no-intro.org/index.php?tit ... Convention)

You can leave mediatitle blank

Removed it

You can leave out release date/release date known

Removed it

I've asked xuom2 about bulk-importing photos along with the XML

Oh that sounds like a nice addition, I can open another task on the issue tracker, then I'll add support


Here's the same example as before, now with your suggested changes (as of commit ed5c4efb):

Code: Select all

<?xml version="1.0"?>
<datafile xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <header />
  <game name="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De)">
    <archive name="Wave Race 64" region="Europe" languages="En,De" showlang="2" />
    <flags bios="0" licensed="1" pirate="0" physical="1" complete="1" nodump="0" public="1" dat="1" />
    <source>
      <details dumpdate="2020-12-01" originalformat="BigEndian" knowndumpdate="1" dumper="Mr. Dump" project="No-Intro" tool="Some Tool 1.0.0" link1="https://example.com" region="Europe" />
      <serials mediaserial1="NUS-NWRP-EUR" pcbserial="NUS-01A-01" romchipserial1="NUS-NWRP-0" boxserial="P-NWRP-NNOE" mediastamp="19" boxbarcode="045496870034" />
      <rom forcename="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De).z64" emptydir="0" extension="z64" date="2020-12-01" format="BigEndian" version="0x00 (1.0)" size="8388608" crc="fb289893" md5="310659115e9939f219a783abdd456ce9" sha1="c20aa97dc25aec7a9b9c6889286bbd216fc7caa4" sha256="8897da0cb6e286659ac4d72c5737f6fe97a51bcdfc6e2328ae7dfc0715bf1761" serial="NWRP" bad="0" unique="0" />
      <rom forcename="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De).v64" emptydir="0" extension="v64" date="2020-12-01" format="ByteSwapped" version="0x00 (1.0)" size="8388608" crc="d74117b0" md5="baff32ee50896e62dc713e23d1a589da" sha1="7ffdd44890b1a00e0c13868a18b482fbf8c752f7" sha256="4c07e69b289777a3ecc3719a1970ba4287737be03bb79536cccce52eb7049ea0" serial="NWRP" bad="0" unique="0" />
    </source>
  </game>
</datafile>

When this task is done I'll do a release so there's a nice downloadable packge for others to use (and also I'm going to use it to submit an unverified game that just dropped in )

Thanks
Rene

(edit: just saw I missed a couple of your additions, will do them now sorry)

[Hiccup]

...

I think you've missed some of the things I pointed out in my post above, but still, looks better than before.

[wolfre]

So here's the things I overlooked:

Leave the file date blank - that's basically deprecated

Removed

Leave native filename (forcename) blank

Removed, also removed the "extension" field, coz I guessed they are also auto generated.

File version and serial are fine (make sure they are parsed from the ROM itself, not manually entered by the user)

Yes it is parsed from the rom header, but the user is asked for confirmation that it is correct

but "0x00 (1.0)" should just be "0".

Hm so supposedly that's a Major.Minor notation: http://en64.shoutwiki.com/wiki/ROM#Cartridge_ROM_Header
So I did a ...

Code: Select all

cat "Nintendo - Nintendo 64 (BigEndian) (20211114-091233).dat" | grep -i "(rev .*)" | sed -e "s/.* (Rev/Rev/" | cut -d ")" -f 1 | sort -u

and found at max things like "Rev 3". I took a look at "Super Mario 64 (Japan) (Rev 3) (Shindou Edition)" and here it is 0x03.
So I guess there is no known case where the upper nibble is anthing but 0b0000 (I wonder what the source for en64 is).
Based on that I changed it to take the raw 8bit field as a integer in regular decimal notation (so 0x03 == 3, 0xff == 255).

File "unique" should be set to the default "1"

Done

For redumps (as opposed to new dumps) you can leave out everything from the archive field, other than "id".

I kept "gameid" and put the name in it, or should this have been "NWRP"?


Here's the example again (as of commit 161b20b5)

Code: Select all

<?xml version="1.0"?>
<datafile xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <header />
  <game name="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De)">
    <archive gameid="Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De)" />
    <flags bios="0" licensed="1" pirate="0" physical="1" complete="1" nodump="0" public="1" dat="1" />
    <source>
      <details dumpdate="2020-12-01" originalformat="BigEndian" knowndumpdate="1" dumper="Mr. Dump" project="No-Intro" tool="Some Tool 1.0.0" link1="https://example.com" region="Europe" />
      <serials mediaserial1="NUS-NWRP-EUR" pcbserial="NUS-01A-01" romchipserial1="NUS-NWRP-0" boxserial="P-NWRP-NNOE" mediastamp="19" boxbarcode="045496870034" />
      <rom emptydir="0" format="BigEndian" version="0" size="8388608" crc="fb289893" md5="310659115e9939f219a783abdd456ce9" sha1="c20aa97dc25aec7a9b9c6889286bbd216fc7caa4" sha256="8897da0cb6e286659ac4d72c5737f6fe97a51bcdfc6e2328ae7dfc0715bf1761" serial="NWRP" bad="0" unique="1" />
      <rom emptydir="0" format="ByteSwapped" version="0" size="8388608" crc="d74117b0" md5="baff32ee50896e62dc713e23d1a589da" sha1="7ffdd44890b1a00e0c13868a18b482fbf8c752f7" sha256="4c07e69b289777a3ecc3719a1970ba4287737be03bb79536cccce52eb7049ea0" serial="NWRP" bad="0" unique="1" />
    </source>
  </game>
</datafile>

Regards
Rene

[wolfre]

Ahh I think I've answered my question about:

I kept "gameid" and put the name in it, or should this have been "NWRP"?

Should this be "0820"? I download the db as text file from here https://datomatic.no-intro.org/index.ph ... s=24&op=db the entry is

Code: Select all

Wave Race 64 - Kawasaki Jet Ski (Europe) (En,De)
  Number: 0820
  Clone: P
  Name: Wave Race 64 - Kawasaki Jet Ski
  Region: Europe
  Languages: En,De
	Trusted Dump
	  Dump date: ...

So I guess this is best auto filled based on that DB file ... or is there an easy way for a user to find this out? (I guess you can look at the url of that game, it also includes the number)

[Hiccup]

...

Looks good. Yeah for some reason the archive number isn't displayed on the page, but you can see it in the URL.

One last thing with the file: the "extension" field should be set - "z64" for BigEndian format and "v64" for ByteSwapped format.

[wolfre]

One last thing with the file

Uff sorry for the delay, real life happened pretty hard ...

Anyways here's the first re-dump using all the above discussed points: viewtopic.php?f=7&t=5260
This was done with v1.1.0 of my dumper / tool: https://gitlab.com/wolfre/usb-cart-read ... ses/v1.1.0 As noted in the project readme, for producing the XMLs (or NFOs) no hardware is needed and so you could create a dump with another dumper and then use my tool generate the xml (nfo) as well

Also thanks for updating https://wiki.no-intro.org/index.php?tit ... Convention I think I also have observed all those points.

And you also suggested that we could create an XML schema which sounds like a good idea. I did some preliminary research starting from here https://en.wikipedia.org/wiki/XML_schema and it looks like the classic language to create those was DTD which was superseded by either RELAX_NG and/or W3C's XML Schema due to DTD being hard to understand / read etc...

Should we move that schema discussion to a new thread? If yes I can start that by doing so more research into the best format etc.