Future of datafiles

Post by **root** » 28 Mar 2008 18:48

Rif: 2080-1 Future of datafiles \ romcenter on 28th March 2008, 18:48 wrote:

Hi

I'm the developer of romcenter and I'm actually working on the new version 3.00. In the process of testing consoles datafiles, I downloaded the nointro genesis and found it very clean.
I would like to know if you're interested to improve your datafiles to make it work at best with romcenter.
The first thing to do would be to upgrade the format to the newly 'generic xml' http://forum.logiqx.com/viewtopic.php?t=363 format. This format is already supported by romcenter, clrmame and other tools.
There is not a lot of differences between your current clrmame format and the new format. It should be quite straightforward to adapt.
On the other hand, you would benefit of a better support with romcenter (mostly by using one parameters in xml header).

Before going further, tell me if you're interested.

Thank you.

===============================================

Re: Future of datafiles \ neoforma on 28th March 2008, 19:02 wrote:

Though I'm not affilated with No-Intro, the new XML format is the one I would most likely use in the future.

===============================================

Re: Future of datafiles \ xuom2 on 28th March 2008, 19:51 wrote:

a bit offtopic, please add in future to RC:

- 7zip support
- check/fix if the zip archives have the selected compression
- remove/add comments from archives
- warning for double filenames / hash dupes in dat

/OT

===============================================

Re: Future of datafiles \ kazumi213 on 28th March 2008, 21:34 wrote:

a bit offtopic, please add in future to RC:

- 7zip support
- check/fix if the zip archives have the selected compression
- remove/add comments from archives
- warning for double filenames / hash dupes in dat

/OT

Plus
- md5 & sha1 check (at least sha1)
- Text List generation from selected items within the program (DAT from selected was already possible in 2.71, but Text List was only possible for have/missing sets and ROMs)

Hi Eric, nice to hear from you here.

I'm not familiar with this new xml format, but it's just a matter of taking a look at it

My main concern about a new DAT format is about it not allowing to do things which we can do now with current format. To be specific the approach we have started to use here when fully merging.

The conventional (arcade) approach when fully merging is to use the archive (let's say a .zip) which is defined as parent to include all roms in clone archives. Example with a GBA "family"

Code: Select all

Golden Sun (France)
Golden Sun (Germany)
Golden Sun (Italy)
Golden Sun (Spain)
Golden Sun (USA, Europe)
Ougon no Taiyou - Hirakareshi Fuuin (Japan)

Let's say we define "Golden Sun (USA, Europe)" as the parent. The conventional way of fully merging would include all ROM files in the archive:

Golden Sun (USA, Europe).zip

What we started to use (and is allowed by current CM and RC formats) is to merge all ROMs in the file:

Golden Sun.zip

(without tags). The problem is that such parent doesn't really exist. I found a way to workaround it by defining "fake parents" like that within CM and RC DATs. It works very well and would be even better if CM and RC supported .7z format.

So new XML format should support this too, in some way. It would also be great if it supported an option already present in GoodMerge xml format: Fake parent title should be picked depending on user Region preference. In the example above a Japanese user would prefer the family archive to be named:

Ougon no Taiyou - Hirakareshi Fuuin.zip

instead of "Golden Sun.zip"

All required info for this would be included by the datter, it's just a matter of the format to support the inclusion of such information.

I know this may sound a bit weird, so please check an actual example: my Parent-Clone DAT and XMDB file for GBA system. It includes DATs which work on latest CM and RC 2.71. Check the readme for further info and feel free to post any feedback.

I think this features would be really simple to implement in new xml format (it is already working in current formats) and has the potential to create a truly awesome integration: a ROM manager with GoodMerge advantages.

The same approach has been succesfully applied to the NDS system but in automated and customizable fashion in xuom2's DAT-O-MATIC.

===============================================

Re: Future of datafiles \ romcenter on 30th March 2008, 20:06 wrote:

Thank you.

I thought about all that, and I found a solution. I will write it down and propose it here for validation.

Stay tuned

===============================================

Re: Future of datafiles \ Stahn on 30th March 2008, 20:32 wrote:

I would love to use Romcenter again... clrMAME still is one of the worst designed apps in the history of software.

===============================================

Re: Future of datafiles \ romcenter on 31st March 2008, 14:33 wrote:

Here are some proposals :

Note: As rc3.00 is almost ready for release, the following changes won't be included. I just included the search function (which were disabled until now) so that you can still use the current datafile.

Generic xml format
First, here is the new 'generic xml' format (current version):

Code: Select all

<?xml version="1.0"?>
<!DOCTYPE datafile PUBLIC "-//Logiqx//DTD ROM Management Datafile//EN" "http://www.logiqx.com/Dats/datafile.dtd">
<datafile>
	<header>
		<name>Sega - Mega Drive - Genesis</name>
		<description>Sega - Mega Drive - Genesis</description>
		<category>no-intro romset</category>
		<version>20080228</version>
		<author>Gigadeath | www.gbadat.altervista.org</author>
		<romcenter 
			plugin="genesis.dll"
			rommode "split" (merged|split|unmerged)
			biosmode "split" (merged|split|unmerged)  
			samplemode "merged" (merged|unmerged)  
			lockrommode "no"
			lockbiosmode "no"
			locksamplemode "no" />
	</header>
	<game name="007 Shitou - The Duel (Japan)">
		<description>007 Shitou - The Duel (Japan)</description>
		<rom name="007 Shitou - The Duel (Japan)" size="524288" crc="aeb4b262" sha1="7e0de7011a60a1462dc48594f3caa956ff942281"/>
	</game>
<datafile>

This format adds a 'header' section, a 'romcenter' and a 'clrmame' section (not shown here). The game format is the mame format.
This new format is used with romcenter 3, clrmame and datutil, and possibly by other tools. You can convert any dat to generic xml with logix tool datutil.
If lock modes is set, options to change modes in romcenter is disabled. This lock the default mode from the dat.

Families
I analysed the snes nointro dat and I am amazed on the work you have done since my last visit (a long time ago)!
The goal is to add a 'family' concept directly in the dat.
I propose to adapt the generic xml like that:

Code: Select all

<datafile>
	<header>
		<name>Nintendo - Super Nintendo Entertainment System (Parent-Clone)</name>
		<description>Sega - Mega Drive - Genesis</description>
		<category>No intro Parent-Clone relations</category>
		<version>20071019</version>
		<author>kazumi213 & tetsuo55</author>
		<romcenter 
			plugin="snes.dll"
			rommode "merged" (default, can be set from rc)
			lockrommode "no" (except if locked)
	</header>
	<game name="Aladdin (Europe)">
		<family name="Aladdin" Language="ENG">
		<family name="JapName" Language="JAP">
		<description>Aladdin (Europe)</description>
		<rom name="Aladdin (Europe)" size="1310720" crc="44ff6e78" sha1="fd155afd8ffa81b0dc6e7990a18212538882e626"/>
	</game>
	<game name="Aladdin (France)" cloneof="Aladdin (Europe)">
		<family name="Aladdin" Language="ENG">
		<description>Aladdin (France)</description>
		
		<rom name="Aladdin (France)" size="1310720" crc="63069939" sha1="d0fc2d457dde02cb28b4d1c7248c97d1711e2fe2"/>
	</game>
	<game name="Aladdin (Germany)">
		<family name="Aladdin" Language="ENG" cloneof="Aladdin (Europe)">
		<description>Aladdin (Germany)</description>
		<rom name="Aladdin (Germany)" size="1310720" crc="a7ad3e5d" sha1="a06aa6f4bb62951884adb3c7813fd93337d4750b"/>
	</game>
<datafile>

Note the new 'family' tag. I will add an option, under the merge rom mode, to enable or not the 'family' names, and to choose the prefered language (extracted from the dat language field). Once enabled, the family name is used instead of game name.

Plugin
I noticed you didn't use the rc plugins. I tried with the genesis dat, using genesis.dll plugin, and it runs very well. The plugin can correctly identify a game in any format. I think it could be really interesting with roms with different formats.

Sha-1
I add it to the todo list for a future version.

7zip
I began to look for a component to handle 7zip.

Save text list of selected items
Ok, in next version.

Tell me what you think, before I submit this change to Logiqx and Roman.

===============================================

Re: Future of datafiles \ kazumi213 on 31st March 2008, 20:37 wrote:

Ok Eric, first of all thanks for taking the time to look into this "family" subject.

I've converted both Parent-Clone GBA and SNES CM DATs to "generic xml" format with 0 problems using latest DatUtil v2.35. I have to say that they look exactly the same as their CM counterparts except for the "<>" usage, closing tags, "&" -> "&" conversion and so on, already seen when dealing with XMDB files.

So conversion of current official No-Intro DATs would be a piece of cake, just a matter of a command line pass, then fine tune the "header" section to take advantage of specific features in CM and RC.

Now regarding your proposal: I like it, that's the idea. It removes all excess "game" entries required to declare the "fake parents" (family archive names), so the problem of wrong game/rom counts should be gone. Also I see that you don't use "romof" declarations. I guess this is unnecessary and I should remove them even on current (listinfo) CM DATs.

However there are some aspects which I would like to discuss. I've highlighted them.

Code: Select all

<datafile>
	<header>
		<name>Nintendo - Super Nintendo Entertainment System (Parent-Clone)</name>
		<description>Sega - Mega Drive - Genesis</description>
		<category>No intro Parent-Clone relations</category>
		<version>20071019</version>
		<author>kazumi213 & tetsuo55</author>
		<romcenter 
			plugin="snes.dll"
			rommode "merged" (default, can be set from rc)
			lockrommode "no" (except if locked)
	</header>
	<game name="Aladdin (Europe)">
		<family name="Aladdin" Language="ENG">
		<family name="JapName" Language="JAP">
		<description>Aladdin (Europe)</description>
		<rom name="Aladdin (Europe)" size="1310720" crc="44ff6e78" sha1="fd155afd8ffa81b0dc6e7990a18212538882e626"/>
	</game>
	<game name="Aladdin (France)" cloneof="Aladdin (Europe)">
		<family name="Aladdin" Language="ENG">  <=======
		<description>Aladdin (France)</description>
		<rom name="Aladdin (France)" size="1310720" crc="63069939" sha1="d0fc2d457dde02cb28b4d1c7248c97d1711e2fe2"/>
	</game>
	<game name="Aladdin (Germany)">
		<family name="Aladdin" Language="ENG" cloneof="Aladdin (Europe)">  <=======
		<description>Aladdin (Germany)</description>
		<rom name="Aladdin (Germany)" size="1310720" crc="a7ad3e5d" sha1="a06aa6f4bb62951884adb3c7813fd93337d4750b"/>
	</game>
<datafile>

I assume "cloneof" declaration for "Aladdin (Germany)" should be next to "game name" declaration, like in the (France) entry.

My question is: Given "family name" options are already declared for the reference/parent/main game ("Aladdin (Europe)" here) and the other games in the family are linked to it via "cloneof" declarations, why is it required to repeat "family name" declarations on them?

Finally, regarding "language" options to provide, note it is a country/region concept, and not language-oriented (ok, they usually match). I mean, I suggest calling it "region" instead of "language" (GoodMerge uses "bias zone" if you prefer it, maybe even better for an easier adoption of format and concept).

I assume you mean that RC will populate the list of choices based on detected "region/bias zone" declaration after parsing the DAT file. Thats the best approach, because this leaves to the datter the option to define as many regions as required. Just remember that the user must choose a "region order preference", not a "specific region". Some families can not have such specific region declaration and the ROM manager should be able to decide which one to use from the actual declared ones. This is the GoodMerge behavior.

Plugin
I noticed you didn't use the rc plugins. I tried with the genesis dat, using genesis.dll plugin, and it runs very well. The plugin can correctly identify a game in any format. I think it could be really interesting with roms with different formats.

I will leave this question this to the more experienced members here.

Sha-1
I add it to the todo list for a future version.

7zip
I began to look for a component to handle 7zip.

Save text list of selected items
Ok, in next version.

Tell me what you think, before I submit this change to Logiqx and Roman.

Except for the last (just a nice and useful addition), all of them are required options for a competitive (vs CM) rom manager nowadays. Just to be sure, 7-zip support means reading *and* creation of .7z files (CM can not create .7z)

Thanks again Eric.

===============================================

Re: Future of datafiles \ NGEfreak on 1st April 2008, 06:40 wrote:

I think it's better to use ISO codes for the language/country definition. Also, in some cases a combination of both may be needed:
Harry Potter and the Sorcerer's Stone --> en-US
Harry Potter and the Philosopher's Stone --> en-GB

Plugin
I noticed you didn't use the rc plugins. I tried with the genesis dat, using genesis.dll plugin, and it runs very well. The plugin can correctly identify a game in any format. I think it could be really interesting with roms with different formats.

This feature is only needed for systems which can not be easily converted to a consistent format (like NES). Mega Drive or SNES for that matter don't need this and can be easily converted.

===============================================

Re: Future of datafiles \ romcenter on 1st April 2008, 09:29 wrote:

Also I see that you don't use "romof" declarations. I guess this is unnecessary and I should remove them even on current (listinfo) CM DATs.

Yes, I dont't think it's necessary for console emulators. Romof is used when you give to the emulator the game name (like in mame), and not the file. Mame have to find itself the roms for that game and 'romof' tells where the roms have to be searched.

I assume "cloneof" declaration for "Aladdin (Germany)" should be next to "game name" declaration, like in the (France) entry.

Yes, right.

My question is: Given "family name" options are already declared for the reference/parent/main game ("Aladdin (Europe)" here) and the other games in the family are linked to it via "cloneof" declarations, why is it required to repeat "family name" declarations on them?

The family info is not necessary on clones, it's correct, but it means you can only put in a family games which are clones. Is it correct for you ?

Finally, regarding "language" options to provide, note it is a country/region concept, and not language-oriented (ok, they usually match). I mean, I suggest calling it "region" instead of "language" (GoodMerge uses "bias zone" if you prefer it, maybe even better for an easier adoption of format and concept).

I prefer "Region", which is also used in some tools.

I assume you mean that RC will populate the list of choices based on detected "region/bias zone" declaration after parsing the DAT file. Thats the best approach, because this leaves to the datter the option to define as many regions as required. Just remember that the user must choose a "region order preference", not a "specific region". Some families can not have such specific region declaration and the ROM manager should be able to decide which one to use from the actual declared ones. This is the GoodMerge behavior.

Yes, this is the idea. I build the family list when parsing the dat, and the user choose the prefered region, and a default one to use when the prefered is not available. If the default is not available, or no family is specified, then the 'main' gamename is used.

Here is the fixed format:

Code: Select all

<datafile>
	<header>
		<name>Nintendo - Super Nintendo Entertainment System (Parent-Clone)</name>
		<description>Sega - Mega Drive - Genesis</description>
		<category>No intro Parent-Clone relations</category>
		<version>20071019</version>
		<author>kazumi213 & tetsuo55</author>
		<romcenter 
			plugin="snes.dll"
			rommode "merged" (default, can be set from rc)
			lockrommode "no" (except if locked)
	</header>
	<game name="Aladdin (Europe)">
		<family name="Aladdin" Region="ENG">
		<family name="JapName" Language="JAP">
		<description>Aladdin (Europe)</description>
		<rom name="Aladdin (Europe)" size="1310720" crc="44ff6e78" sha1="fd155afd8ffa81b0dc6e7990a18212538882e626"/>
	</game>
	<game name="Aladdin (France)" cloneof = "Aladdin (Europe)">
		<description>Aladdin (France)</description>
		<rom name="Aladdin (France)" size="1310720" crc="63069939" sha1="d0fc2d457dde02cb28b4d1c7248c97d1711e2fe2"/>
	</game>
	<game name="Aladdin (Germany)" cloneof = "Aladdin (Europe)">
		<description>Aladdin (Germany)</description>
		<rom name="Aladdin (Germany)" size="1310720" crc="a7ad3e5d" sha1="a06aa6f4bb62951884adb3c7813fd93337d4750b"/>
	</game>
<datafile>

I think it's better to use ISO codes for the language/country definition.

As the family name list is build from the dat, it's up to the dat maker to choose region labels. Now, you can give some rules to build new dats.

This feature is only needed for systems which can not be easily converted to a consistent format (like NES). Mega Drive or SNES for that matter don't need this and can be easily converted.

For sure, it will be better to use only one format, but this is not the case. Plugins can analyse the file format, and adapt crc calculation to identify correctly the game. Look at the screenshot.

In winzip, you see that the three files have three different crc. A standart rom manager will only recognize the .bin and if you only have the smd, you don't know if the file is correct or not.
The genesis plugin give the same crc for the three files, and romcenter correctly identify them.

===============================================

Re: Future of datafiles \ bigfred on 1st April 2008, 10:54 wrote:

For sure, it will be better to use only one format, but this is not the case. Plugins can analyse the file format, and adapt crc calculation to identify correctly the game

We want to document the dumps as they are physically stored on the ROM if possible - therefore we classify every format that is not the original format as an incorrect dump. Such a feature would only help if the dumps could be converted to the original format.

===============================================

Re: Future of datafiles \ kazumi213 on 1st April 2008, 14:24 wrote:

Yes, this is the idea. I build the family list when parsing the dat, and the user choose the prefered region, and a default one to use when the prefered is not available. If the default is not available, or no family is specified, then the 'main' gamename is used.

Keep the idea in bold, as that would be the best option for some "old school" users.

However just a "prefered" and "default" region is not enough. That would work when only 3 regions are declared (like in current No-intro Parent-Clone DATs).

Example: a French user chooses "FRA" as prefered and "EUR" as default. But there are the "Shin-chan" games (GBA/NDS) which exist only for "JPN" and "ESP". As the datter, I choose the "JPN" one to be the parent. But the French user could prefer to use the "ESP" name. It would be forced to use the "JPN" family name (or even worse, the tagged JPN name, according to your description when no default family name is available).

So the user should be able to "sort" the list of detected region labels. I imagine this in a "move up/down" box list fashion. Then the ROM manager would pick the "family name" based on the label priority according to list (again, this is the approach in GoodMerge, sorry )

The family info is not necessary on clones, it's correct, but it means you can only put in a family games which are clones. Is it correct for you ?

Absolutely yes. Here at No-Intro the games in a family are "region dupes", so they are clones indeed.

However wait a minute. Do you mean that a game would be included in a given family archive just by declaring the "family name" on its entry? Without requiring it to be linked to anything by "cloneof" declaration?

Because that would lead to an unexpected but very, very interesting feature: super families (archives containing i.e. a whole series). Example (GBA):

Code: Select all

<game name="Breath of Fire (Europe)">
	<family name="Breath of Fire" region="EUR">
	<family name="Breath of Fire" region="USA">
	<family name="Breath of Fire - Ryuu no Senshi" region="JPN">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire (Europe) (EUR PARENT) (EUR PARENT SET)</description>
	<rom name="Breath of Fire (Europe).gba" size="4194304" crc="a1c3165d" sha1="f47870d25665588d19b75e90d0bd32a759e64918"/>
</game>
<game name="Breath of Fire (Europe) (En,Fr,De)" cloneof="Breath of Fire (Europe)">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire (Europe) (En,Fr,De)</description>
	<rom name="Breath of Fire (Europe) (En,Fr,De).gba" size="4194304" crc="d3660549" sha1="fdc7e7b6ab680229dcd159eb4dd7d1967b9e88ee"/>
</game>
<game name="Breath of Fire (USA)" cloneof="Breath of Fire (Europe)">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire (USA) (USA PARENT) (USA PARENT SET)</description>
	<rom name="Breath of Fire (USA).gba" size="4194304" crc="f06422a8" sha1="b30533f68037b47d5439bab0182169e4a643a38d"/>
</game>
<game name="Breath of Fire - Ryuu no Senshi (Japan)" cloneof="Breath of Fire (Europe)">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire - Ryuu no Senshi (Japan) (JPN PARENT) (JPN PARENT SET)</description>
	<rom name="Breath of Fire - Ryuu no Senshi (Japan).gba" size="4194304" crc="66e96b35" sha1="b09dc97ab42f5e32dc6254d09b0b195295b24932"/>
</game>

<game name="Breath of Fire II (Europe)">
	<family name="Breath of Fire II" region="EUR">
	<family name="Breath of Fire II" region="USA">
	<family name="Breath of Fire II - Shimei no Ko" region="JPN">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire II (Europe) (EUR PARENT) (EUR PARENT SET)</description>
	<rom name="Breath of Fire II (Europe).gba" size="4194304" crc="ff5e8e39" sha1="307798ca71bda7f6b313ff1c18880f22c5a81711"/>
</game>
<game name="Breath of Fire II (USA)" cloneof="Breath of Fire II (Europe)">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire II (USA) (USA PARENT) (USA PARENT SET)</description>
	<rom name="Breath of Fire II (USA).gba" size="4194304" crc="6f098da3" sha1="d35e452d467093c4a788d290a65adf05a4270343"/>
</game>
<game name="Breath of Fire II - Shimei no Ko (Japan)" cloneof="Breath of Fire II (Europe)">
	<family name="Breath of Fire Series" region="SUPER EUR">   <=====
	<description>Breath of Fire II - Shimei no Ko (Japan) (JPN PARENT) (JPN PARENT SET)</description>
	<rom name="Breath of Fire II - Shimei no Ko (Japan).gba" size="4194304" crc="8a824a70" sha1="e862d66da9286b4c938441bee2fd1a8e980d694f"/>
</game>

Do you mean that if the user select "SUPER EUR" as the first choice from the region labels list, all ROMs in *both* families would be included in the archive "Breath of Fire Series"?

I'm not saying No-Intro would use such a feature, but that would satisfy ALL end-user collection customization needs. Add .7z support here and that could lead to HUGE size savings. In example: GBA Pokemon games share 60%+ common code across the series.

===============================================

Re: Future of datafiles \ snakemeat on 2nd April 2008, 03:34 wrote:

This is a great topic, I was wondering if there was an XSD available for this new XML format?

===============================================

Re: Future of datafiles \ romcenter on 2nd April 2008, 06:34 wrote:

Not a xsd, but a dtd: http://www.logiqx.com/Dats/datafile.dtd

===============================================

Re: Future of datafiles \ NGEfreak on 2nd April 2008, 08:10 wrote:

Hmm, I don't fully understand how this works.

Here is an example:

Code: Select all

We have the following roms with titles:

Super Mario World (USA).rom
  -> Super Mario World (English)

Super Mario World (Europe).rom
  -> Super Mario World (English)

Super Mario World - Super Mario Bros. 4 (Japan).rom
  -> Super Mario World - Super Mario Bros. 4 (Japanese)

--------------------

Super Contra (USA).rom
  -> Super Contra (US English)

Super Probotector (Europe).rom
  -> Super Probotector (UK English)

--------------------

Harry Potter and the Sorcerer's Stone (USA, Europe).rom
  -> Harry Potter and the Sorcerer's Stone (US English)
  -> Harry Potter and the Philosopher's Stone (UK English)
  -> Harry Potter und der Stein der Weisen (German)

Harry Potter to Kenja no Ishi (Japan).rom
  -> Harry Potter to Kenja no Ishi (Japanese)

==========================================================

Now, we have two users.

User A wants to merge these roms like this:

Super Mario World - Super Mario Bros. 4.zip
  Super Mario World (USA).rom
  Super Mario World (Europe).rom
  Super Mario World - Super Mario Bros. 4 (Japan).rom

Super Contra.zip
  Super Contra (USA)
  Super Probotector (Europe)

Harry Potter to Kenja no Ishi.zip
  Harry Potter and the Sorcerer's Stone (USA, Europe).rom
  Harry Potter to Kenja no Ishi (Japan).rom

This user prefers Japanese titles the most, US English the second most, etc:
Japanese, US English, UK English, English, German

--------------------

User B wants to merge them like this:

Super Mario World.zip
  Super Mario World (USA).rom
  Super Mario World (Europe).rom
  Super Mario World - Super Mario Bros. 4 (Japan).rom

Super Probotector.zip
  Super Contra (USA)
  Super Probotector (Europe)

Harry Potter und der Stein der Weisen.zip
  Harry Potter and the Sorcerer's Stone (USA, Europe).rom
  Harry Potter to Kenja no Ishi (Japan).rom

This user prefers German titles the most. UK English the second most, etc:
German, UK English, US English, English, Japanese

How do I implement this concept into the dat?

From my understanding so far, the dat would look like this:

Code: Select all

<game name="Super Mario World (USA)">
	<family name="Super Mario World" region="en" />
	<description>Super Mario World (USA)</description>
	<rom name="Super Mario World (USA).rom" ... />
</game>
<game name="Super Mario World (Europe)" cloneof="Super Mario World (USA)">
	<family name="Super Mario World" region="en" />
	<description>Super Mario World (Europe)</description>
	<rom name="Super Mario World (Europe).rom" ... />
</game>
<game name="Super Mario World - Super Mario Bros. 4 (Japan)" cloneof="Super Mario World (USA)">
	<family name="Super Mario World - Super Mario Bros. 4" region="ja" />
	<description>Super Mario World - Super Mario Bros. 4 (Japan)</description>
	<rom name="Super Mario World - Super Mario Bros. 4 (Japan).rom" ... />
</game>

<game name="Super Contra (USA)">
	<family name="Super Contra" region="en-US" />
	<description>Super Contra (USA)</description>
	<rom name="Super Contra (USA).rom" ... />
</game>
<game name="Super Probotector (Europe)" cloneof="Super Contra (USA)">
	<family name="Super Probotector" region="en-GB" />
	<description>Super Probotector (Europe)</description>
	<rom name="Super Probotector (Europe).rom" ... />
</game>

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<family name="Harry Potter and the Sorcerer's Stone" region="en-US" />
	<family name="Harry Potter and the Philosopher's Stone" region="en-GB" />
	<family name="Harry Potter und der Stein der Weisen" region="de" />
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<family name="Harry Potter to Kenja no Ishi" region="ja" />
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

Post by **root** » 02 Apr 2008 10:18

Rif: 2080-2 Re: Future of datafiles \ romcenter on 2nd April 2008, 11:18 wrote:

Hi

Do you mean that a game would be included in a given family archive just by declaring the "family name" on its entry?

Yes I do. No link between family and anything else. And I think that finally, cloneof should not be used here.
The 'SUPER EUR' region is already a kind of 'hack' to do special things with families.

I prefer add now a new feature to handle 'super family', or 'group'. See below.

From my understanding so far, the dat would look like this:

It's almost correct, but I have several comments:

- cloneof is not used as it should. cloneof is only needed in multi roms games, with clones using identical roms. It avoids to duplicate files. It should not be used to link families.
- If we remove cloneof tag, we must set correct families info for each file.
- If we want to be able to handle several families level (kazumi213 'super family'), I introduced a 'group' concept:

(and I also change the layout to be more xml-friendly)

Code: Select all

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<families>
	  <groups name="A">
	    <family region="en-US">Harry Potter and the Sorcerer's Stone</family>
	    <family region="en-GB">Harry Potter and the Philosopher's Stone</family>
	    <family region="de">Harry Potter und der Stein der Weisen</family>
	    <family region="ja">Harry Potter to Kenja no Ishi</family>
	  </groups>
	  <groups name="B">
	    <family region="en">Harry Potter</family>
	  </groups>
	</families>
	...
</game>
<game name="Harry Potter and the Goblet of Fire (USA, Europe)">
	<families>
	  <groups name="A">
	    <family region="en">Harry Potter and the Goblet of Fire</family>
	  </groups>
	  <groups name="B">
	    <family region="en">Harry Potter</family>
	  </groups>
	</families>
	...
</game>

Group is selected for all the datafile. If you choose to use the 'families', you must select the family group and then sort the family regions.

Code: Select all

Group A, en-US, en:
  Harry Potter and the Sorcerer's Stone.zip
	- Harry Potter and the Sorcerer's Stone (USA, Europe)
  Harry Potter and the Goblet of Fire.zip
  	- Harry Potter and the Goblet of Fire (USA, Europe)

Code: Select all

Group A, ja, en
  Harry Potter to Kenja no Ishi.zip
	- Harry Potter and the Sorcerer's Stone (USA, Europe)
  Harry Potter and the Goblet of Fire.zip
  	- Harry Potter and the Goblet of Fire (USA, Europe)

Code: Select all

Group B, en
  Harry Potter.zip
	- Harry Potter and the Sorcerer's Stone (USA, Europe)
  	- Harry Potter and the Goblet of Fire (USA, Europe)

Here is the exemple of NGEfreak with new generic xml format:
(no need to use the Group feature in this case)

fixed NGEfreak xml

Code: Select all

<game name="Super Mario World (USA)">
	<families>
	  <groups name="A">
	    <family region="ja">Super Mario World - Super Mario Bros. 4</family>
	    <family region="en">Super Mario World</family>
	  </groups>
	</families>
	<description>Super Mario World (USA)</description>
	<rom name="Super Mario World (USA).rom" ... />
</game>
<game name="Super Mario World (Europe)">
	<families>
	  <groups name="A">
	    <family region="ja">Super Mario World - Super Mario Bros. 4</family>
	    <family region="en">Super Mario World</family>
	  </groups>
	</families>
	<description>Super Mario World (Europe)</description>
	<rom name="Super Mario World (Europe).rom" ... />
</game>
<game name="Super Mario World - Super Mario Bros. 4 (Japan)">
	<families>
	  <groups name="A">
	    <family region="ja">Super Mario World - Super Mario Bros. 4</family>
	    <family region="en">Super Mario World</family>
	  </groups>
	</families>
	<description>Super Mario World - Super Mario Bros. 4 (Japan)</description>
	<rom name="Super Mario World - Super Mario Bros. 4 (Japan).rom" ... />
</game>

<game name="Super Contra (USA)">
	<families>
	  <groups name="A">
	    <family region="en-US">Super Contra</family>
	    <family region="en-GB">Super Probotector</family>
	  </groups>
	</families>
	<description>Super Contra (USA)</description>
	<rom name="Super Contra (USA).rom" ... />
</game>
<game name="Super Probotector (Europe)" cloneof="Super Contra (USA)">
	<families>
	  <groups name="A">
	    <family region="en-US">Super Contra</family>
	    <family region="en-GB">Super Probotector</family>
	  </groups>
	</families>
	<description>Super Probotector (Europe)</description>
	<rom name="Super Probotector (Europe).rom" ... />
</game>

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<families>
	  <groups name="A">
	    <family region="en-US">Harry Potter and the Sorcerer's Stone</family>
	    <family region="en-GB">Harry Potter and the Philosopher's Stone</family>
	    <family region="de">Harry Potter und der Stein der Weisen</family>
	    <family region="ja">Harry Potter to Kenja no Ishi</family>
	  </groups>
	</families>
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<families>
	  <groups name="A">
	    <family region="en-US">Harry Potter and the Sorcerer's Stone</family>
	    <family region="en-GB">Harry Potter and the Philosopher's Stone</family>
	    <family region="de">Harry Potter und der Stein der Weisen</family>
	    <family region="ja">Harry Potter to Kenja no Ishi</family>
	  </groups>
	</families>
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

===============================================

Re: Future of datafiles \ kazumi213 on 2nd April 2008, 11:36 wrote:

requested example:

Code: Select all

<game name="Super Mario World (USA)">
	<family name="Super Mario World" region="USA" />
	<family name="Super Mario World" region="EUR" />
	<family name="Super Mario World - Super Mario Bros. 4" region="JPN" />
	<description>Super Mario World (USA)</description>
	<rom name="Super Mario World (USA).rom" ... />
</game>
<game name="Super Mario World (Europe)" cloneof="Super Mario World (USA)">
	<description>Super Mario World (Europe)</description>
	<rom name="Super Mario World (Europe).rom" ... />
</game>
<game name="Super Mario World - Super Mario Bros. 4 (Japan)" cloneof="Super Mario World (USA)">
	<description>Super Mario World - Super Mario Bros. 4 (Japan)</description>
	<rom name="Super Mario World - Super Mario Bros. 4 (Japan).rom" ... />
</game>

<game name="Super Contra (USA)">
	<family name="Super Contra" region="USA" />
	<family name="Super Probotector" region="EUR" />
	<description>Super Contra (USA)</description>
	<rom name="Super Contra (USA).rom" ... />
</game>
<game name="Super Probotector (Europe)" cloneof="Super Contra (USA)">
	<description>Super Probotector (Europe)</description>
	<rom name="Super Probotector (Europe).rom" ... />
</game>

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<family name="Harry Potter and the Sorcerer's Stone" region="USA" />
	<family name="Harry Potter and the Philosopher's Stone" region="EUR" />
	<family name="Harry Potter und der Stein der Weisen" region="GER" />
	<family name="Harry Potter to Kenja no Ishi" region="JPN" />
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

I've just used different "region" labels (country-oriented) and put all "family names" below the reference/parent/main in each family because they are "true" families ("cloneof"-linked). Works as requested and the end-user just need to sort the following list of choices:

EUR
GER
JPN
USA

User A would sort it as:

JPN
USA
EUR
GER

While for user B:

GER
EUR
USA
JPN

Note I've used just 4 labels while you used 5.

From No-Intro point of view and regarding the official implementation of this concept in DATs, "family names" should come from existing ROM titles in DAT and ROMs in DAT are included in a per country/region basis, not language.

According to our naming convention (which must prevail) the "Harry Potter (USA, Europe)" ROM must be named using US English. We don't list it with all possible translated names. So your "en-GB" and "de" (or my EUR and GER) declaration should not be allowed in any No-Intro DAT. As a rule of thumb: we should use *at most* as many "family name" declarations as ROMs in a family, using their country/region to create the label and its convention-compliant name to create the "family names". In the end I think this is more intuitive than a language approach, which could lead to a real mess in ROM collection naming across the world.

But that's the good point of this "family name" concept: your example allows to show that such a complex DAT which goes into the detail of using all possible translated names is possible. If the "super families" are possible too, it will allow for really, really creative DATs.

EDIT:

This post was a reply to NGEfreak, I was typing while you posted Eric.

Regarding your changes... mmh I don't like them

If the previous format has "super familes" as a bonus, ok. But If they require this new format, well, they come at some important problems cost.

- Previous format is essentially well known listinfo structure with a couple new variables to parse: "family name" and "region". From ROM manager (any) support implementation point of view, previous format advantages/engine-recoding-troubles ratio is more appealing.

- Previous format is more intuitive and does the work as well. Simplicity is elegant. Datters will be happier

- While I agree that in previous format "family name" usage makes "cloneof" declarations unnecesary, that doesn't mean they are useless. I want to keep "cloneof" declarations, as they are used to determine how many "unique" games exist in the DAT (CM and RC set/games counters) and allows for a much cleaner and organized display (at least in RC v2.71, hope you keep that game tree view on v3). Plus removing "cloneof" declaration increases DAT size a lot.

===============================================

Re: Future of datafiles \ romceter on 3rd April 2008, 11:31 wrote:

It's a good solution, quite clean and not too diffucult to code (except region sorting ).

Automerged Doublepost.

I changed the family info layout
from
<family name="Super Mario World" region="USA"/>
to
<family region="USA">Super Mario World</family>

final solution:

Code: Select all

<game name="Super Mario World (USA)">
	<family region="USA">Super Mario World</family>
	<family region="EUR">Super Mario World</family>
	<family region="JPN">Super Mario World - Super Mario Bros. 4</family>
	<description>Super Mario World (USA)</description>
	<rom name="Super Mario World (USA).rom" ... />
</game>
<game name="Super Mario World (Europe)" cloneof="Super Mario World (USA)">
	<description>Super Mario World (Europe)</description>
	<rom name="Super Mario World (Europe).rom" ... />
</game>
<game name="Super Mario World - Super Mario Bros. 4 (Japan)" cloneof="Super Mario World (USA)">
	<description>Super Mario World - Super Mario Bros. 4 (Japan)</description>
	<rom name="Super Mario World - Super Mario Bros. 4 (Japan).rom" ... />
</game>

<game name="Super Contra (USA)">
	<family region="USA">Super Contra</family>
	<family region="EUR">Super Probotector</family>
	<description>Super Contra (USA)</description>
	<rom name="Super Contra (USA).rom" ... />
</game>
<game name="Super Probotector (Europe)" cloneof="Super Contra (USA)">
	<description>Super Probotector (Europe)</description>
	<rom name="Super Probotector (Europe).rom" ... />
</game>

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<family region="USA">Harry Potter and the Sorcerer's Stone</family>
	<family region="EUR">Harry Potter and the Philosopher's Stone</family>
	<family region="GER">Harry Potter und der Stein der Weisen</family>
	<family region="JPN">Harry Potter to Kenja no Ishi</family>
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

===============================================

Re: Future of datafiles \ kazumi213 on 3rd April 2008, 12:58 wrote:

Those kind of changes are no problem, of course

I love your solution

and hope it is accepted by logiqx and included in generic xml format.

Thank you very much Eric.

===============================================

Re: Future of datafiles \ NGEfreak on 4th April 2008, 10:17 wrote:

From No-Intro point of view and regarding the official implementation of this concept in DATs, "family names" should come from existing ROM titles in DAT and ROMs in DAT are included in a per country/region basis, not language.

According to our naming convention (which must prevail) the "Harry Potter (USA, Europe)" ROM must be named using US English. We don't list it with all possible translated names. So your "en-GB" and "de" (or my EUR and GER) declaration should not be allowed in any No-Intro DAT. As a rule of thumb: we should use *at most* as many "family name" declarations as ROMs in a family, using their country/region to create the label and its convention-compliant name to create the "family names". In the end I think this is more intuitive than a language approach, which could lead to a real mess in ROM collection naming across the world.

Language or a combination of language and country tags are better than just country tags. Country alone is not sufficient enough. A country can also have more than one release in different languages. For example there are some cases in Spain with one release in Spanish and another one in Catalan.

Imho, it makes more sense if we include all official translated names. Otherwise this feature is almost pointless. If I want to name my sets using German for example, I would want to name all sets in German (of course only official names) and not just a very small bunch. It’s kinda silly to have a collection where Pokemon games are in German named sets, but Harry Potter in English named sets.

I changed the family info layout
from
<family name="Super Mario World" region="USA"/>
to
<family region="USA">Super Mario World</family>

All families are currently defined in the parent set. Is it also possible to split these to the clone sets? I want to list for example the Japanese family title in the actual Japanese set and not in the parent that has no relation to the Japanese title. It’s much easier like this from a DAT creation / back end source database perspective.

===============================================

Re: Future of datafiles \ BigFred on 4th April 2008, 11:17 wrote:

Maybe at this opportunity we could find a solution for the NES-header issue? If one could just include the header info in the dat and RC would fix the header accordingly the debate would finally end.

===============================================

Re: Future of datafiles \ kazumi213 on 4th April 2008, 14:25 wrote:

Language or a combination of language and country tags are better than just country tags. Country alone is not sufficient enough. A country can also have more than one release in different languages. For example there are some cases in Spain with one release in Spanish and another one in Catalan.

Good point. However, as long as the Catalan title(s) exist in the current DAT, I don't see the problem in creating additional tags like "SPA (Catalan)" or "SPA-Cat" or whatever is finally decided.

Imho, it makes more sense if we include all official translated names. Otherwise this feature is almost pointless.

In no way is pointless as described in other posts. Currently we can only offer EUR>USA>JPN (or USA>EUR>JPN in SNES P/C DAT). If new format is finally available user could choose the specific order for those 3 or (if we go ahead and include labels for all possible regions in a given DAT) for a list of 10+ regions. The main point here is that even the latter is feasible and error free.

If I want to name my sets using German for example, I would want to name all sets in German (of course only official names) and not just a very small bunch. It’s kinda silly to have a collection where Pokemon games are in German named sets, but Harry Potter in English named sets.

Just to be sure, please note that family names will be used only to name the archives (.zip/.7z) resulting from fully merging the collection. Inside these archives there will be all/available ROMs in each family and they will retain their current naming. If not fully-merging, current names will be used for archives.

What you want is too "personal tastes oriented". We have a convention to name the ROMs. If you as an end-user want to go ahead and rename all your archives with official german translations, ok (this could be hard to find & confirm for some systems). But honestly I think this has nothing to do with No-Intro. We still have consistancy problems using just 1 title set. And it's not silly to have a multilanguage-named collection as the fact is many of the ROMs were not released as Germany-exclusive titles.

I'm not saying what you want is impossible, but very unrealistic to accomplish to the same degree of quality (and timeframe) as in the intended implementation that I've described.

All families are currently defined in the parent set. Is it also possible to split these to the clone sets? I want to list for example the Japanese family title in the actual Japanese set and not in the parent that has no relation to the Japanese title. It’s much easier like this from a DAT creation / back end source database perspective.

Yes, it's possible. But as I said above, family names are just alternate names for the archive containing all the ROMs in the family. I think it makes more sense to have them nicely grouped below the family reference game.

===============================================

Re: Future of datafiles \ romcenter on 5th April 2008, 09:08 wrote:

Maybe at this opportunity we could find a solution for the NES-header issue? If one could just include the header info in the dat and RC would fix the header accordingly the debate would finally end.

What is the "nes header issue" ?

===============================================

Re: Future of datafiles \ NGEfreak on 7th April 2008, 08:12 wrote:

In no way is pointless as described in other posts. Currently we can only offer EUR>USA>JPN (or USA>EUR>JPN in SNES P/C DAT). If new format is finally available user could choose the specific order for those 3 or (if we go ahead and include labels for all possible regions in a given DAT) for a list of 10+ regions. The main point here is that even the latter is feasible and error free.

You misunderstood me. It’s pointless to limit ourselves to only use the major and not all regions. In the Harry Potter example people have only the choice between USA and JPN. Why not give them the choice for all regions?

Just to be sure, please note that family names will be used only to name the archives (.zip/.7z) resulting from fully merging the collection. Inside these archives there will be all/available ROMs in each family and they will retain their current naming. If not fully-merging, current names will be used for archives.

Yes, we are only talking about set titles. ROM titles won’t change.

What you want is too "personal tastes oriented".

Not really. Myself I would prefer a much simpler approach like virtual parents, i.e. only list one parent title.

We have a convention to name the ROMs.

This has nothing to do with the convention.

I'm not saying what you want is impossible, but very unrealistic to accomplish to the same degree of quality (and timeframe) as in the intended implementation that I've described.

Huh? Why is it unrealistic? We just need to add like another table to Dat-o-matic and recheck all titles. Not really a big deal.

Yes, it's possible. But as I said above, family names are just alternate names for the archive containing all the ROMs in the family. I think it makes more sense to have them nicely grouped below the family reference game.

Good. What happens if the same family name is defined in the parent and clone set?

Like this:

Code: Select all

 <game name="Super Mario World (USA)">
	<family region="en">Super Mario World</family>
	<description>Super Mario World (USA)</description>
	<rom name="Super Mario World (USA).rom" ... />
</game>
<game name="Super Mario World (Europe)" cloneof="Super Mario World (USA)">
	<family region="en">Super Mario World</family>
	<description>Super Mario World (Europe)</description>
	<rom name="Super Mario World (Europe).rom" ... />
</game>

And what happens if the same family region is defined twice, but with different data (typo, etc)?

Like this:

Code: Select all

<game name="Super Mario World (USA)">
	<family region="en">Super Mario World</family>
	<description>Super Mario World (USA)</description>
	<rom name="Super Mario World (USA).rom" ... />
</game>
<game name="Super Mario World (Europe)" cloneof="Super Mario World (USA)">
	<family region="en">Super Mario Wrold</family>
	<description>Super Mario World (Europe)</description>
	<rom name="Super Mario World (Europe).rom" ... />
</game>

===============================================

Re: Future of datafiles \ romcenter on 7th April 2008, 08:51 wrote:

And what happens if the same family region is defined twice, but with different data (typo, etc)?

The first one is used, the second is dropped and a message is shown in an error log.

===============================================

Re: Future of datafiles \ Logiqx on 7th April 2008, 12:05 wrote:

Hello,

Having read the above debate, can I add my own thoughts to this thread? Let me just confirm the requirement here:

From what I can see, the basic requirement is to allow users to store all regions of a game (referred to as a family) within a single archive (i.e. what we already refer to as full merging). However, we would like to improve on current capabilities and allow them to state their personal preference for the naming of ZIPs without the need for multiple data files. For example, an English user might want to see their ZIPs named according to the European titles whereas a Japanese user might want to see their ZIPs named according to the Japanese titles (where available).

Taking some of ideas from above and making some minor tweaks I would suggest the following:

1) Use the existing cloneof functionality for family groupings.
2) Remove dummy parents and simply use the EUR or USA regions as the parent. It doesn't actually matter which region is chosen as parent because of how this idea works. However, if a ROM manager doesn't support this idea then the user will just be stuck with the dat creators choice of parent so a standard still makes sense.
3) Use a "<archive>" tag to specify the archive name (minus the ".zip", ".rar", ".7z", etc). This removes the need for dummy parents. n.b. This tag is already available in listinfo, listxml and generic xml within my tools (it has been for years)!
4) Specify the region of each clone in the datafile using an attribute of the game. This will allow the ROM manager to identify the "preferred clone" (i.e. which later leads to the preferred ZIP name) for each family group, based on the user's region preferences.

For example:

Code: Select all

<game name="Golden Sun (USA, Europe)" region="USA">
<archive>Golden Sun</archive>
</game>

<game name="Golden Sun (France)" cloneof="Golden Sun (USA, Europe)" region="FRA">
</game>

<game name="Golden Sun (Germany)" cloneof="Golden Sun (USA, Europe)" region="GER">
</game>

<game name="Golden Sun (Italy)" cloneof="Golden Sun (USA, Europe)" region="ITA">
</game>

<game name="Golden Sun (Spain)" cloneof="Golden Sun (USA, Europe)" region="ESP">
</game>

<game name="Ougon no Taiyou - Hirakareshi Fuuin (Japan)" cloneof="Golden Sun (USA, Europe)" region="JPN">
<archive>Ougon no Taiyou - Hirakareshi Fuuin</archive>
</game>

Putting aside how the user specifies their region preferences (a GUI design consideration), this will result in all of the Golden Sun family being in one archive, which may have one of two names:

a) Golden Sun.zip
b) Ougon no Taiyou - Hirakareshi Fuuin.zip

At a risk of going in to too much detail, the logic for identifying ZIP names is as follows:

Code: Select all

1) for each family group (i.e. games related via cloneof)
	1.1) identify the "preferred clone" based on region preferences.
	     if a preferred region is not identified then use the actual parent.
	1.2) determine the archive name of the "preferred clone" using the following evaluation order:
		1.1.1) look if an archive name is specified in the "preferred clone"
		1.1.2) look if an archive name is specified in the actual parent
		1.1.3) take the game description of the preferred clone
	1.3) for each game in the family group
		1.3.1) determine the archive name based on the user's split/merge preferences:
		       1.3.1.1) if using split sets, use the "archive" name in preference to the description.
		       1.3.1.2) if using fully merged sets, use the archive name that was determined in 1.2.

This approach should be simple for datafile creators to manage as there are only two changes:

1) A new "region" attribute.
2) Use of the existing "archive" element (listinfo, listxml, generic xml).

For the most part, data file authors need only specify "archive" attribute for the parent and regional exceptions, rather than for every region:

As a user, I would just need to specify my regional preferences as EUR, USA, JPN.

Once agreed, valid regions can be specified in the DTD and validated by DatUtil.

===============================================

Re: Future of datafiles \ kazumi213 on 7th April 2008, 17:08 wrote:

@NGEfreak

Sorry, but IMHO yes, finding and verifying all official title translations for all games would be a BIG deal. But this isn't really the question right now. What matters is that the suggested format so far would allow to suit your needs.

Regarding your last couple of questions. The first case is the same as declaring a super family labeled "en". Acording to Eric, both ROMs would be included in the archive "Super Mario World". However this declaration is redundant if the ROMs are linked by "cloneof" declarations.

The second case, according to the above, would lead to the creation of 2 archives: "Super Mario World" and "Super Mario Wrold", each containing 1 ROM.

To avoid both mistakes in DAT, it is better to group all "family region" declarations below the reference game, as I suggested. This is less prone to errors. Nevertheless both cases could be reported when parsing the DAT on initial load (like CM Profiler does) or by performing a DatUtil pass when the datter is ready to release (always a good idea). It would be a matter of serching for "family region" declarations *within families* whose labels match but the titles don't, and reporting the found conflicts. Checks like this are currently present in DatUtil.

3) Use a "<archive>" tag to specify the archive name (minus the ".zip", ".rar", ".7z", etc). This removes the need for dummy parents. n.b. This tag is already available in listinfo, listxml and generic xml within my tools (it has been for years)!

First of all thanks for your interest on this question.

This "archive" thing is interesting because of its inmediate advantage on currently offered Parent-Clone DATs for CM here at No-Intro (to instantly eliminating the need of fake parents).

I assume that *currently* the filename declared with "archive" can appear only under parent entries and CM uses it only when fully merging. Otherwise the DAT could not be used in Unmerged mode (those entries with "archive" declarations would be named according to "archive" and not "game name" which is the proper in Unmerged mode)

Now regarding your proposal: it perfectly fits the originally requested features on this thread. However it also is strictly limited to them i.e: super families or NGEfreak's multi-titles would not be possible.

Please, don't get me wrong, I like it and is enough for me, but offers less at about the same cost as Eric's proposal. I agree "archive" atribute is already there, but in the end the way it is parsed has to be redefined along with the inclusion of "region" atribute, so it's essentially the same as including parsing-ability to Eric's "family region" declarations.

Also, I don't like the idea of "fixing" the allowed "region" labels. If possible, Eric's aproach of picking them when parsing the DAT and then offering to the user on a per-DAT basis sounds better to me. This allows for more flexibility when datting.

Code: Select all

1) for each family group (i.e. games related via cloneof)
	1.1) identify the "preferred clone" based on region preferences.
	     if a preferred region is not identified then use the actual parent.
	1.2) determine the archive name of the "preferred clone" using the following evaluation order:
		1.1.1) look if an archive name is specified in the "preferred clone"
		1.1.2) look if an archive name is specified in the actual parent
		1.1.3) take the game description of the preferred clone <======================
	1.3) for each game in the family group
		1.3.1) determine the archive name based on the user's split/merge preferences:
		       1.3.1.1) if using split sets, use the "archive" name in preference to the description.
		       1.3.1.2) if using fully merged sets, use the archive name that was determined in 1.2.

Thanks for this. Really interesting. I've highlighted a couple of logical steps. I think they should follow the "game name" declaration, not the "description" one. We use the description field to include tags which allow for some special searching features (not intended for file naming).

Also, I'm not sure, but according to that scheme it seems like the "archive" name for the parent is used when the prefered one is not found. But the expected behaviour is that the user's next prefered one is used, which could not necessarily be the parent chosen by the datter.

===============================================

Re: Future of datafiles \ Logiqx on 7th April 2008, 21:33 wrote:

CMPro doesn't understand the archive attribute currently but it is supported by my tools because Shark and RAINE include it in their listinfo. It shows the various archives that the emulator will also search when looking for ROMs but could be used for other purposes instead (since no ROM manager makes use of it yet, hence my suggestion).

Also, I'm not sure, but according to that scheme it seems like the "archive" name for the parent is used when the prefered one is not found. But the expected behaviour is that the user's next prefered one is used, which could not necessarily be the parent chosen by the datter.

Step 1.1 would iterate through the preferred regions until it finds one of them (or if it doesn't, adopt the parent).

One of the things that I may not have made clear about my suggestion is that initially you needn't use the "archive" element at all. Just by filling in region alone (which needn't be restricted) you would get ZIPs resembling the names that you want (just with an unwanted country name at the end). This I believe is useful because it can be done with very little effort on the part of data file creators. It is also a minor change to my tools (one extra attribute) and will work in several data file formats (cmpro, listxml, generic xml and tab delimited for database loads) whereas the group system is something that I probably won't have time to do.

I'm also not convinced that it is a good idea to link games on textual names that will be repeated in several places and could contain typos (which is relied upon by the group system). It feels like going back to the trouble with RomCenter data files where a typo can break parent/clone relationships without being easy to spot. You could have some users with 102 ZIPs and others with 98 when they should both have 100.

Personally I think that that the group syntax is too prone to error and could lead to some very odd behavior.

===============================================

Re: Future of datafiles \ romcenter on 8th April 2008, 06:29 wrote:

The idea to use archive tag is a good idea I think. (I didn't know about this tag..)

And adding a 'region' tag is also very good. This tag can be used for other purpose like country filtering etc...

===============================================

Re: Future of datafiles \ NGEfreak on 8th April 2008, 07:55 wrote:

Thinking about it, regions are not the only factor to determine which family name should be used.

For example we have this case:

Arcus Odyssey (USA) (Proto)
Arcus Spirits (Japan)

We have two Users, both are using this region priority list: USA>JPN
User A wants to use this list no matter if the roms are finals or betas. However User B prefers final titles over beta titles and thus wants to name his set Arcus Spirits. This means aside of regions we need to add another criterion: release type/state.

Another case:

Super Puyo Puyo Tsuu (Japan)
Super Puyo Puyo Tsuu Remix (Japan)

Re-releases may have different titles. User A prefers the first released version, but User B prefers the latest version. So, we need to add yet another criterion: release date.

There are probably even more criteria. Maybe a more simple approach is better after all?

Post by **root** » 08 Apr 2008 07:35

Rif: 2080-3 Re: Future of datafiles \ Logiqx on 8th April 2008, 08:35 wrote:

Thanks. Yes, region filtering was something that I also thought would be useful and hence my suggestion for the region tag which can be used for multiple purposes (filtering and organising). Whether or not official translations are supported in the future (which seems a bit ott to me), a region tag makes perfect sense and there is no doubting where it belongs in the XML specification.

This one tag will allow for region filtering and will also allow users to store families within a ZIP named after their preferred regions. Personally I think that it is a big step forwards and has many immediate benefits to users with very little effort on the part of you, myself or datafile creators.

For example, users may have very different preferences:

"I only want the EUR version of each family but if that does not exist, the USA one, otherwise nothing".
"I want all versions of each game in a family but the ZIP named after the Japanese name, otherwise the EUR name, otherwise the default parent".
"I want all games in Japanese, nothing else".

For example, if we do have a region tag and everyone uses region standard names (which eventually will be a finite list and can be in the DTD) then users won't have to specify their regional preferences for every datafile that they load. RomCenter will be able to remember their preferences. I don't necessarily think that allowing every data file in the world to name regions differently is a good thing. It shouldn't be forgotten that my region tag is an attribute and means that it is for the benefit of software, it is not intended as something that is routinely displayed to humans.

What I hadn't mentioned in my original post was that I thought there could be a region AND language specified for each game. The two together allow RomCenter to provide users with lots of exciting new functionality (bearing in mind that the datafile changes are so simple - addition of region and language to games). The examples above are just the tip of the iceberg.

My suggestion would be to implement region codes and languages first, shortly followed by archive names and get them 100% correctly (n.b. there will be a fair bit of development effort just to get RomCenter doing that). At that point, niceties such as official translations and supergroups can be considered. I'm not convinced anyone will have the patience to do them.

At this point, I plan to add region and language attributes to the DTD anyway as they make a lot of sense.

Automerged Doublepost.

@NGEfreak

Nice thought. We could add a date attribute to games also:

<game name="SOMETHING" region="EUR" language="ENG" date="20080408">

Simplicity is the best policy. My favorite quote from a fairly bright guy named Albert:

"Make everything as simple as possible, but not simpler."

===============================================

Re: Future of datafiles \ NGEfreak on 8th April 2008, 11:44 wrote:

Easier said than done. Please note that a game (=ROM) can have multiple languages, multiple countries and multiple release dates.

For example the NTSC version of Super Metroid has
- Languages: English, Japanese
- Countries: USA, Japan (probably a few more like Canada)
- Release dates: 19994.03.19 (Japan), 1994.04.18 (USA)

If you are going to add a list of all regions/languages to the DTD, at least use ISO codes (+ some non-standard extra codes like EU, II (International/World), etc). This way you can be sure that you got all possibilities.
Maybe the ISO country/language codes are already built-in into XML DTD?

===============================================

Re: Future of datafiles \ kazumi213 on 8th April 2008, 15:39 wrote:

Ok, I think this discussion is starting to deviate. I've asked for 2 specific, well defined, reasonable and feasible features:

- Ability to use family archive filenames different to the actual "game names".
- User ability to choose which family archive filename to use according to a region preference order.

This was provided by Eric's "family name" + "region" combo or "family region" modification. It is also provided by Logiqx's "region" + "archive" combo.

*As a bonus*, Eric's proposal would allow for "super families" and "NGEfreak's multi-titles". Both are NOT requested features, but they would be nice to have. Could they cause troubles? Yes, but just for a mediocre datter.

There is no inmediate need for all other features discussed on last posts. No-Intro uses country and language tags along with titles on their DATs, and there's the search function on ROM managers. More tags can be added on the "description" field if required to cover specific needs (I do on my Parent-Clone DATs). I mean, please don't let this country and language features delay the MAIN 2 above.

Thinking about it, regions are not the only factor to determine which family name should be used.

For example we have this case:

Arcus Odyssey (USA) (Proto)
Arcus Spirits (Japan)

We have two Users, both are using this region priority list: USA>JPN
User A wants to use this list no matter if the roms are finals or betas. However User B prefers final titles over beta titles and thus wants to name his set Arcus Spirits. This means aside of regions we need to add another criterion: release type/state.

Come on NGEfreak, if the user prefers USA names, USA names will be used, period. If using a (Proto) title to name a family is a problem for you, it's your problem. But the fact is that "Arcus Odyssey" is a legit USA name.

Another case:

Super Puyo Puyo Tsuu (Japan)
Super Puyo Puyo Tsuu Remix (Japan)

Re-releases may have different titles. User A prefers the first released version, but User B prefers the latest version. So, we need to add yet another criterion: release date.

I see no problem here. As a datter I would create the JPN family name based on "Super Puyo Puyo Tsuu" only. It is the parent by definition.

No reason to wait for the implementation of a cumbersome format just to satisfy each and every little user needs out there.

There are probably even more criteria. Maybe a more simple approach is better after all?

@Logiqx: Is it possible to use <archive> to implement something like this?

Yes, if it is parsed as I've described on my previous post.

===============================================

Re: Future of datafiles \ Logiqx on 8th April 2008, 16:50 wrote:

@kazumi213

Sorry for this long post.

I have just read your post about deviating but since I took the trouble to type it up prior to posting, I may as well still post it!

However, my point still stands that linking games using #PCDATA from a <family> element is not the best of designs (sorry Eric).

@NGEfreak

Thanks for your comments. I agree that ISO codes make sense for regions and languages.

You make the point that for some games there is no single region, language or release date. The following works around the limitation of one region, language and release date but would it be sufficient? It is almost going back to some of the earlier XML ideas but the subtle difference is that it's not for linking (which could lead to errors or unexpected behavior). It is for filtering and identifying preferred versions of games (which in turn leads to ZIP names):

Code: Select all

<game name="Super Metroid (NTSC)">
	<release region="us" language="en" date="1994-04-18">Super Metroid</release>
	<release region="jp" language="ja" date="1994-03-19">Japanese Name</release>
	... existing game info ...
</game>

Many games will offer a choice of languages (selected by the player from a configuration menu) but would dat creators want to list all supported languages within one region? If so, they could be handled by having multiple release lines for a region.

<release region="ca" language="en" date="1994-04-18"/>
<release region="ca" language="fr" date="1994-04-18"/>

The introduction of this <release> tag still allows for all of the clever filtering and identification of preferred names based on region, language and date. Remember that all of the release information is optional though so it doesn't place a large burden on the datafile maintainer, unless they choose to provide all of the regional/language details.

Another note is that <release> elements could replace the need for using the <archive> element as I suggested earlier. In the Super Metroid example above, people with a US preference would end up with the NTSC and PAL version in a zip called "Super Metroid.zip" whilst Japanese users would end up with the same ZIP contents but a Japanese ZIP filename. By using cloneof for linking (as opposed to the family concepts), it wouldn't even matter if there is a typo of "Supre Mterodi" in other versions because cloneof still would group them and the ZIP name derived from the preferred release. Of course, a ZIP name derived from the typo would still be wrong but you wouldn't end up with some ROMs in "Super Metroid.zip" and some in "Supre Mterodi.zip" (they'd all be in one or the other, depending on your region/language/date preferences).

The datafile creator could be as detailed or lazy as they feel like for each game.

Code: Select all

For example:

	<release region="eu" language="en" date="1994-04-18">English name</release>
	<release region="eu" language="fr" date="1994-04-18">French name</release>
	<release region="eu" language="de" date="1994-04-18">German name</release>

or

	<release language="en">English name</release>
	<release language="fr">French name</release>
	<release language="de">German name</release>

or

	<release region="eu">Game name</release>

or

	<release>Game name</release>

or

	<release region="eu"/>

Hopefully this is a happy balance between simplicity and flexibility. Even the lazy examples above would still support filtering and grouping (based on user preferences). The last example provides a means for naming the ZIPs (when used within the default parent) and the first example would also offer the benefit of official translations as well (although under those circumstances, the user may struggle to identify the appropriate ROM within the ZIP).

This proposal doesn't support super-groups but they could be defined in other ways. I think that keeping release information is useful and should be kept seperate from advanced group hierarchies.

As far as the inital requirements from kazumi213 go, each game only needs a simple <release>Name</release> in the parent and <release region="gb"/> in each clone (sometimes a <release region="jp">Japanese Name</release>). The new tags offer a lot of flexibility for future features but of course, they need not be implemented in RomCenter straight away. I understand that your original two requirements are why this thread started.

Automerged Doublepost.

@kazumi213

Would you be content with using my proposed syntax to meet your two requirements as follows?

Code: Select all

<game name="Golden Sun (USA, Europe)">
    <release>Golden Sun</release>
    <release region="us"/>
</game>

<game name="Golden Sun (France)" cloneof="Golden Sun (USA, Europe)">
    <release region="fr"/>
</game>

<game name="Golden Sun (Germany)" cloneof="Golden Sun (USA, Europe)">
    <release region="de"/>
</game>

<game name="Golden Sun (Italy)" cloneof="Golden Sun (USA, Europe)">
    <release region="it"/>
</game>

<game name="Golden Sun (Spain)" cloneof="Golden Sun (USA, Europe)">
    <release region="es"/>
</game>

<game name="Ougon no Taiyou - Hirakareshi Fuuin (Japan)" cloneof="Golden Sun (USA, Europe)">
    <release region="jp">Ougon no Taiyou - Hirakareshi Fuuin</release>
</game>

Automerged Doublepost.

@NGEfreak

Would you be content with using my proposed syntax to support official translations as follows?

Code: Select all

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<release>Harry Potter and the Sorcerer's Stone</release>
	<release region="us"/>
	<release region="eu" language="en">Harry Potter and the Philosopher's Stone</release>
	<release region="eu" language="de">Harry Potter und der Stein der Weisen</release>
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<release region="jp">Harry Potter to Kenja no Ishi</release>
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

I think that this syntax meets both of your needs. Both of the above examples make use of a generic "<release>" element for the default ZIP name (which in it's absense would be the game name).

===============================================

Re: Future of datafiles \ kazumi213 on 9th April 2008, 01:46 wrote:

No need to apologize for posting on this thread, whatever the size of your post. This thread is not mine and I didn't start it. I just took the opportunity to ask Eric for him to support this improved ROM manager behaviour when fully merging.

A simple solution for a simple concept was proposed, but then proposals for the inclusion of some unrelated database features and support for complex end-user preferences were added into the mix, and it started to look like all those new troubles were required to be solved (with a complex format) for the initial simple concept to be possible. I was just trying to note about this fact. Sorry if my post sounded rude. It wasn't my intention.

Of course I like your new proposal. It allows for such a detailed DAT like in NGEfreak's multi-title approach, but it also allows for a simple DAT structure which works as I originally requested, so kudos Logiqx.

I have a question. Could it cause trouble if when datting I decide to simplify the layout as follows?

<game name="Golden Sun (USA, Europe)">
<release>Golden Sun</release>
<release region="us"/>
<release region="us">Golden Sun</release>
</game>
<game name="Golden Sun (France)" cloneof="Golden Sun (USA, Europe)">
<release region="fr"/>
</game>
<game name="Golden Sun (Germany)" cloneof="Golden Sun (USA, Europe)">
<release region="de"/>
</game>
<game name="Golden Sun (Italy)" cloneof="Golden Sun (USA, Europe)">
<release region="it"/>
</game>
<game name="Golden Sun (Spain)" cloneof="Golden Sun (USA, Europe)">
<release region="es"/>
</game>
<game name="Ougon no Taiyou - Hirakareshi Fuuin (Japan)" cloneof="Golden Sun (USA, Europe)">
<release region="jp">Ougon no Taiyou - Hirakareshi Fuuin</release>
</game>

===============================================

Re: Future of datafiles \ Logiqx on 9th April 2008, 07:22 wrote:

Nice spot and I think it demonstrates how this approach can be as simple or complex as the datafile creator likes. Yes, in your example it would work because the European games are so similar (all called "Golden Sun"). Unless the user has a Japanese preference then they will just need a ZIP called "Golden Sun.zip".

The only amendment that I need to make is in relation to picking up the default name:

Code: Select all

<game name="Golden Sun (USA, Europe)">
    <release region="us" default="yes">Golden Sun</release>
</game>

In the Golden Sun example it may not be clear why but in games with multiple titles (like Harry Potter), a default name should be explicitly stated (rather than it being undefined or a fixed approach within the code; first or last).

Your example again in its complete + simplified form:

Code: Select all

<game name="Golden Sun (USA, Europe)">
    <release region="us" default="yes">Golden Sun</release>
</game>

<game name="Golden Sun (France)" cloneof="Golden Sun (USA, Europe)">
</game>

<game name="Golden Sun (Germany)" cloneof="Golden Sun (USA, Europe)">
</game>

<game name="Golden Sun (Italy)" cloneof="Golden Sun (USA, Europe)">
</game>

<game name="Golden Sun (Spain)" cloneof="Golden Sun (USA, Europe)">
</game>

<game name="Ougon no Taiyou - Hirakareshi Fuuin (Japan)" cloneof="Golden Sun (USA, Europe)">
    <release region="jp">Ougon no Taiyou - Hirakareshi Fuuin</release>
</game>

In the above example, the ZIP name is identified based on the region being "jp" or "us". If neither of those are specified by the user (let's not assume that they must maintain a full list of preferred codes) then the default parent is used. In order to get the ZIP name (rather than using Golden Sun (USA, Europe).zip") the default entry is used.

Now the Harry Potter example again:

Code: Select all

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<release region="us" default="yes">Harry Potter and the Sorcerer's Stone</release>
	<release region="eu" language="en">Harry Potter and the Philosopher's Stone</release>
	<release region="eu" language="de">Harry Potter und der Stein der Weisen</release>
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<release region="jp">Harry Potter to Kenja no Ishi</release>
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

The same program logic can be applied. Users specifying a "jp" preference over "us" and "eu" will end up with "Harry Potter to Kenja no Ishi.zip" whereas anyone else will end up with one of the ZIP names specified in "Harry Potter and the Sorcerer's Stone (USA, Europe)". Based on their region and language preferences, one of the 3 names may be chosen but if none are suitable, the default is used (i.e. the "us" name).

If the datafile creator wants other "eu" regional users to get the English name (let's say the Spanish), then he can do the following:

Code: Select all

<game name="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<release region="us" default="yes">Harry Potter and the Sorcerer's Stone</release>
	<release region="eu">Harry Potter and the Philosopher's Stone</release>
	<release region="eu" language="en">Harry Potter and the Philosopher's Stone</release>
	<release region="eu" language="de">Harry Potter und der Stein der Weisen</release>
	<description>Harry Potter and the Sorcerer's Stone (USA, Europe)</description>
	<rom name="Harry Potter and the Sorcerer's Stone (USA, Europe).rom" ... />
</game>
<game name="Harry Potter to Kenja no Ishi (Japan)" cloneof="Harry Potter and the Sorcerer's Stone (USA, Europe)">
	<release region="jp">Harry Potter to Kenja no Ishi</release>
	<description>Harry Potter to Kenja no Ishi (Japan)</description>
	<rom name="Harry Potter to Kenja no Ishi (Japan).rom" ... />
</game>

It should be clear that the release details are only applicable if all of the attributes match a user preference (order of priority: region+language, region, language). The only exception is when the "default" entry of the parent game is being used, after failing to find a preferred region and/or language. The only thing for a datafile creator to avoid is stating a "default" twice within the parent. DatUtil could warn about that though.

Hopefully these examples are enough to get the thumbs up from both of you.

I just had another thought about this. What would happen if RomCenter named all of the ZIPs according to the user preferences and you then scanned all of those ROMs with another manager (let's say CMPro)? Actually, it's not too bad at all! It will simply say that the ZIP names are wrong because the contents will still be recognisable (both the ROM names and CRCs will match exactly).

===============================================

Re: Future of datafiles \ NGEfreak on 9th April 2008, 08:12 wrote:

@Logiqx

Looks great. This syntax addresses all issues so far.

@kazumi213

No-Intro is not the only group that uses RC. The talk of this feature shouldn't be limited to our needs only.

===============================================

Re: Future of datafiles \ romcenter on 9th April 2008, 10:10 wrote:

great

I'm actually trying to know how I will manage that in rc

===============================================

Re: Future of datafiles \ Logiqx on 9th April 2008, 10:53 wrote:

Great, it looks like we are close to getting something that we are all happy with.

Eric, I'll discuss how it might be implemented (privately by e-mail). It's actually not too hard to do. It's almost a light relief from the solutions that I have to design and build at work.

I will do a draft of the DTD and we'll take it from there.

Automerged Doublepost.

Ok, a draft of the DTD:

Code: Select all

<!--
   ROM Management Datafile - DTD

   For further information, see: http://www.logiqx.com/

   This DTD module is identified by the PUBLIC and SYSTEM identifiers:

   PUBLIC "-//Logiqx//DTD ROM Management Datafile//EN"
   SYSTEM "http://www.logiqx.com/Dats/datafile.dtd"

   $Revision: 1.3 $
   $Date: 2008/04/09 11:45:00 $

-->

<!ELEMENT datafile (header?, game+)>
	<!ATTLIST datafile build CDATA #IMPLIED>
	<!ATTLIST datafile debug (yes|no) "no">
	<!ELEMENT header (name, description, category?, version, date?, author, email?, homepage?, url?, comment?, clrmamepro?, romcenter?)>
		<!ELEMENT name (#PCDATA)>
		<!ELEMENT description (#PCDATA)>
		<!ELEMENT category (#PCDATA)>
		<!ELEMENT version (#PCDATA)>
		<!ELEMENT date (#PCDATA)>
		<!ELEMENT author (#PCDATA)>
		<!ELEMENT email (#PCDATA)>
		<!ELEMENT homepage (#PCDATA)>
		<!ELEMENT url (#PCDATA)>
		<!ELEMENT comment (#PCDATA)>
		<!ELEMENT clrmamepro EMPTY>
			<!ATTLIST clrmamepro header CDATA #IMPLIED>
			<!ATTLIST clrmamepro forcemerging (none|split|full) "split">
			<!ATTLIST clrmamepro forcenodump (obsolete|required|ignore) "obsolete">
			<!ATTLIST clrmamepro forcepacking (zip|unzip) "zip">
		<!ELEMENT romcenter EMPTY>
			<!ATTLIST romcenter plugin CDATA #IMPLIED>
			<!ATTLIST romcenter rommode (merged|split|unmerged) "split">
			<!ATTLIST romcenter biosmode (merged|split|unmerged) "split">
			<!ATTLIST romcenter samplemode (merged|unmerged) "merged">
			<!ATTLIST romcenter lockrommode (yes|no) "no">
			<!ATTLIST romcenter lockbiosmode (yes|no) "no">
			<!ATTLIST romcenter locksamplemode (yes|no) "no">
	<!ELEMENT game (comment*, release*, description, year?, manufacturer?, biosset*, rom*, disk*, sample*, archive*)>
		<!ATTLIST game name CDATA #REQUIRED>
		<!ATTLIST game sourcefile CDATA #IMPLIED>
		<!ATTLIST game isbios (yes|no) "no">
		<!ATTLIST game cloneof CDATA #IMPLIED>
		<!ATTLIST game romof CDATA #IMPLIED>
		<!ATTLIST game sampleof CDATA #IMPLIED>
		<!ATTLIST game board CDATA #IMPLIED>
		<!ATTLIST game rebuildto CDATA #IMPLIED>
		<!ELEMENT release (#PCDATA)>
			<!ATTLIST release region CDATA #REQUIRED>
			<!ATTLIST release language CDATA #IMPLIED>
			<!ATTLIST release date CDATA #IMPLIED>
			<!ATTLIST release default (yes|no) "no">	
		<!ELEMENT year (#PCDATA)>
		<!ELEMENT manufacturer (#PCDATA)>
		<!ELEMENT biosset EMPTY>
			<!ATTLIST biosset name CDATA #REQUIRED>
			<!ATTLIST biosset description CDATA #REQUIRED>
			<!ATTLIST biosset default (yes|no) "no">
		<!ELEMENT rom EMPTY>
			<!ATTLIST rom name CDATA #REQUIRED>
			<!ATTLIST rom size CDATA #REQUIRED>
			<!ATTLIST rom crc CDATA #IMPLIED>
			<!ATTLIST rom sha1 CDATA #IMPLIED>
			<!ATTLIST rom md5 CDATA #IMPLIED>
			<!ATTLIST rom merge CDATA #IMPLIED>
			<!ATTLIST rom status (baddump|nodump|good) "good">
			<!ATTLIST rom date CDATA #IMPLIED>
		<!ELEMENT disk EMPTY>
			<!ATTLIST disk name CDATA #REQUIRED>
			<!ATTLIST disk sha1 CDATA #IMPLIED>
			<!ATTLIST disk md5 CDATA #IMPLIED>
			<!ATTLIST disk merge CDATA #IMPLIED>
			<!ATTLIST disk status (baddump|nodump|good) "good">
		<!ELEMENT sample EMPTY>
			<!ATTLIST sample name CDATA #REQUIRED>
		<!ELEMENT archive EMPTY>
			<!ATTLIST archive name CDATA #REQUIRED>

Once the changes have had some time to sink in to our heads, I'll finalise the DTD and add any required functionality to DatUtil.

I can't really do any testing (trying out some examples, including validation against the DTD) right now because I am at work so hopefully there are no typos in the DTD.

===============================================

Re: Future of datafiles \ romcenter on 9th April 2008, 13:04 wrote:

Thank you for the dtd. I'll think about it.

What is the tag 'archive' for ?

===============================================

Re: Future of datafiles \ Logiqx on 9th April 2008, 16:16 wrote:

I'll send you a PM about the "archive" tag because it is not related to these requirements anymore.

===============================================

Re: Future of datafiles \ tetsuo55 on 9th April 2008, 17:06 wrote:

My datting skills are not good enough to understand everything said in the above posts i am confused about one thing though.

These last examples look very good but it's not fully clear to me how i would be able to create a 1rom1game set based on my preferences of country/language (the possibilities have expanded greatly above what we currently do with the 1g1r parent-clone dat's thanks to the new fields)

===============================================

Re: Future of datafiles \ kazumi213 on 9th April 2008, 19:10 wrote:

Ok Logiqx, here's my thumb up!

Just one last question: could it be discarded when no multi-titles are used? I mean, the fact it is the parent could be used as condition for defaulting to it. Consider it as a "safety switch" in case the datter forgets to include the default="yes" declaration, as all region titles are default="no" in the DTD.

My datting skills are not good enough to understand everything said in the above posts i am confused about one thing though.

These last examples look very good but it's not fully clear to me how i would be able to create a 1rom1game set based on my preferences of country/language (the possibilities have expanded greatly above what we currently do with the 1g1r parent-clone dat's thanks to the new fields)

It would be possible and at zero cost for the datter. Just in case let me explain that a 1G1R set means 1 Game, 1 ROM set, i.e. only 1 ROM from each family is included in the set (so it is a set of unique games). It is currently offered on my Parent-Clone DATs by means of special tags (suggested by tetsuo55 ) on the description field.

Eric would just need to include a filtering feature which would work as follows:

- The user specifies the same region preferences as if he were going to fully merge (note languages would have nothing to do here as there can be multi-titles but not multi-ROMs for them).

- The filtering feature would determine the matching games (same iteration as when searching DAT entries to pick family titles before fully merging), but instead of picking the associated family titles it just would present to the user the list of matching entries ("game name" and "description")

- The user selects all games on screen and moves, copies, or simply creates a DAT/text list from them. *No* need to physically have the ROMs for just displaying them or DAT/text exporting is assumed.

The above feature could also be used to display the family titles which would be used when fully merging, like a preview of resulting merged collection names.

===============================================

Re: Future of datafiles \ Logiqx on 10th April 2008, 14:28 wrote:

Could it be discarded when no multi-titles are used? I mean, the fact it is the parent could be used as condition for defaulting to it. Consider it as a "safety switch" in case the datter forgets to include the default="yes" declaration, as all region titles are default="no" in the DTD.

Yes. The algorithm that I have come up with will handle that (a neat scoring system for the identification of preferred ZIP names / 1G1R).

I'll explain it when I have time to write it up - it's very tempting to do it now but I should be doing real work.

Automerged Doublepost.

Ok, here is some pseudo code that will choose the preferred game (in the case of 1g1r) and region name (for the ZIP) based on the user preferences.

It isn't restricted to any fixed number of regions or languages but I have only used a few to keep the example simple. It's uses a simple scoring system based on the following priorities:

1) region
2) language
3) default
4) original parent

I think it will probably hurt a few peoples heads but really, it is the simplest approach to take in order to handle really complex data files.

I do this kind of thing for a living...

Code: Select all

// Let's assume some user preferences (which would be maintained from a GUI)

preferences.regions = ['uk', 'eu', 'us']            // Indexed from 0 to 2
preferences.languages = ['en', 'de']                // Indexed from 0 to 1

numRegionPrefs = count(preferences.regions)         // i.e. 3
numLanguagePrefs = count(preferences.languages)     // i.e. 2

// Iterate through games in the datafile, looking for parents (i.e. families)
for game in datafile.games
{
    // Decide if the game is a parent - if it doesn't specify "cloneof" it is a parent!
    if game.cloneof is null
    {
        // Retain the parent details initially - better details will overwrite them soon!
        preferredGame = parent
        preferredName = parent.name
        bestScore = 0

        // Iterate through all of the games in the family (i.e. the parent + clones)
        for clone in parent + parent.clones
        {
            // Iterate through all of the release entries
            for each release in clone.releases
            {
                // Start with a score of zero
                score = 0
            
                // Region is the most important attribute
                if release.region is not null
                {
                    // Check if it appears in the user preferences
                    for regionNo in 0 to numRegionPrefs - 1
                    {
                        if preferences.regions[regionNo] == release.region
                        {
                            // Increase the current score (uk=3, eu=2, us=1)
                            score += numRegionPrefs - regionNo;
                        }
                    }
                }

                // Score will now be 3 to 0 based on the following matches:
                // uk=3, eu=2, us=1, <none>=0

                // Shift the score left to make way for the language
                score *= (numLanguagePrefs + 1);

                // Score will now be 9 to 0 based on the following matches:
                // uk=9, eu=6, us=3, <none>=0

                // Language is the next most important attribute
                if release.language is not null
                {
                    // Check if it appears in the user preferences
                    for languageNo in 0 to numLanguagePrefs - 1
                    {
                        if preferences.languages[langauageNo] == release.language
                        {
                            // Increase the current score (en=2, de=1)
                            score += numLanguagePrefs - languageNo;
                        }
                    }
                }

                // Score will now be 11 to 0 based on the following matches:
                // uk+en=11, uk+de=10, uk+<none>=9
                // eu+en=8, eu+de=7, eu+<none>=6
                // us+en=5, us+de=4, us+<none>=3
                // <none>+en=2, <none>+de=1, <none>+<none>=0

                // Shift the score left to make way for the "default" release
                score *= 2;

                // Score will now be 22 to 0 based on the following matches:
                // uk+en=22, uk+de=20, uk+<none>=18
                // eu+en=16, eu+de=14, eu+<none>=12
                // us+en=10, us+de=8, us+<none>=6
                // <none>+en=4, <none>+de=2, <none>+<none>=0

                // The "default" release is the next most important consideration
                if release.default = "yes"
                {
                    // Increase the current score
                    score += 1;
                }

                // Score will now be 23 to 0 (the default will be an odd number).

                // Shift the score left to make way for parent
                score *= 2;

                // Score will now be 46 to 0 (the default will be a multiple of 2).

                // Original parent is the next most important attribute
                if close.cloneof is not null
                {
                    // Increase the current score
                    score += 1;
                }

                // Score will now be 47 to 0.

                // If the score beats the previous best then retain the details
                if score > bestScore
                {
                    preferredGame = clone

                    if release.name is not null
                    {
                        preferredName = release.name
                    }
                    else
                    {
                        preferredName = clone.name
                    }

                    bestScore = Score
                }
            }
        }

        // We now have the preferred game (1d1r) and preferred name (for the ZIP)
    }
}

===============================================

Re: Future of datafiles \ kazumi213 on 10th April 2008, 16:27 wrote:

Code: Select all

(snip)
        // Retain the parent details initially - better details will overwrite them soon!
        preferredGame = parent
        preferredName = parent.name   <====
        bestScore = 0
(snip)
                // If the score beats the previous best then retain the details
                if score > bestScore
                {
                    preferredGame = clone

                    if release.name is not null
                    {
                        preferredName = release.name
                    }
                    else   <====
                    {      <====
                        preferredName = clone.name   <====
                    }      <====

                    bestScore = Score
                }
            }
        }

        // We now have the preferred game (1d1r) and preferred name (for the ZIP)
    }
}

Simply perfect. I was just going to ask what would happen for "families" with only 1 member: the parent itself (like many Japan-only games).

According to that pseudo-code the whole "release region" declaration is not required for those parents (of course) and due to these declarations not being found the "game name" will be used instead.

Not sure if I had cleared this point so far, but this is exactly the expected behavior even when fully merging: names of archives containing only 1 ROM should be named according to "game name" (i. e. with tags) as such filename matches the contents of archive (as oppossed to family archives wich contain ROMs with different tags and must use "region name")

I've already said this several times, but in addition to it being a reasonable naming convention it has the advantage of allowing to quickly identify family archives (untagged) vs archives containing only 1 ROM when browsing your fully merged collection.

Post by **root** » 10 Apr 2008 16:42

Rif: 2080-4 Re: Future of datafiles \ Logiqx on 10th April 2008, 17:42 wrote:

Simply perfect.

Thanks. It's been a fun topic.

I'll make the DTD official soon and add the required enhancements to DatUtil.

Hopefully Eric enjoys translating my pseudo-code into Delphi.

===============================================

Re: Future of datafiles \ romcenter on 11th April 2008, 09:40 wrote:

Yep, I think I will try to do it directly into the database.
I will calculate the correct familyname for all games one for all and then use this field instead of filename...

Thank for the dtd. Do you thing a xml schema could offer a better xml validation than dtd ?

I could validate the xml with the schema and report errors before processing it.

===============================================

Re: Future of datafiles \ Logiqx on 11th April 2008, 13:22 wrote:

I'm not familiar with XML schemas.

The DTD will have to do for now.

===============================================

Re: Future of datafiles \ snakemeat on 11th April 2008, 22:46 wrote:

Thank for the dtd. Do you thing a xml schema could offer a better xml validation than dtd ?

Visual C++/C# 2008 Express http://www.microsoft.com/express/vc/ can generate the XSD. Open the DTD with it and choose generate XSD under the XML menu (that may not be exactly right, but close enough). It's the first free tool I found to do it (actually Trang http://www.thaiopensource.net/relaxng/trang.html can do it too, but is a tiny bit more tedious). XMLSpy and Stylus Studio can do it as well, but cost money or are limited trials. You may have to do a bit of manual checking, but for version 1.2 the output looked right.

You can then run xsd.exe http://msdn2.microsoft.com/en-us/library/x6c1kb0s.aspx to make a nice C# class out of the xsd, should you want to do that. For the java folk, I think Castor http://www.castor.org can help you with this part also.

Lastly, I finally found another simple free solution for generating sample XML from either the DTD or XSD. Using the Eclipse IDE http://www.eclipse.org/downloads/ one can generate a sample XML as described here: http://www.eclipsecon.com/webtools/comm ... zards.html I just wanted to add that because it is was hard for me to find a free solution for sample XML generation.

BTW thanks for all the work!

===============================================

Re: Future of datafiles \ Logiqx on 16th April 2008, 08:00 wrote:

I've added region support into DatUtil v2.38:

http://forum.logiqx.com/viewtopic.php?p=1096#1096

Whilst adding support to the various formats (listinfo, listxml, generic xml, tab delimited), it became apparent to me that the release name should be an attribute. This is consistent with the other filenames in datafiles (which are all attributes called "name"). The official DTD is therefore as follows:

Code: Select all

 
	<!ELEMENT release EMPTY>
		<!ATTLIST release name CDATA #REQUIRED>
		<!ATTLIST release region CDATA #REQUIRED>
		<!ATTLIST release language CDATA #IMPLIED>
		<!ATTLIST release date CDATA #IMPLIED>
		<!ATTLIST release default (yes|no) "no">

Name is also mandatory - again, for consistency with the existing elements.

If you are taking the "lazy" approach to releases and don't want to specify a name, just don't specify the release element for games that don't have any clones. The algorithm means that they'll be chosen anyway and their standard name used.

ttfn, Logiqx

===============================================

Re: Future of datafiles \ Bogy on 24th April 2008, 21:19 wrote:

ok...here we go...

clrmamepro 3.114 fully supports releases/regions/languages.

===============================================

Re: Future of datafiles \ kazumi213 on 25th April 2008, 14:18 wrote:

This is really great news. Thank you very much Roman.

===============================================

Re: Future of datafiles \ Logiqx on 25th April 2008, 18:57 wrote:

As I mentioned previously, I added a region syntax to the listinfo format and the new build of CMPro supports it too.

If it is makes your lives easier, you can add region info to your existing datafiles in their listinfo format and then make the change to XML at a later date.

However, XML has the advantage of being supported by CMPro and RomCenter 3.

I recently updated all of my datafiles to XML format.

===============================================

Re: Future of datafiles \ kazumi213 on 25th April 2008, 19:59 wrote:

Well, to be honest I'm not sure about the "region" syntax in listinfo format

so I'm working on a test GBA P/C DAT in generic XML format...

Could you show one of the recent examples (Golden Sun or Harry Potter) in listinfo format? I guess listinfo supports "region" but not the "language" declaration (so no multi-titles with listinfo), but not sure.

Finally do you know if CM 3.114 also included 1G1R filtering in some way or it is supporting only the fully merging usage of "region"? (I have not tested it yet, sorry)

===============================================

Re: Future of datafiles \ Logiqx on 25th April 2008, 20:51 wrote:

All of the XML region information is also possible in the listinfo format.

One way to find out the syntax is to run an XML datafile through DatUtil but here is a one-line example for you:

release ( name "Harry Potter und der Stein der Weisen" region eu language de )

CM 3.114 supports the merging concepts that we discussed. It doesn't support 1G1R yet because I forgot to mention to it Roman!

===============================================

Re: Future of datafiles \ kazumi213 on 26th April 2008, 00:00 wrote:

Understood

I'm sure Roman can add the 1G1R feature in no time since the selection algorithm is already there, so don't worry. Maybe like an additional (direct) option in the "Set Selection" group under the "Set-Information" window in Scanner. That options group is currently used to filter displayed sets, so it's a perfect fit.

EDIT:

I've finished some tests with a full GBA DAT supporting all countries/regions which have releases (11). So far the new format additions and CM implementation seem to merge as expected

When 1G1R list filtering/generation is implemented in CM/RC users will be able to display such list for France, Germany, Spain, Korea, etc. preference in addition to current Europe, USA or Japan.

===============================================

Re: Future of datafiles \ Logiqx on 29th April 2008, 22:15 wrote:

It's good to hear that it is working properly for you.

I've sent Roman an e-mail to explain what the 1G1R requirement is.

===============================================

Re: Future of datafiles \ Bogy on Yesterday, 22:25 wrote:

well...was on holiday for some time....then some long weekend...then my car's engine died (~4000 euro to get it fixed

) but tonight I had some time to implement the 1G1R mode.... already gave out a test version to Logiqx...time will tell...but I assume I can give out a final version this coming week....