GoodMerge XMDB No-Intro GBA Support

Post by **root** » 19 Sep 2006 16:02

Rif: 824-1 GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 19th September 2006, 17:02 wrote:

I began working on a XMDB for the No-Intro GBA set for GoodMerge support. You can see it started in the attached zip file.....

I tried it with the first 100 roms and it went from 300MB in 100 files to 175MB in 85 files. The more roms you have, the more space you'll save because similar games are put together. You can see that the space you'd save if you put all of the games together is huge....

What I need from you guys is help to complete the sets. If you know a lot of the games that are similar, go through the XML and put similar games together and upload a new version of the XMDB.

Note that you have to have a program like RomCenter to generate the Have.txt file needed for GoodMerge.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ xuom2 on 19th September 2006, 17:18 wrote:

you can use the parent-clone list by the great kazumi213

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 20th September 2006, 05:24 wrote:

you can use the parent-clone list by the great kazumi213

Perfect! I made a small program to parse through that and give out the XMDB file. Here's the completed set. Could someone please test it out and say what you think?

I tested it with a bunch of different setups and it worked. One interesting one was when the four "Cars" roms went from 17MB in 4 zip files to 3.4MB in 1 7zip file. Much easier to manage.....

If this proves to work, I'll do a bit more hacking on the program and release the source.

Note that you have to get the "have.txt" from a program like RomCenter.

I tried it out on 2,484 of the roms and it went from:

10.4 GB in 2,484 Zip files to....
5.75 GB in 1,504 7-Zip files....

I've attached the merge log.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 22nd September 2006, 14:48 wrote:

I've updated the XMDB to #2488 as well as cleaned up the source a bit....

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 22nd September 2006, 21:42 wrote:

Just noticed your new program to output the XMDB file: GBA nointro 2460

Very nicely done! Looks good to me.... I think development should shift to that instead of this one.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 24th September 2006, 08:31 wrote:

I certainly have an interest in this, as you may have gathered from my posts in the thread that xuom2 linked to.

I have been creating my xmdb file by hand, and will compare my results with yours once I have it completed.

Edit:
As I mentioned in the other thread, I don't know much about programing, but I may be able to at least help you with comparisons of your archive outputs and my own.

Automerged Doublepost

Since you seem to have more programming experience than I do, Rob, maybe you can give me a better idea of exactly how to read these lines from the xmdb file:

<flag reg=" (\(([JUEFGISNACK]{0,2}|Beta|MB(2GBA)?)\)|\[)"/>
<ignore reg=" \+ [0-9]{1,3} NES"/>

My understanding is that the first line uses a regular expression to determine a set of strings for GoodMerge to flag as version, or region, identifiers so that it can ignore them when attempting to determine the correct game name of a given input file. My testing seemed to confirm this when adding ACK into the group at the start succeded in removing separate archives for any (A) (C) or (K) games.

However, as of now there is apparently one (Ca) game in the collection, and that one is being separated from its clones because the (Ca) marker is not being recognized as a region tag. I'm wary of adding too many letters to that set because I don't want to end up with an expression that says any set of letters in parantheses should be considered a country code.

If you could provide some insight on how to read these lines, and what the second line's function is in regards to GM's operation, it should help resolve this for me.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 27th September 2006, 12:46 wrote:

Ahhh, I didn't see your post here. Anyway, yes. That RegEx says that it will be a string with 0 to 2 of those captical letters, or "Beta", or MB2, MBG, MBA. Although I'm not sure how GoodMerge handles them from there, that's what the regex says.

I change the Ignore regex to:

<ignore reg="[0-9]{4} \- "/>

so that it would ignore the first four numbers of the rom.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 27th September 2006, 14:33 wrote:

Ah, that makes sense, now that you've explained the ignore line. I will change it in my file, as well, for people who prefer to leave their ROMs numbered. It didn't affect me because my personal preference is for un-numbered naming.

The flag regex is actually the first error I noticed in your new tool's XMDB. Our GBA DAT includes several country codes that are not marked by the flag regex in the official GoodGBA file. To account for this, I've changed the flag line for my file from

<flag reg=" (\(([JUEFGISN]{0,2}|Beta|MB(2GBA)?)\)|\[)"/>

to

<flag reg=" (\(([JUEFGISNACKa]{0,3}|Beta|MB(2GBA)?)\)|\[)"/>

This allows GoodMerge to recognize the country tags of (A), (C), (K) and (Ca). Changing to {0,3} also allows for recognition of up to 3 letter country codes to be recognize. This is important because there is at least 1 3-letter combination in our set (a (UCa) ROM).

I suspect we can probably remove the "MB(2GBA)" part, but I haven't tested it yet, so I'm not sure. Also, does the "?" have a special meaning as part of a regex, or can it just be interpreted as a litteral part of the string?

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 30th September 2006, 07:23 wrote:

I implemented your fixes, but haven't tried them out. I also haven't updated to 2493 since the parent-clone list hasn't been updated.

The "?" in the Regex means that the previous element can either be there, or not be there. For example: "He?llo" could be either "Hello" or "Hllo"....

On a similar note, do you have any clue of what to do for the Kim Possible problem?

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 1st October 2006, 03:52 wrote:

2 more fixes:

<flag reg=" (\(([JUEFGISNACKaL]{0,2}|Beta|MB(2GBA)?)\)|\[)"/>
<ignore reg="^[0-9]{4} \- "/>

Note the additon of "L" to the flag regex (to account for a single NL game, grr) and of "^" to the ignore regex, so that any game w/ a 4-digit number in the title is still processed correctly.

On a similar note, do you have any clue of what to do for the Kim Possible problem?

Yes, but I don't like it. It makes the generated XMDB file much more complex, with a lot of redundant information that shouldn't be necessary, but it's the only way I can think of (so far, anyway) to make the naming messes.

If we remove the flag regex from the start of the file, then all ROM titles will be read in their entirety when GM is attempting to determine which archive they should be put in.

As near as I can tell, the problem stems from the fact that GM is designed to recognize country codes and (with the assistance of our revamped regex) ignore ROM numbers when parsing files for merging. So, what happens is it takes this file for input:

Code: Select all

1274 - Kim Possible (E) (M4) [b].gba

ignores the ROM number (in bold) and flags the country code and everything that follows (in itallics). It uses some of the first few lines from XMDB to confirm that .gba is a valid file extension for a GBA ROM. It then follows the same process with the input file "2061 - Kim Possible (J)."

In both cases it ends up with a "name" of "Kim Possible" for that ROM. It then searches through the ZONEDs looking for a matching name. When it finds one, it drops the ROM into that archive.

I can think of 2 possible solutions to this. First is the group regex that I have implemented for myself. When grouping GM will include parts of the filename that have been flagged for removal from the ROM (game) name. The ignore regex will not be overridden.

<group reg="^Kim Possible \(E\)"/>

This will include all file names that start with "Kim Possible (E)" into the current ZONED archive. Because we've stipulated with our ignore regex that "xxxx - " should be remove from the start of the line, ROM 1274 will fit in this group, but ROM 2061 will not. A second group regex ("^Kim Possible \(J\)") is then used to set a group for ROM 2061.

This solution works great, but is tough to implement in a tool designed to scan through a list of parent-clones and then output an XMDB based on that. The only way to truely implement this w/o setting up an elaborate check scheme to see if group registration is necessary is to default to creating a group regex for every single entry. This causes even more problems because if we aren't careful we could accidentally create a set of regex statements that are ambiguous.

A better option, IMO, is just to ditch the flag regex altogether. If we parse the entire file name (except the ROM number) when determining a ROM's name, then any possibility of duplication is eliminated. We can then use clone statements to assign the individual ROMs to their correct groups.

Code: Select all

	<zoned>
	  <bias zone="E" name="Kim Possible"/>
	    <clone name="Kim Possible (E) (M4) (b)"/>
	  <bias zone="U" name="Kim Possible - Revenge of Monkey Fist"/>
	    <clone name="Kim Possible - Revenge of Monkey Fist (U) (b)"/>
	</zoned>

If we forgo the flag regex it will eliminate any name duplication, but the clone names will have to be specified in full. We could skip the clone sections and just use the full file names in the bias section, but then we'd have archives called

Code: Select all

"Kim Possible (E) (M4) [b]"

or something, and that doesn't appeal to me.

That is the only thing I can think of to solve the issue without, as I said, trying to set complex sub routines that will check to see if any name duplication exists within the output XMDB before creating the file. It's not pretty, but it should work.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 2nd October 2006, 05:13 wrote:

Your changes are in and it resulted in the attached XMDB file in the zip.

The good news is that it looks like it matches exactly what you suggested, the unfortunate news is that I tested it out and it resulted in the following error:

Error: XML loading error.
The element 'zoned' has invalid child element 'bias'. List of possible elements expected: 'clone group'.

Any suggestions?.... Just noticed that I forgot some of the ignore regex, it's in, but still resulted in the same error.

Code: Select all

 <zoned>
  <bias name="Kim Possible" zone="E"/> <!-- 1274 -->
    <clone name="Kim Possible (E) (M4) [b]"/>
  <bias name="Kim Possible - Revenge of Monkey Fist" zone="U"/> <!-- 0772 -->
    <clone name="Kim Possible - Revenge of Monkey Fist (U) [b]"/>
  </zoned>

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 3rd October 2006, 13:41 wrote:

Hmm. Not sure. Just out of curiosity, was it parsing properly before? Once possible conflict arises from the fact that kazumi has listed the dual/tri-game paks multiple times in her parent-clone list; so we get this in our XMDB:

Code: Select all

      <zoned>
        <bias name="Rayman Advance" zone="E"/> <!-- 0051 -->
          <clone name="Rayman Advance (E) (M5)"/>
        <bias name="Rayman Advance" zone="U"/> <!-- 0046 -->
          <clone name="Rayman Advance (U) (M5)"/>
        <bias name="Rayman 10th Anniversary - Rayman Advance + Rayman 3" zone="E"/> <!-- 2121 -->
          <clone name="Rayman 10th Anniversary - Rayman Advance + Rayman 3 (E) (M5+M10)"/>
        <bias name="Rayman 10th Anniversary - Rayman Advance + Rayman 3" zone="U"/> <!-- 2144 -->
          <clone name="Rayman 10th Anniversary - Rayman Advance + Rayman 3 (U) (M5+M3)"/>
      </zoned>
-
-
-
      <zoned>
        <bias name="Rayman 3" zone="E"/> <!-- 0875 -->
          <clone name="Rayman 3 (E) (M10)"/>
        <bias name="Rayman 3" zone="U"/> <!-- 0985 -->
          <clone name="Rayman 3 (U) (M3)"/>
        <bias name="Rayman 10th Anniversary - Rayman Advance + Rayman 3" zone="E"/> <!-- 2121 -->
          <clone name="Rayman 10th Anniversary - Rayman Advance + Rayman 3 (E) (M5+M10)"/>
        <bias name="Rayman 10th Anniversary - Rayman Advance + Rayman 3" zone="U"/> <!-- 2144 -->
          <clone name="Rayman 10th Anniversary - Rayman Advance + Rayman 3 (U) (M5+M3)"/>
      </zoned>

Also, don't forget to remove, or at least comment out, the flag regex.

I will try to take a closer look at the file later today to see if I can find any other possible causes.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 7th October 2006, 06:29 wrote:

Did you make any progress with your set, Cartoonite?

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ xuom2 on 14th October 2006, 10:27 wrote:

XMDB was updated?

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 15th October 2006, 00:46 wrote:

Sorry it took me so long to get back to you. I've been rather busy lately so it took a while before I could take a look at this.

My XMDB file is done. All that remains now is to try and find a way to get your tool to create something similar. I will post my files so you can use them for comparison, but since mine was created by hand, it is going to be a lot different from the auto-generated one.

I see there have been some name changes in the DAT file over the past few weeks, so I am now in the process of updating the XMDB file to reflect the new names. In the meantime, here are the files as they existed based on the 20060916 DAT.

Edit: I also see that kazumi has changed the format of her parent-clone file, which will likely require a recoding of your tool if you plan to continue to use her lists as input.

Post by **root** » 15 Oct 2006 06:12

Rif: 824-2 Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 15th October 2006, 07:12 wrote:

After taking a look at your XMDB file, would it be possible to use <zoned type="deferred"> for each game group? We could just manually state <group reg="^____"/> for each rom. Would that work?

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 15th October 2006, 15:20 wrote:

I'm not really sure.

Based on my experience so far, I believe the type="deferred" is required for groupings that use the clone item but the group item should work fine without it. However, when I tried replacing all the <zoned> tags in your latest with <zoned type="deferred"> I still got a parsing error in GoodMerge, although I can't recall what it was at the moment.

<edit>
Seems even that's not right, because the following section works just fine in my XMDB file:

<zoned>
<bias zone="U" name="e-Reader"/>
<bias zone="J" name="Card e-Reader"/>
<clone name="Card e-Reader+"/>
</zoned>

The only thing I can think, then, is that the fact there are titles listed multiple times in the file is causing errors. kazumi's latest list has separated the 2-in-1 titles from their single-game counterparts, so that should resolve any issues of that nature.
</edit>

In theory what you've suggested should work, but in theory what I suggested shouldn't be giving a parsing error in GoodMerge, either.

One thing I do strongly urge you to do is download kazumi's latest lists and take a look at the new format. I'm not sure of the best way to parse that file and, unless you want to manually maintain a list of your own that keeps the original format, that's something that will need to be adressed sooner or later.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ kazumi213 on 15th October 2006, 19:03 wrote:

Original list (.txt) format is being maintained internally. Please report if the new format is less useful for your XMDB purposes.

Have in mind that the "(p)" tags and the numbering can be easily removed by performing a very simple regular expresion replace action in your text editor of choice before parsing the file. Also, it is suppossed the new format is helpful in providing a "group name", but not sure on this.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 17th October 2006, 01:01 wrote:

Here's my latest XMDB. Any update on your end, Rob?

It would be really useful if we could determine the exact difference between <zoned> and <zoned type="deferred"> entries in XMDB. The only reason I even know about the latter is from copying original code from the official GoodMerge GBA file.

Edit: Updated attachment. Several corrections in XMDB were made.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Rob Loach on 20th October 2006, 14:38 wrote:

I tried two different ways of doing it. The first was using:

Code: Select all

	<parent name="Bakuten Shoot Beyblade 2002 - Gekisen! Team Battle!!">
		<group reg="^Bakuten Shoot Beyblade 2002 Team Battle! "/>
		<group reg="^Bakuten Shoot Beyblade 2002 - Gekisen! Team Battle!! "/>
	</parent>

The second method was using:

Code: Select all

      <zoned type="deferred">
         <bias zone="U" name="Kim Possible 2 - Drakken's Demise"/>
         <bias zone="J" name="Kim Possible"/>
         <bias zone="E" name="Kim Possible 2 - Drakken's Demise"/>
         <group reg="^Kim Possible 2 - Drakken's Demise"/>
         <group reg="^Kim Possible"/>
         <group reg="^Kim Possible 2 - Drakken's Demise"/>
      </zoned>

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 21st October 2006, 23:19 wrote:

I can't say why the second version isn't being parsed correctly by GM, but I do know you need to be careful with group statements. Order is important if you are going to use statements where one group is essentially a sub-group of another.

For example, you can't use <group reg="^Megaman Zero 2"/> later in the XMDB file than <group reg="^Megaman Zero"/> because any thing that matches the "^Megaman Zero 2" regex will have already been matched to the unnumbered regex.

I still think our best bet is to remove the flag regex (<flag reg=" (\(([JUEFGISNACKL]{0,3}|Beta|MB(2GBA)?)\)|\[)"/>) from the start of the file and then use clone statements that include the full ROM title for the rest of the XMDB. This way GM will use the whole file name (except for the "XXXX - " ROM number that we tell it to ignore) when looking for a match.

The only thing that we would need to do then is to edit your program to stop it from stripping the sections in parantheses from the end of the ROM titles. Rather than entering the Japanese version of Kim Possible into the KP2 group with <clone name="Kim Possible"/>, we would need <clone name="Kim Possible (J)"/> or, if we want to use group statements <group reg="^Kim Possible /(J/)"/>.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ Cartoonite on 5th November 2006, 03:39 wrote:

Here's the latest XMDB file.

===============================================

Re: GoodMerge XMDB No-Intro GBA Support \ massin on 13th November 2006, 00:13 wrote:

So xuom's tool no longer works for converting to an xmdb? Either way, great job guys