Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

General No-Intro related discussions.
Locked
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

rarenight wrote: 06 Jan 2024 04:25 Correct me if I'm wrong, but it shouldn't affect the legacy set P/C relationships. If an Aftermarket migration to their own DATs would destroy the legacy P/C relationships then yes that would put a wrench in my plan.
Maybe @omonim2007 would know more, who brought up the warning about the P/C earlier.
omonim2007 wrote: 05 Oct 2023 09:53 Having gone through the length and breadth of retro sets over the past few years, I can confidently say that there is NO UNIVERSAL RULE for how to divide sets in the manner discussed here. We will not be able to split the dats, as these changes will be destructive and the Parent-Clone relationship will be broken FOREVER.
I'm greatly assuming that's the (only?) major objection to the new plan. Otherwise I'd be thrilled if something new came along that gave you datters less trouble to work with.
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

Found one possible P/C troublemaker.

Crystal Mines II (Lynx) = vintage Atari published 1992
https://datomatic.no-intro.org/index.ph ... =30&n=0019


Crystal Mines II - Buried Treasure (Lynx) = licensed to Songbird but later, like 2003
https://datomatic.no-intro.org/index.ph ... =30&n=0266

It's the original game + expansion maps. But still actively on/off being sold by Songbird, 20 years later. And by the Evercade.
https://songbird-productions.com/produc ... nes-ii-bt/
https://evercade.co.uk/cartridges/atari ... cartridge/


When the dats are split, how to restore the P/C link?


EDIT: The Evercade version of CM2BT has some tweaked maps from the Songbird one, so the rom hash won't match. Creating a future P/C problem that will need handling.


EDIT2: There's also the other Songbird titles. He continues to restore prototypes and make commercial games out of them.
- https://songbird-productions.com/product/centipede/
- https://songbird-productions.com/produc ... -missions/

It's more Centipede I'd be worried since there is an existing proto floating around
https://datomatic.no-intro.org/index.ph ... =30&n=0016


EDIT3: Well, if a new feature was created to allow cross-P/C linking between dat platforms, it could solve those messy GB/C relationships and the aftermarket ones too!
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

rarenight wrote: 06 Jan 2024 18:31 The aftermarket DATs would be for wholly new games released after the console's discontinuation that weren't an older game or do not have any potential P/C link to an older game. A clean break between old-school and new-school software ecosystems for systems that have a thriving homebrew scene. Original games + their re-release derivatives = original DATs.
That explanation is very satisfying to someone like me, and it's simple! :)

The existing P/C links are database "hyperlinks" or something locally "relative"? I can't think how they'd break if the entries are not auto-modified by a script. Even if some get broken, you'll be on top of it when they get reported!
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

I do have one concern. Better explain with an example.

Dat
0050
0051 = Aftermarket
0052 = Parent
0053 = Aftermarket
0054 = Aftermarket
0055 = Clone (0052)

So 0051 - 0053 - 0054 will be moved out to the new dat. This will leave holes in the old database. Is it okay to leave those entries as "dead" space?

There's likely 1M+ links on the net pointing to the existing entries as-is today. I don't think they can be safely reused, merged or defragged except for redirecting "moved" aftermarket items to their new dat homes.

Otherwise I'm completely sold on the migration.
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

rarenight wrote: 08 Jan 2024 10:03 ^ Those superfluous tags (SGB Enhanced) (GB Compatible) (Aftermarket) (Unl) that we now don't have to check or manually add—because all of that is implied or irrelevant in a catchall aftermarket DAT with no legacy baggage—really adds up across hundreds of new entries. And we don't have to worry about manually determining the Private status anymore either because the whole DAT is private.
Those strings of tags can be really annoying to track, even visually speaking. They do have some usefulness but maybe placed inside the dat entry itself (like eeprom chips or attached hardware) and not part of the actual title.

-----
rarenight wrote: 08 Jan 2024 10:03 If I remember correctly, those gaps would be automatically reordered by the database but @xuom2 would know a lot more if such a move is feasible or if it would break more than just aftermarket P/C associations.
Worst case I can imagine would be to "freeze" the old dat into some "museum" dat, in case something gets overlooked later. And create two clean new dats from the original ones, with hopefully a script auto-repairing any damage.

I hope your thought-out plan eventually transitions from abstract to concrete, because of the exploding amount of Aftermarkets that keep flooding the dats.
sCZther
Datter
Posts: 156
Joined: 16 Jun 2014 21:09

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by sCZther »

rarenight wrote: 08 Jan 2024 10:03 [Nintendo - Game Boy Color (Private)]
Pilgrim's Peril (World) (v1.0.2) (SGB Enhanced) (GB Compatible) (Aftermarket) (Unl)

Under my proposed system, it would be datted much cleaner like this:

[Aftermarket - Game Boy & Game Boy Color]
Pilgrim's Peril (World) (v1.0.2)

^ Those superfluous tags (SGB Enhanced) (GB Compatible) (Aftermarket) (Unl) that we now don't have to check or manually add—because all of that is implied or irrelevant in a catchall aftermarket DAT with no legacy baggage—really adds up across hundreds of new entries. And we don't have to worry about manually determining the Private status anymore either because the whole DAT is private.
I don't understand it at all? I would of course like to know if the homebrew release is compatible with SGB or GBC only. That unless your proposal is to not track it in any of the dats. But then it has nothing to do with aftermarket releases.
herocopter wrote: 07 Jan 2024 01:20 There's likely 1M+ links on the net pointing to the existing entries as-is today. I don't think they can be safely reused, merged or defragged except for redirecting "moved" aftermarket items to their new dat homes.
This is very important. It would invalidate any discussion about no-intro database entries over the years that are tied to a link or even just a id.
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

sCZther wrote: 10 Jan 2024 21:56 I don't understand it at all? I would of course like to know if the homebrew release is compatible with SGB or GBC only. That unless your proposal is to not track it in any of the dats. But then it has nothing to do with aftermarket releases.
I like those tags too, listed optionally somewhere at least.


I think the proposal is that:
[Nintendo - Game Boy Color (Private)]
Pilgrim's Peril (World) (v1.0.2) (SGB Enhanced) (GB Compatible) (Aftermarket) (Unl)
== GBC dat only. Tags added to notate cross-dat relationship to Game Boy (Private) dat.


[Aftermarket - Game Boy & Game Boy Color]
Pilgrim's Peril (World) (v1.0.2)
== Meaning all GB and GBC aftermarkets are combined into one dat, instead of 2 like today. So we can throw away SGB + GB + Aftermarket + Unl by default.

Maybe we'd have to file reports on the forum about the compatibility: DMG only - CGB only - DMG + SGB only - DMG + SGB + GBC.
(asking datters to notate them somewhere inside the entry itself, like comments)
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

I still like the new plan - just wanted to think out anything bad that could happen before it becomes irreversible. But we can only theorize about the internal database workings so I guess nothing can actually get done.


How about starting with something simple? The GB + GBC are a high priority so merge those two private dats into a new one. Because they're already invisible to the public, who cares about breaking any net links. If an aftermarket P/C breaks, @rarenight can step in.

All new Aftermarkets (unlicensed games) will be pushed into "Aftermarket - Gameboy + Color".

The existing public dats can be dealt with later. The current handling of the never-ending GB/C aftermarkets is creating lots of unnecessary burden - Expedited processing would be beneficial to the team.


As for those troublesome (GB) (SGB) tags, don't make them a priority when 1st datting. Headers are unreliable and not used by the licensed hardware themselves. The burden should be placed on the *users* to file reports that a game does not work in DMG / CGB mode, is incredibly DMG / CGB buggy (alpha or beta quality, not release candidate quality), or cannot be 100% completed in (non-)GBC mode. By default, assume GB + SGB + GBC.

- SNES headers can flat-out lie about their size (1.5 MB reported as 1 MB), FastRom (not true - never works), Mapping mode (not lorom), Expansion chip (doesn't exist or wrong config), Manufacturer code, misc.

- Commercial Megadrive headers lie about their checksum (as anti-hack protection) or expansion flags (I think it was Demons of Asteborg Steam release).


There will always be lots of arguments over this - less important details. But I think you've strongly thought things out enough that something can be started. Perfection is not required - it takes too much time to get there. But maybe the dom doesn't have to be transformed in 1 giant step.



EDIT: Because the Aftermarket is 100% invisible to the public, maybe change it so that just hashes (+ misc) are hidden to non-staff? So that users can read just the non-critical stuff. Otherwise most of us won't see the GB - SGB tags and header comments. :lol:



EDIT2: Going to try and make this my final blathering. Then run away forever and do something more fun.

In the meantime while the current dat system is kept, how about changing the rules for all new entries?

- By default treat everything as Aftermarket, because there's a 99.9999% that game is unlicensed. Does not matter if it's free or paid. Blanket rule. If a title is known to be not Aftermarket, then a datter can switch the flag off (Berzerk Enhanced 2600 situation).

- No longer mark GBC games as (GB compatible) or (SGB compatible) in the title. Assume not backward compatible, until completely proven otherwise by a playthrough or gamer testimony. Or explicit (S)GB-compatible marketing, not by the rom header or whatever GB studio claims.

- SGB .. well. Has to show it uses a SGB feature like border or audio or something special. And works on real hardware.



EDIT3: By tagging as Aftermarket, Unlicensed is now automatically implied. So no double-tag needed.
User avatar
Arctic Circle System
Datter
Posts: 64
Joined: 21 May 2020 04:43

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by Arctic Circle System »

I do agree with splitting aftermarket Game Boy/Game Boy Color ROMs into a separate dat, particularly those that can't easily be sorted into one of the standard GB(C) dats due to compatibility weirdness. Those cases can be viewed as using a different format from the standard one if that helps. Though just moving all ROMs in those dats that are currently labeled as aftermarket and aren't in a P+C relationship with any ROMs that don't have an aftermarket tag would be acceptable as well for the sake of simplicity. I'm not staunchly opposed to aftermarket titles for the NES, Genesis, SNES, A26, etc being split either, though I would advise looking into the viability of solutions that keep those dats together before going through a split. More specifically, refining the download settings to have a cleaner UI would likely be useful here. Adding an advanced search feature, allowing users to search by tags, would also alleviate any problems with aftermarket stuff "overshadowing" the licensed stuff. Splitting the dats may make things more complicated than they need to be. Beyond those issues, I don't see what the problem is with there being a lot of aftermarket titles in the main dats for their platform. I also take issue with the idea that they should be split because they're mostly played on emulators. While I won't deny that it's likely that they are mostly played on emulators, one could say the same about Virtual Console ROMs and Star Fox 2. Ultimately, they typically run on original hardware just fine, and many people do buy physical cartridges of these games to play on original hardware.
herocopter
Posts: 24
Joined: 08 Apr 2019 16:57

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by herocopter »

Arctic Circle System wrote: 11 Jan 2024 12:17 I also take issue with the idea that they should be split because they're mostly played on emulators.
Did @rarenight slightly imply that? I sure wouldn't split the dats on that.


--- Seeing as the proposal has lost traction and likely dying, time to play the meanie ---


Thinking on a tangent.


Some discs like Iron Helix are both PC and Mac compatible (and other edutainment titles). I would dupe that same title under both PC and Mac dats.

Some Win98 titles are not Win95 compatbible. It could either go into the Win95-98-ME dat, or even just the 98 dat for purists. Titles that are Win98 and 95 backward would slot only into the 98 dat for me, with auto-implied 95 compatibility.
-- EDIT: Why not. Just place it into both 95 and 98 dats for people searching.

NT-2K-XP are largely different enough OS (nuts, bolts code architecture) from 95-98-ME that compatibility is not guaranteed at all. Two dats.

NES, MD, et al can be left alone. There's not much argument here, whether digital-only or CIB.

I think WS is largely different enough from WSC, same with NGP and NGPC. Don't know enough about technical details (programming) to comment.


GB is a different enough hardware than the GBC. You can't just flip some flags and make it run correctly on a GBC, or other way. It takes some work to even make a DMG game run on CGB mode (still B/W) but double speed, aka hacker "dmg-fastrom" mode.

And SGB is nearly the same interface as GB - basic compatibility is basically guaranteed? They cannot auto-claim that with GBC.

-- (GB Studio can be a high-level "cheat" and doesn't ensure good cross-compatibility. If the developer actually cared about adding DMG functionality, it wouldn't be that broken imo.) --

What this means? I'd keep S/GB separate from GBC, because of the technical silicon differences. SGB would be a footnote in the comments.


Aftermarket sample: YNXA.
- Marketed as GBC first priority.
- GB + SGB is a feature. A bonus. They do work well actually.


EDIT: Chopped comment.


I'd dat YNXA under both GB and GBC, just because they're different enough technical platforms. And we have 2 dats already. XD
(Annoying = fine with me now!)


Other examples of GB/C straddlers that are not clearly defined? We must teardown those roms!


EDIT2: Ugh - sigh. Must stop with the edits.
Arctic Circle System wrote: 11 Jan 2024 12:17 While I won't deny that it's likely that they are mostly played on emulators, one could say the same about Virtual Console ROMs and Star Fox 2.
I thought VC, Star Fox 2, Trials of Mana, Devi and Pii, Aleste 3, Run and Jump and others would stay as non-aftermarket even if there is a dat split. Being official and all.
User avatar
Arctic Circle System
Datter
Posts: 64
Joined: 21 May 2020 04:43

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by Arctic Circle System »

herocopter wrote: 11 Jan 2024 15:42 I thought VC, Star Fox 2, Trials of Mana, Devi and Pii, Aleste 3, Run and Jump and others would stay as non-aftermarket even if there is a dat split. Being official and all.
I mean I think at least Aleste 3 should be listed as aftermarket because it was developed and released long after the Game Gear's life. At least I think it was developed after its EoL. I don't know about the others though.
Fugus
Posts: 65
Joined: 11 Oct 2022 17:50

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by Fugus »

Arctic Circle System wrote: 21 Jan 2024 22:39 I mean I think at least Aleste 3 should be listed as aftermarket because it was developed and released long after the Game Gear's life. At least I think it was developed after its EoL. I don't know about the others though.
I personally don't know about that game, but my logic behind it would be this.

If the game was released on the virtual console or one of the mini-consoles or by the consoles developers I could see it considered part of the official stuff.

If the game was released by someone other than Sega as part of some other collection similar to how Steins Gate was on the Switch, it should go to the aftermarket since it was not released by the systems manufacturer or as part of the virtual console or console-mini hardware since.
KingMike
Posts: 699
Joined: 22 Sep 2012 16:36

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by KingMike »

Arctic Circle System wrote: 21 Jan 2024 22:39
herocopter wrote: 11 Jan 2024 15:42 I thought VC, Star Fox 2, Trials of Mana, Devi and Pii, Aleste 3, Run and Jump and others would stay as non-aftermarket even if there is a dat split. Being official and all.
I mean I think at least Aleste 3 should be listed as aftermarket because it was developed and released long after the Game Gear's life. At least I think it was developed after its EoL. I don't know about the others though.
Trials is an authorized localization of a game that was released during its original lifespan.

Yeah, GG Aleste 3 was a game developed in 2020 for a compilation released in Japan for Switch. I heard it wasn't completely compatible with OG hardware but was able to be patched by fans. (so sort of like DK Original Edition wasn't properly compatible with OG NES hardware)
rarenight
Datter
Posts: 705
Joined: 19 Mar 2017 09:41

Re: Poll: Should No-Intro split unlicensed ROMs into separate DATs from licensed ROMs?

Post by rarenight »

@xuom2's approach (as discussed on Discord) has solved this issue elegantly. Within DOM, the sets remain combined because it is more efficient to aggregate all ROMs of a given format on a per-system basis. But when exporting No-Intro sets into daily packs and within DATVault, the Aftermarket titles are split apart into separate DATs, which represents their unique ecosystems, and accommodates those of us (like me) who don't want to collect the burgeoning, never-ending homebrew scenes.

It was also important to define "Aftermarket" games as games originally released after the console lifespan so that they are reliably separated into two distinct buckets: old games (and their re-release derivatives) in one bucket, and entirely new homebrew games in the other bucket. See https://wiki.no-intro.org/index.php?tit ... rket_Guide for more info. No-Intro must bundle all ROM re-releases in the same sets for consistency in order for this DAT split to work, and I believe we have now accomplished this task in a uniform manner.

We have received minimal pushback and near-unanimous praise since the introduction of this paradigm shift, so I think this thread has finally run its course. Thanks to all who participated!
Locked