unlicensed archives

[xuom2]

I shared some ideas with rarenight and here is the proposal:

1) in "download dat" page:

01.jpg

- remove "homebrew" option from the form (already removed from archives);
- uncheck "other unlicensed" from default option. this will remove tons of crap from datfiles.

2) in "submit/edit archive" page:

03.jpg

- remove "homebrew" option;
- change "no" to "no: other".

3) in "search" page:

04.jpg

- add "exclude unlicensed" option, as default.

4) in "download daily" page:

05.jpg

- add a "unlicensed" set, with datfiles (if available) set with the following "unlicensed options" flagged:

02.jpg

(please note: we will still have one single "NES" db, containing both licensed and unlicensed archives as today. only filters and exported dats will change).

what do you think?

[Psychofox11]

I might not just have the best grip on the desired outcome. Typically I'd aim for the aftermarket tag, cause I want to see the original era retail (licensed and unlicensed) Like Sega Genesis games by Accolade, etc. I'd rather split on the original era/aftermarket line as many unlicensed were just regular parts of the lineup at the time. It is tricky though, as we don't put (unl) or (aftermarket) on like Limited Run releases.
I guess my brain likes to split it on "could buy it legally in a store while the console was still out" or not.
Also the fact that the same title was sometimes licensed in one territory but not another.
I'm definitely not the person to argue for the final outcome here, just some thoughts after first read.

[rarenight]

Here are my thoughts:

- The original scope of No-Intro was to document proper GBA carts. Removing "Other Unlicensed" from the default DATs heralds back to that. Unlicensed (both Pirate and Aftermarket) is starting to get out of control and is overwhelming DATs, with many people in the community complaining about it. So if splitting off Unlicensed as unique systems is not an option (and we have never been able to agree on how the split should look as a community), the next best option is to split it at the DAT generation level where the only thing that fundamentally changes are the composition of each XML file. That way we won't have to wait for Retool-esque functionality to get integrated into Romvault.

- I noticed that in your second (4) image @xuom2, in "Life span" for the unlicensed set only the "Aftermarket" lifespan was checked. Under the previous logic this would have to be "Both" instead. If your intent is to only split apart Aftermarket then the verbiage should be adjusted.

- While I'm not a fan of Unlicensed in general, I would also be open to only splitting "Aftermarket" (aka @Psychofox11's opinion) in the DAT generation since it is the primary cause of DATs becoming overwhelmed. Pirate is also ridiculous but only really impacts NES, whereas Aftermarket impacts many retro sets from Game Boy to Sega Genesis to Atari Jaguar and has no end to it with the burgeoning NES Maker / GB Studio / etc scenes.

- In a daily pack generation the DATs should manifest like this:
Nintendo - Game Boy
Nintendo - Game Boy (Unlicensed)
Nintendo - Game Boy (Unlicensed) (Private)

Or alternatively this:
Nintendo - Game Boy
Nintendo - Game Boy (Aftermarket)
Nintendo - Game Boy (Aftermarket) (Private)

- Homebrew tag needs to be scrubbed from the form to prevent datters from accidentally using it. Some datters may be unaware that the (Homebrew) tag is deprecated and I've seen it being applied by accident as a result

- Replacing "no" with "no: other" is more specific verbiage which is helpful

- Adding an "Exclude Unlicensed" option in the search box is smart for those of us like me who don't care about it

[Psychofox11]

It seems like in either case we should consider how we tag re-releases so those can be included/excluded. They don't get the Unl or Aftermarket tags. Even an option that doesn't show up as a tag (like physical yes/no) for re-release yes/no may be helpful.

For unlicensed, it just feels odd to me to leave out ones that were in many libraries and on the same store shelves, that people of the time didn't even understand were unlicensed for the most part. It would be leaving out games on Genesis that were official but unlicensed releases during the console's era such as Double Dragon, Star Control, and Test Drive II. I feel like most people making a list of 'original' games for a console would want titles such as these included (just as they'd probably want Tengen for NES if that was affected). Not sure if there's a way to make both paths possible easily.

I just think of the biggest divide being whether it could be purchased in a store legally while the system was still being made (ie no aftermarket or pirate). As someone going off the memory of being there when they were new, they were typically mixed in with the licensed ones on the shelves as well. But I'm just speaking on my thoughts, and am down with whatever makes the most sense or the most people want.

[xuom2]

- I noticed that in your second (4) image @xuom2, in "Life span" for the unlicensed set only the "Aftermarket" lifespan was checked. Under the previous logic this would have to be "Both" instead. If your intent is to only split apart Aftermarket then the verbiage should be adjusted.

true. also, instead of 3 radiobuttons, 2 checkboxes are enough.

you say to keep "other unlicensed" in default options? in this case, "other unlicensed" in "machine lifespan" would be in both datfiles.

[rarenight]

you say to keep "other unlicensed" in default options? in this case, "other unlicensed" in "machine lifespan" would be in both datfiles.

Nah that wouldn't make any sense. It would have to be either this in the DAT pack export:

Nintendo - Game Boy Color
- Licensed -> Licensed
- Machine Life Span -> Machine Life Span

Nintendo - Game Boy Color (Unlicensed)
- Licensed -> Pirate, Other Unlicensed
- Machine Life Span -> Both
- Private archives -> Exclude

Nintendo - Game Boy Color (Unlicensed) (Private)
- Licensed -> Pirate, Other Unlicensed
- Machine Life Span -> Both
- Private archives -> Include (only)


Or something like this, which is @Psychofox11's preference:

Nintendo - Game Boy Color
- Licensed -> Licensed, Pirate, Other unlicensed
- Machine Life Span -> Machine Life Span

Nintendo - Game Boy Color (Aftermarket)
- Licensed -> Pirate, Other unlicensed
- Machine Life Span -> Aftermarket
- Private archives -> Exclude

Nintendo - Game Boy Color (Aftermarket) (Private)
- Licensed -> Pirate, Other unlicensed
- Machine Life Span -> Aftermarket
- Private archives -> Include (only)

[xuom2]

Ok, I think I will proceed with the "Aftermarket" way, because the "Unlicensed" would exclude a lot of "interesting" archives.

(Expect the usual mountain of bugs injected directly in production lololol)

[rarenight]

I think this is a smart move. As someone who has personally bought and datted hundreds of Aftermarket games within DOM, I have come to the opinion that they are a fundamentally different ecosystem and need to be separated in some capacity.

The way our database is structured given that re-releases retroactively inherit the licensed status of their original release, Aftermarkets are new / new-ish indie games that just so happen to conform to the limitations of old retro systems for nostalgia. They have no connection to the physical consoles retail lifespan and are often designed to work specifically for browser emulators with no regard for the original console (they'll say in-game "Press Space to jump" etc).

And on the No-Intro end, the Aftermarket tag is a relatively new addition. Given that I think it makes sense that there is some formal separation between their DATs so that the Aftermarket sets grow independently and are easy to ignore.

I've always had an issue with someone's first game made in GB Studio being right next to Legend of Zelda: Link's Awakening and Pokemon Gold in Romvault, and exporting them separately solves that inconsistency.

[xuom2]

Perfect, it's time to run a giant script that probably will lock DOM for hours, and let's see what happens XD

[rarenight]

FYI @xuom2 I just tested your changes and created a Daily pack with the following settings:

Type -> Standard DAT
Private Systems -> Include
Set -> Main, Source Code, Aftermarket, Unofficial

And it exported the following DATs:

Code: Select all

Aftermarket Folder:
-> Nintendo - Game Boy (Aftermarket) (20240713-090345)

No-Intro Folder:
-> Nintendo - Game Boy (Private) (20240713-090345)

When it should export this instead:

Code: Select all

Aftermarket Folder:
-> Nintendo - Game Boy (Aftermarket) (20240713-090345)
-> Nintendo - Game Boy (Aftermarket) (Private) (20240713-090345)

No-Intro Folder:
-> Nintendo - Game Boy (20240713-090345) (excluding the Aftermarket ROMs)

This is because the Private entries are also Aftermarket by design.


When excluding the Private systems, it generates correctly. These settings:

Type -> Standard DAT
Private Systems -> Exclude
Set -> Main, Source Code, Aftermarket, Unofficial

Correctly generates the following:

Code: Select all

Aftermarket Folder:
-> Nintendo - Game Boy (Aftermarket) (20240713-090345)

No-Intro Folder:
-> Nintendo - Game Boy (20240713-090345) (excluding the Aftermarket ROMs)

[EggmanPEI]

I spot-checked two systems with these new daily dats.. and there are discrepancies. I chose 2 systems that have not had a dat update anytime in the last 2 days or before. date/timestamps between both sets are identical. Any roms deemed "MIA" have been included in counts.

"newest old" dat prior to new changes, rom counts:
Atari - 2600 (20240803-015308) - 849
Atari - 2600 (Private) (20240803-015308) - 6
--------
total # of roms: 855

new dats from the daily, rom counts:
Atari - 2600 (20240803-015308) - 810
Atari - 2600 (Aftermarket) (20240803-015308) - 29
Atari - 2600 (Private) (20240803-015308) - 6
--------
total # of roms: 845

List of the 10 roms missing, specifically that should belong in the new aftermarket set but are no longer part of any Atari 2600 dats generated by the daily love pack:
Anguna 2600 (World) (Aftermarket)
Go Fish! (World) (NTSC) (Aftermarket)
Go Fish! (World) (PAL) (Aftermarket)
Halo 2600 (Europe) (PAL) (Aftermarket)
INV+ (World) (2004-04-08) (Beta) (Aftermarket)
INV+ (World) (2004-04-29) (Beta) (Aftermarket)
INV+ (World) (Aftermarket)
Oozy the Goo Slime Quest (World) (Cart) (Aftermarket)
synthcart (Europe) (PAL) (Aftermarket)
synthcart (USA) (NTSC) (Aftermarket)


I did the exact same thing with the "Nintendo - Nintendo Entertainment System (Headered)" set. dat dates match

"newest old" dat prior to new changes, rom counts:
Nintendo - Nintendo Entertainment System (Headered) (20240804-081828) - 6412
Nintendo - Nintendo Entertainment System (Headered) (Private) (20240804-081828) - 697
--------
total # of roms: 7109

new dats from the daily, rom counts:
Nintendo - Nintendo Entertainment System (Headered) (20240804-081828) - 4021
Nintendo - Nintendo Entertainment System (Headered) (Aftermarket) (20240804-081828) - 2389
Nintendo - Nintendo Entertainment System (Headered) (Private) (20240804-081828) - 697
--------
total # of roms: 7107

List of the 2 roms missing, specifically that should belong in the new aftermarket set but are no longer part of any NES Headered dats generated by the daily love pack:
Mighty Final Fight (USA) (Capcom Classics Mini Mix) (Aftermarket)
Strider (USA) (Capcom Classics Mini Mix) (Aftermarket)

I hope this is explained clear enough and helps to find out what is happening with these "missing" entries in the new aftermarket dats generated by the daily dat process.

[rarenight]

Anguna 2600 (World) (Aftermarket)
Go Fish! (World) (NTSC) (Aftermarket)
Go Fish! (World) (PAL) (Aftermarket)
Halo 2600 (Europe) (PAL) (Aftermarket)
INV+ (World) (2004-04-08) (Beta) (Aftermarket)
INV+ (World) (2004-04-29) (Beta) (Aftermarket)
INV+ (World) (Aftermarket)
Oozy the Goo Slime Quest (World) (Cart) (Aftermarket)
synthcart (Europe) (PAL) (Aftermarket)
synthcart (USA) (NTSC) (Aftermarket)

Mighty Final Fight (USA) (Capcom Classics Mini Mix) (Aftermarket)
Strider (USA) (Capcom Classics Mini Mix) (Aftermarket)

I noticed that all of the missing entries in the Aftermarket DAT which you posted are mislabeled as "Licensed" in DOM. That's probably the culprit, the Aftermarket DATs probably only count the entries which are explicitly labeled as Unlicensed.

I'm going to go through each set and assign all of them as Unlicensed and that'll probably repopulate them within the Daily Pack generation.

@xuom2 I recommend adding a function on the Submit page where if a datter selects "Aftermarket: yes" within DOM, they cannot also select "Licensed: yes". The option should become greyed out or a message box should pop up saying "This combination is not allowed. All Aftermarkets must be Unlicensed."

Also, the "Licensed: yes" and "Private: yes" combo should be forbidden as well. The Private ROM designation was created to designate paid-only homebrew ROMs from small indie devs, it should have nothing to do with licensed re-releases of classic ROMs on Switch / Steam / etc

[xuom2]

@xuom2 I recommend adding a function on the Submit page where if a datter selects "Aftermarket: yes" within DOM, they cannot also select "Licensed: yes". The option should become greyed out or a message box should pop up saying "This combination is not allowed. All Aftermarkets must be Unlicensed."

ok, now it's forbidden. I checked the DB and fixed a couple of items left in Source Code - Atari - 2600.

Also, the "Licensed" and "Private" combo should be forbidden as well.

done. I don't totally agree with this but probably it avoids mess in download options.

[rarenight]

@xuom2 I recommend adding a function on the Submit page where if a datter selects "Aftermarket: yes" within DOM, they cannot also select "Licensed: yes". The option should become greyed out or a message box should pop up saying "This combination is not allowed. All Aftermarkets must be Unlicensed."

ok, now it's forbidden. I checked the DB and fixed a couple of items left in Source Code - Atari - 2600.

Also, the "Licensed" and "Private" combo should be forbidden as well.

done. I don't totally agree with this but probably it avoids mess in download options.

Thank you! Having built-in failsafes to avoid potential datter error is awesome @EggmanPEI's issue shouldn't be a factor for daily pack DAT generation moving forward, as datters can't mistakenly choose the option that would exclude the ROMs from the daily packs anymore.

[CoreyEMTP]

From a user perspective, I really appreciate the new changes as it goes a long way towards replacing the functionality of the (now defunct) Retool. My only concern, given the discrepancies already identified, is that there will be little-attended-to DATs that may not receive vetting. Is there a plan in place to address this, e.g. a community forum to report findings?