Question about digital Switch title hashing

General No-Intro related discussions.
Post Reply
nqwpharz
Posts: 13
Joined: 19 Sep 2018 10:38

Question about digital Switch title hashing

Post by nqwpharz »

For context, I have a hacked Switch and a bunch of digitally installed titles. My original goal was to dump them as NSPs, verify them against no-intro's NSP dat, contribute anything missing, and possibly convert them to XCIs, if it's possible to do that in a way that also leads to a dat match. I'm stuck at the "verify them against no-intro's NSP dat" step.


I was following along this guide for dumping, but it ends by extracting data from the NSP files and datting them like the CDN dumps, as a bunch of NCA files. I was able to follow along successfully to the end, and verify my results with the CDN dat, but it's not really what I was trying to do.

I'm aware an NSP dat exists, though - and nxdumptool has an option to verify dumps against it. When using that tool, along with the "Verify dump using No-Intro database" option enabled, I did get some database matches, plus some that weren't matches (I think because the NSP dat is only bothering with update data and DLC data?).

The part that's confusing me, though, is how the CRCs are calculated - as an example, when I dumped the update data for Katamari Damacy Reroll, nxdumptool told me I had a database match with a CRC of 5647acb4. When I calculate the CRC hash of the NSP generated on my computer, I get a value of 59b90a18. And the NSP dat I actually have says it should be 347b0506.

Is there some special subset of the NSP I should be hashing instead of the full file? Is there a wiki guide for this process that I just haven't been able to find? Where is nxdumptool pulling the hashes it's verifying against from, any why are they different from the values in the NSP dat I have?

Any information is appreciated. Thanks.
Hiccup
Datter
Posts: 1002
Joined: 09 Oct 2015 11:29

Re: Question about digital Switch title hashing

Post by Hiccup »

I'm not sure what exactly nxdumptool does, but if you want to verify the data against the dat, you should just compare the NCAs (and titlekey or cetk, if its in the dat). The rest of the NSP data is variable - it depends on which program is used to create the NSP and which settings are used. Its the same with XCI conversions. There's no standard for NSP or XCI conversions of digital titles, so its not suitable for datting.
nqwpharz
Posts: 13
Joined: 19 Sep 2018 10:38

Re: Question about digital Switch title hashing

Post by nqwpharz »

I understand - though why do xci and nsp no-intro dats exist then? Are they some attempt to standardize NSPs and XCIs? Outdated relics? Does any wiki information exist on them?

I also just found the 5647acb4 hash nxdumptool shows me comes from the cdn dat, as the hash for the .cnmt.nca file, so nxdumptool itself seems to be doing what you suggested.

I haven't really dug into my dumps yet because I was still confused over these points, but my understanding is the cdn datting is somewhat automated for digital titles and as such should be relatively complete - is it not worth my time to look into the dumped titles that nxdumptool says don't have a database match, or could they still be interesting? (Edit: The only dumps nxdumptool identifies as already in the database are both BOTW DLCs, a Shantae: Half-Genie Hero: Ultimate Edition update, a Rayman Legends update, and a Katamari Damacy Reroll update - so not really sure what's up with all of that.)
Hiccup
Datter
Posts: 1002
Joined: 09 Oct 2015 11:29

Re: Question about digital Switch title hashing

Post by Hiccup »

nqwpharz wrote: 06 Jan 2021 18:07 I understand - though why do xci and nsp no-intro dats exist then? Are they some attempt to standardize NSPs and XCIs? Outdated relics? Does any wiki information exist on them?
XCI are game cart dumps. Digital NSPs are datted due to scene releases using that format.
nqwpharz wrote: 06 Jan 2021 18:07 I also just found the 5647acb4 hash nxdumptool shows me comes from the cdn dat, as the hash for the .cnmt.nca file, so nxdumptool itself seems to be doing what you suggested.
Ah. But for redumps you should compare all the files just to make sure.
nqwpharz wrote: 06 Jan 2021 18:07 I haven't really dug into my dumps yet because I was still confused over these points, but my understanding is the cdn datting is somewhat automated for digital titles and as such should be relatively complete - is it not worth my time to look into the dumped titles that nxdumptool says don't have a database match, or could they still be interesting? (Edit: The only dumps nxdumptool identifies as already in the database are both BOTW DLCs, a Shantae: Half-Genie Hero: Ultimate Edition update, a Rayman Legends update, and a Katamari Damacy Reroll update - so not really sure what's up with all of that.)
CDN datting is not automated. But there are sometimes bulk imports of batch-processed data. It is still worth you submitting data - its not really possible to rip directly from the CDN anymore, so this data has to be dumped from consoles.
nqwpharz
Posts: 13
Joined: 19 Sep 2018 10:38

Re: Question about digital Switch title hashing

Post by nqwpharz »

I finally started getting around to this, and following the instructions for datting on this page is reasonably straightforward, until I get to the "Optional" step 2 in the final section, where a few instructions become unclear.

Are these extra steps "optional" enough that I shouldn't bother? Or is it worth clarifying? I've determined in step 1 that I have a stripped personalized ticket. In step 2, is the "[lowercase_rights_id]" the 16 bytes I grabbed in that same step? In step 3, how to I determine which .nca is the "Program" type nca? Can there be more than one? Can there be none?

Also, are the .txt files generated from each .nca file in the previous section meant to be included in the information I provide? I already made a batch script to generate these (Which I can share, since the wiki says one would be desirable), but don't know if they're actually wanted, or if they maybe contain information tied to my nintendo account specifically.
Hiccup
Datter
Posts: 1002
Joined: 09 Oct 2015 11:29

Re: Question about digital Switch title hashing

Post by Hiccup »

nqwpharz wrote: 12 Jan 2021 03:58 I finally started getting around to this, and following the instructions for datting on this page is reasonably straightforward, until I get to the "Optional" step 2 in the final section, where a few instructions become unclear.

Are these extra steps "optional" enough that I shouldn't bother? Or is it worth clarifying? I've determined in step 1 that I have a stripped personalized ticket. In step 2, is the "[lowercase_rights_id]" the 16 bytes I grabbed in that same step?
The 16 bytes is the title id. You can get the rights id from the ticket filename or from the ticket itself using nstool.
nqwpharz wrote: 12 Jan 2021 03:58 In step 3, how to I determine which .nca is the "Program" type nca? Can there be more than one? Can there be none?
nstool should allow you to print out the NCA listing contained in the cnmt nca - this should say which is the "Program" nca.
[/quote]
nqwpharz wrote: 12 Jan 2021 03:58 Also, are the .txt files generated from each .nca file in the previous section meant to be included in the information I provide? I already made a batch script to generate these (Which I can share, since the wiki says one would be desirable), but don't know if they're actually wanted, or if they maybe contain information tied to my nintendo account specifically.
The text files aren't needed. Those are just for you to take information from for the submission.
DarkMatterCore
Posts: 13
Joined: 06 Apr 2011 16:34

Re: Question about digital Switch title hashing

Post by DarkMatterCore »

nqwpharz wrote: 06 Jan 2021 07:36 I was following along this guide for dumping, but it ends by extracting data from the NSP files and datting them like the CDN dumps, as a bunch of NCA files. I was able to follow along successfully to the end, and verify my results with the CDN dat, but it's not really what I was trying to do.
Just as Hiccup previously explained, NSP packages can't be datted because they're essentially concatenated files, one after another, along with a header that provides information about how to access each individual file entry. The actual file order varies between each program/tool capable of generating NSPs.

XCI dumps are standard because they represent raw data from Switch gamecards, thus they can be fully datted.
nqwpharz wrote: 06 Jan 2021 07:36 I'm aware an NSP dat exists, though - and nxdumptool has an option to verify dumps against it. When using that tool, along with the "Verify dump using No-Intro database" option enabled, I did get some database matches, plus some that weren't matches (I think because the NSP dat is only bothering with update data and DLC data?).
nxdumptool calculates the CRC32 checksum from the CNMT NCA and uses the No-Intro API to look up for a match.

The other titles you dumped most likely haven't been fully datted, which means their NCA hashes aren't currently available at No-Intro.
nqwpharz
Posts: 13
Joined: 19 Sep 2018 10:38

Re: Question about digital Switch title hashing

Post by nqwpharz »

DarkMatterCore wrote: 30 Jan 2021 16:24 Just as Hiccup previously explained, NSP packages can't be datted because they're essentially concatenated files, one after another, along with a header that provides information about how to access each individual file entry. The actual file order varies between each program/tool capable of generating NSPs.
For what it's worth, I do own a .dat file with over 6000 entries for .nsp files, so someone is datting them - though I accept your point about there being no consistent standard for them. I don't know how the entries in this dat were generated, and I'm not too concerned with it at this point either.
DarkMatterCore wrote: 30 Jan 2021 16:24 XCI dumps are standard because they represent raw data from Switch gamecards, thus they can be fully datted.

nxdumptool calculates the CRC32 checksum from the CNMT NCA and uses the No-Intro API to look up for a match.

The other titles you dumped most likely haven't been fully datted, which means their NCA hashes aren't currently available at No-Intro.
I did eventually figure out the CRC32s were from the CNMT NCA after I found a CDN dat file to compare my own hashes against. For what it's worth, I also posted all of the hashes for the .nca files I extracted in another thread, both the matches and non-matches.

Also forgot to thanks Hiccup for their last post explaining those details - they were helpful. Thanks!
Post Reply