1G1R mode / future of dats

[kazumi213]

Again, this is awesome... Thanks a lot for these features Roman

I will do some tests tomorrow

[resxto]

many thanks

[kazumi213]

Ok, I've been testing 7z compression and I think there is a problem when operating in Merged Mode. Maybe it's something that I'm doing wrong, so please feel free to correct me.

To take full advantage of size reduction when merging using 7z, merged archives must be solid ones. However merged archives are not being created as solid even when specified. I think the cause is that CMPro Rebuilder doesn't create the merged archive in "1 step" (by compressing all clones in 1 operation) but it updates the archive by adding new clones as the Rebuilder finds them. This "update" approach can not create a solid archive because 7zip currently doesn't support updating solid archives.

For my tests I used "A", "B", "P" and "T" folders (219 files) from No-Intro N64, with DAT-o-MATIC P/C DAT and 7zip standalone commandline (7za.exe) v4.57. I rebuilt to 7z compressed archives in Unmerged and Merged Mode.

The default packer.ini switches for 7z are:

7zip_compress_para = a -y -t7z %1 %2
7zip_delete_para = d %1 %2

(sidenote: "-y" switch doesn't apply to "a" command and "-t7z" switch is not required)

(Normal compression, solid mode off, 16 Mbyte dictionary)

When I used the following switches:

7zip_compress_para = a -mx9 -ms -m0=LZMA:d28 %1 %2
7zip_delete_para = d -mx9 -ms -m0=LZMA:d28 %1 %2

(Ultra compression, solid mode, 256 MByte dictionary)

I got virtually the same overall final size as with default switches, even for the test in Merged Mode. And both Merged and Unmerged tests for each settings yield the same overall size. After checking, merged archives were not solid.

This can be tricky to fix as it would require to modify Rebuilder logic, so when in Merged Mode, it should first locate all clones, then compress them all at once. Maybe it would be easier to modify the "Merger" to work this way (it would require to allow to select compression format here too).

I know this is probably out of question. Thanks anyway for all the time and effort you put into CMPro Roman

[Lukeage]

Said stuff about merge mode/solid archives

Firstly, another thanks to Roman for all his hard work. I've been meaning to complement you for ages, but I've been too lazy to make an account on the new forum

To get it to do solid archives is going to take a bit more work, but I don't think it would require a huge overhaul of the current process (though that depends on how its actually coded ). While not the most efficent way, if you flagged any archives that are created or modified during the rebuild process (or just stored them elsewhere), then afterwards, decompress them to a temp directory then recompress them as solid archives.

This is fundamentally how I do it now. I store (no compression) to zip files, then I run a script to extract each zip file and recompress the files with 7zip.

I agree with kazumi213 though, that is is probably out of the question and as far as I am concerned is of extremely low priority. I only have run a batch file after clrmame has done its stuff to get my solid archives.

[kazumi213]

This is fundamentally how I do it now. I store (no compression) to zip files, then I run a script to extract each zip file and recompress the files with 7zip.

Could you (re)post your batch file? I'm interested on your switches

[Lukeage]

Batch files are attached. Be warned that my settings are aimed at people with 4gig or more of ram (7-zip can require up to 2.7 gig with those options). Reducing the dictionary size to 192m will be fine for 3gig and 128m for 2gig.

There are only 10 sets which actually benifit from using a dictionary larger than 128m (read: are larger than 128meg), so anyone using 128m will produce almost an optimal set anyway.

2436 + x103 compresses to 32,960,085,697 bytes using this.

[kazumi213]

Great Lukeage

Thanks for the quick tips on dictionary size. I assume you're using a 64bit OS. Btw, what's that "-bd" switch for?

Amazing numbers you get. Uncompressed NDS is 110+ Gbytes and just 75+ zipped.

[Lukeage]

I run 64-bit Vista, and use both 64-bit CMPro and 7-zip.

I'm not sure why I have the -bd flag. It disables the percentage indicator. Was probably left over from where I copied the settings from originally and I never removed it.

[Bogy]

well....rebuilder is file based.....not set based. So it takes a file from source (no matter if it's compressed) and rebuilds all instances of it (e.g. if a crc is listed multiple times in the datfile).
Theoretically it can be modified to work a bit semi-setbased (the remove-source-after-rebuild works this way)...but in fact I'm a lazy bum....and got too many real life job's tasks to do...yeah...I have to earn some money actually....

Time will tell....maybe if I find an evening or two magic things can happen....Since 1997 I think clrmame is complete but then users come ...

But even if the rebuilder is set based, the call of the external tool will be file by file. Currently packer.ini holds only values for a single-add and a single-delete operation. File lists aren't yet supported. I only added rar/7z simply because of getting it a bit more complete.

[kazumi213]

The fact that you leave that door open is enough for me Roman When it's done then.

Regarding the file by file external tool call limitation: how about replacing the contents of the placeholders when in Merged Mode? One would contain a working/temp dir (named according to set, root path to be defined by user) with uncompressed ROMs, the other a mask (*.*). This is the approach in the batch file by Lukeage.

[Bogy]

fyi, be sure you keep a copy of your packer.ini before updating to the next cmpro....the file will be overwritten otherwise...I will think if I support something like a packer01.ini which is used when specified...

[kazumi213]

That sounds promising...

Btw, I suggest the following 7z parameters as default in your distribution. My tests show that overall rebuild time just increases by seconds vs current ones.

7zip_compress_para = a -mx9 -ms %1 %2
7zip_delete_para = d -mx9 -ms %1 %2

(Ultra compression, solid mode on, 64 Mbyte dictionary)

The user then just needs to add his own parameters to increase dictionary size, activate the multi-threading support or whatever tweak he likes.

[Bogy]

There is a general problem with solid archives. As soon as you created one, you can't add or delete files from them using "7z.exe d" or "7z.exe a"......so once the file is generated (e.g. by the rebuilder), you won't be able to fix them (if something changes) using the scanner....

(assuming you got more than 1 file in the archive)

...so I think I may take your ini but turn off solid archives by default. You may want to turn it on again for a final rebuild or something similar....I will think about a good way of packer.ini profile switching...

[kazumi213]

Well, you can't work on single files in a solid archive, but I think I'm missing something: is there any problem in uncompressing the whole archive to a temp folder, applying fixes, then repacking all files?

Probably this has to do with the file-based vs set-based limitation you mentioned before. But all files are linked to an unique archive name (temp folder name), and all files within that folder can be covered with a mask for a 1 step packing/unpacking operation. Here is a suggestion. Please sorry because I'm not taking into account possible big incompatibilities with current CMPro. Note that all this stuff applies to 7z archives, which Scanner is able to "detect".

Usually users first perform a scan-only operation. This could internally buffer changes that involve several files/archives in a priority list fashion. I mean:

archive 1
file1
file2 <- requires renaming
file3 <- requires moving to archive2
(is missing file4)

archive2
(is missing file3)
file4 <-requires moving to archive1
file5
file6 <- requires renaming
(is missing file7)

archive3
file7 <- requires moving to archive2
file8

DAT contains required info for Scanner to know which archives are involved.

1. Rename required archives
2. Unneeded files in all archives are uncompressed to backup directory.
3. Unpacks archive1 to temp folder "archive1". All file changes involving this archive are to be applied now. Suggested order:
3a. Deletes file3.
3b. Copies file4 from backup folder.
3c. Rename required files.
3d. Repacks all temp folder contents.
4. Unpacks archive2...
4a. Deletes file4
·
·

Again, sorry if all these suggestions doesn't make sense when considering actual CMPro mechanics.

[Bogy]

You already thought right....this would mean immense recoding which I simply don't have the time for. Scanner's fix operations are done nearly directly. So e.g. when a file is removed and one is added, 2 operations are done. Usually fixes of the same type are done in one go (e.g. all renames, all removes) but a mixture is done fix by fix.

Of course I could modify the current 7z/rar RemoveFileFromArchive and AddFilesToArchive functions to always do a complete decompress to temp, remove/add file, compress full archive and replace old one..

But to be honest...I think this is a waste of time gaining a minimum amount of space just to stay with solid archives. So most likely the next default packer.ini will disable solid archives.

But maybe this always full decompress/recompress operation can be enabled optionally for some power users (which should get a bigger hd imo)...but then they have to be aware of the possible immense long fixing time.