Multi Checksum Script for Linux and BSD (bash)

General No-Intro related discussions.
Post Reply
User avatar
Madeline
Datter
Posts: 87
Joined: 29 Oct 2020 00:28

Multi Checksum Script for Linux and BSD (bash)

Post by Madeline »

Hi all,

I've created simple scripts to collect checksums for Unix operating system like linux or BSD.
I called it "multisum" but you can save the script with the name you like better. Just save them and place it anywhere in your $PATH.
The script uses an array (I know space seperated variables also works but I was too lazy and this was the first approach that came into my mind). So if you want to use something else than bash just modify the loop and the array variable.

Features:
* Display all checksums formatted uppercase
* display the complete size of the file in bytes

Requirements:
The only program you need to install beforehand is cksfv
Also check if md5, sha1 and sha256 for BSD and md5sum, sha1sum and sha256sum are installed. But I've never faced a unix system without them in the base installation.

Usage:

Code: Select all

multisum <FILENAME>
Licence: WTFPL

Unfortunately linux and BSD do not use the same checksum tool implementations so do not work exactly the same.

So I have two scripts (one for linux and one for [free]BSD).
linux:

Code: Select all

#!/bin/bash
CHECKSUMS=("cksfv" "md5sum" "sha1sum" "sha256sum");
for f in "${@}";
do
    echo "${f}:";
    for i in "${CHECKSUMS[@]}";
    do  
        if [ "${i}" == "cksfv" ]
        then
            printf "crc32: ";
            "${i}" "${f}" | grep -v "^;" | awk '{print toupper($NF)}';
        else
            printf "${i}: ";
            "${i}" "${f}" | awk '{print toupper($1)}';
        fi
    done;
    printf "Size: ";
    ls -l "${f}" | awk '{ print $5 }';
done;
BSD:

Code: Select all

#!/usr/local/bin/bash
CHECKSUMS=("cksfv" "md5" "sha1" "sha256");
for f in "${@}";
do  
    echo "${f}:";
    for i in "${CHECKSUMS[@]}";
    do  
        if [ "${i}" == "cksfv" ]
        then
            printf "crc32: ";
            "${i}" "${f}" | grep -v "^;" | awk '{print toupper($NF)}'
        else
            printf "${i}: ";
            "${i}" -q "${f}" | awk '{print toupper($1)}';
        fi
    done;
    printf "Size: ";
    ls -l "${f}" | awk '{ print $5 }';
done;
I hope this helps you people in any way.
User avatar
Macarro
Posts: 29
Joined: 06 Jun 2008 18:55

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Macarro »

Excellent!!!

I made a couple of visual modifications in the Linux version so information is a bit more organised (for my taste). Sample output:

Code: Select all

plots_2020.png:
  crc32    : 21F53095
  md5sum   : 8D290FCD79D3F63AC588FE2BC8E2292B
  sha1sum  : 74CE5D6D81797C6ADDFD2086305BF3BD3D1B97CB
  sha256sum: 936B318B07AACA4BB18725DDD37C561CAAA863D9F0F8D9DBA6568764A8FA97B4
  size     : 49838
And here is the modified code:

Code: Select all

#!/bin/bash
CHECKSUMS=("cksfv" "md5sum" "sha1sum" "sha256sum");
for f in "${@}";
do
    echo "${f}:";
    for i in "${CHECKSUMS[@]}";
    do  
        if [ "${i}" == "cksfv" ]
        then
            printf "  crc32    : ";
            "${i}" "${f}" | grep -v "^;" | awk '{print toupper($NF)}';
        else
            printf "  %-9s: " "$i";
            "${i}" "${f}" | awk '{print toupper($1)}';
        fi
    done;
    printf "  size     : ";
    ls -l "${f}" | awk '{ print $5 }';
done;
User avatar
Madeline
Datter
Posts: 87
Joined: 29 Oct 2020 00:28

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Madeline »

Ah pretty nice. Maybe i add some OS checks and unify the script so it will work on Linux and BSD. And use sh instead of bash.
Thanks for your modifications.
Hiccup
Datter
Posts: 1033
Joined: 09 Oct 2015 11:29

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Hiccup »

Here's another version

Code: Select all

#!/usr/bin/env bash
rhash -CMH --sha256 -p "%f:\n  size     : %s\n  crc32    : %C\n  md5      : %M\n  sha1     : %H\n  sha256   : %{Sha-256}\n" "$*"
Requires the rhash tool, but its a single line and is quicker because it does all the checksums in one read of the file. I also put the size first and made it so the "field" names just used the hash name rather than the utility names (since those aren't being used in this script anyway). It might be possible to reimpliment this using the built-in utilities, although it may not be as fast as rhash.

Code: Select all

$ time multi-checksum OOT.z64 
OOT.z64:
  crc32    : 6C348AA8
  md5sum   : 21F7B4A4FF463464BFC23498C1AB9DA1
  sha1sum  : 70537A3144C8813B115252C40065C117CB139DCD
  sha256sum: 68BBDD74169510D128E193B93047CAD9197E689C33DBBD52BA7414F5B2FFB1CA
  size     : 33554432

real	0m0.433s
user	0m0.400s
sys	0m0.045s
$ time rhash -CMH --sha256 -p "%f:\n  size     : %s\n  crc32    : %C\n  md5      : %M\n  sha1     : %H\n  sha256   : %{Sha-256}\n" OOT.z64
OOT.z64:
  size     : 33554432
  crc32    : 6C348AA8
  md5      : 21F7B4A4FF463464BFC23498C1AB9DA1
  sha1     : 70537A3144C8813B115252C40065C117CB139DCD
  sha256   : 68BBDD74169510D128E193B93047CAD9197E689C33DBBD52BA7414F5B2FFB1CA

real	0m0.243s
user	0m0.227s
sys	0m0.016s
User avatar
Madeline
Datter
Posts: 87
Joined: 29 Oct 2020 00:28

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Madeline »

Nice. I didn't know about rhash. This looks good as it's faster than calling 4 individual commands.
Let's see if FreeBSD has it packaged.
Thank you!

/e: Works like a charm

/e2: Edited the script to let it work with multiple files

Code: Select all

for i in "$@"
do
    rhash -CMH --sha256 -p "%f:\n  size     : %s\n  crc32    : %C\n  md5      : %M\n  sha1     : %H\n  sha256   : %{Sha-256}\n" "$i"
done
Hiccup
Datter
Posts: 1033
Joined: 09 Oct 2015 11:29

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Hiccup »

I've started a python version (using the rhash bindings, so the hashing itself should be just as fast, but my printing code is probably not that efficient).
https://github.com/mariomadproductions/multisum

Advantages:
can output in plain format and datfile xml format
should be easier to maintain
can be (and probably is) multi-platform

Edit: seems like this is producing incorrect hashes for everything but crc32. I'll try using the stdlib hashing stuff instead of the rhash python bindings (which seem a bit outdated).
Icyelut
Posts: 151
Joined: 30 Apr 2020 03:02

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Icyelut »

I'm currently using this python code to hash files with only standard python libraries:

Code: Select all

from zlib import crc32
from hashlib import sha1, sha256, sha512, md5, sha3_512

def hash_file(full_file_path):
    calculated_crc32 = 0
    calculated_md5 = md5()
    calculated_sha1 = sha1()
    calculated_sha256 = sha256()
    calculated_sha512 = sha512()
    calculated_sha3_512 = sha3_512()
    with open(full_file_path, "rb") as f:
        while True:
            data = f.read(65536)
            if not data:
                break
            calculated_crc32 = crc32(data, calculated_crc32)
            calculated_md5.update(data)
            calculated_sha1.update(data)
            calculated_sha256.update(data)
            calculated_sha512.update(data)
            calculated_sha3_512.update(data)

    output_crc32 = ("%08X" % (calculated_crc32 & 0xffffffff)).upper()
    output_md5 = calculated_md5.hexdigest().upper()
    output_sha1 = calculated_sha1.hexdigest().upper()
    output_sha256 = calculated_sha256.hexdigest().upper()
    output_sha512 = calculated_sha512.hexdigest().upper()
    output_sha3_512 = calculated_sha3_512.hexdigest().upper()
    
    return output_crc32, output_md5, output_sha1, output_sha256, output_sha512, output_sha3_512
Some refactoring to make it more pythonic would make it prettier, but I wanted to keep it simple.

EDIT: Updated CRC32 output to support all Python versions and platforms (https://stackoverflow.com/questions/300 ... ne-results)
Hiccup
Datter
Posts: 1033
Joined: 09 Oct 2015 11:29

Re: Multi Checksum Script for Linux and BSD (bash)

Post by Hiccup »

Thanks, thats useful. I've implemented that (with somes insignificant changes) to my script. Correct hashes are produced now.
Post Reply