Is it possible to look into a safetensors file and grab its metadata?

[unphased]

I have obtained 50 or so safetensors models for stable diffusion and now i'd like to organize them based on their version (1.5 or 2.1). However I can't find any suitable APIs or any other way to open up these files. For example, https://huggingface.co/docs/safetensors/metadata_parsing shows a way to do it if i had HuggingFace URLs for the models, but I don't. I have them locally. So i just want to extract out the format. Is this possible? Surely it is possible.

[unphased]

Never mind, the header of the file is json so all I need to do now is reverse engineer where the length of the json is and then use it to read out the json.

[Narsil]

This should get you started: https://gist.github.com/Narsil/3edeec2669a5e94e4707aa0f901d2282

[unphased]

Thanks. I went the underengineered route.

sd_model_discriminator.sh

#!/bin/bash

FILE=$1
METADATA_DUMP_SCRIPT=~/util/safetensormetadump

COUNT_1024=$($METADATA_DUMP_SCRIPT $FILE | grep '^\s*1024' | wc -l)

if (( $COUNT_1024 > 50 )); then
    echo "guess v2: $FILE"
else
    echo "guess v1.5: $FILE"
fi

safetensormetadump:

#!/bin/bash

# Specify the file
FILE="$1"

# Extract the first 8 bytes and convert them to a decimal integer
HEADER_LENGTH=$(dd "if=$FILE" bs=1 count=8 2>/dev/null | od -An -vtu8)

# Extract the metadata, starting from the 9th byte
dd "if=$FILE" bs=1 skip=8 "count=$HEADER_LENGTH" 2>/dev/null | jq

• Extracts the metadata json from the safetensor
• sends it to jq to prettify it
• counts instances of some dimension of layers being 1024
• if greater than 50 instances, mark as v2.x, otherwise assume v1.5

works great

[julien-c]

nice job @unphased