CLIP ViT-B/32 image tower (int8) for the food photo recogniser ============================================================== Used by: /food-calorie-calculator ("From a photo"). Runs on the device with ONNX Runtime Web; the files are served from this site (copied to /public/clip by scripts/copy-clip.mjs) and cached in the browser's Cache Storage ("food-clip-v1"). The photo is never uploaded. Files ----- clip-vit-b32-image-int8.onnx.part0 .. part5 one ONNX model split into 6 byte ranges (each under 15 MB); the worker joins them. Total 89,202,676 bytes, SHA-256 411c032503fd7a622237febe4f4b63edcd5193360a1cddafb5df31dc875dc5ae Input "pixels" float32 [1, 3, 224, 224] (CLIP preprocessing), output "embedding" float32 [1, 512], L2-normalised. food-labels.json 265 photo labels (258 foods, 7 "not food") with their CLIP text embeddings: int8 + one scale per label, base64. Made by scripts/clip/embed_labels.py from scripts/clip/food-labels.txt; every food label points at a curated USDA food in scripts/food-selection.txt. Weights and licence ------------------- Model: CLIP ViT-B/32 by OpenAI, https://github.com/openai/CLIP (Radford et al., "Learning Transferable Visual Models From Natural Language Supervision", 2021). Licence: MIT, see LICENSE-clip-MIT.txt. The original checkpoint is ViT-B-32.pt, SHA-256 40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af (the hash in the download URL in clip/clip.py). Training data: OpenAI did not release it. The model card (openai/CLIP, model-card.md, "Data") says it is publicly available image-caption data crawled from the internet plus datasets such as YFCC100M, and that OpenAI does "not intend for this dataset to be used as the basis for any commercial or deployed model". The model card's "Out-of-Scope Use Cases" also says that any deployed use case is currently out of scope and that constrained uses need "thorough in-domain testing of the model with a specific, fixed class taxonomy". These are recommendations, not licence terms (the weights are MIT), but they are why this tool: uses a fixed, food-only label list; never labels people (a single "a person" label only says "not food"); shows the top guesses with their scores instead of one answer; and requires the user to pick the food and the portion. In-domain testing so far is small (see README, "Food calorie calculator"). Where the ONNX file came from (Hugging Face and openaipublic.azureedge.net were not reachable from the build environment): Release "models-v1" of the repository 05Narwhal/lumen-models on GitHub, file clip-vit-b32-image-fp16.onnx (176,214,230 bytes, SHA-256 0e08a2484ca1a1a5bc12cfd31f6cc6e8744c919bd6a132400d7cbcb8f44deeb1), described there as the image tower of OpenAI CLIP ViT-B/32 exported with OpenCLIP 2.26.1 (MIT), fp16 weights, from the checkpoint above. The matching text tower clip-vit-b32-text-fp16.onnx (127,465,976 bytes, SHA-256 b879d9e2ee5364962a5782cc7ddd5c49590ef5689f460835536487c4aac18025) and clip-tokenizer.json (SHA-256 b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842, CLIP's BPE tokenizer) from the same release were used offline only, to embed the labels. The SHA-256 values match the ones that repository publishes. Check that the weights are CLIP ViT-B/32: the example in the openai/CLIP README (CLIP.png against "a diagram", "a dog", "a cat"; README: 0.9928 / 0.0042 / 0.0030, computed on a GPU in float16) gives 0.9963 / 0.0027 / 0.0010 with these towers in float32 (same ranking, same order of magnitude). Modifications (scripts/clip/convert.py; Python onnx 1.23, onnxruntime 1.30, numpy 2; reproducible, the output above is byte-identical on re-runs) ------------------------------------------------------------------------------------------------ 1. float16 -> float32 compute: float16 initializers and constants become float32 and Cast(to=float16) becomes Cast(to=float32). (ONNX Runtime 1.30's CPU provider crashes on the float16 graph.) 2. Weight-only int8: the 50 MatMul/Gemm weights and the patch-embedding convolution are stored as int8 with one float32 scale per output channel, turned back into float32 by Cast + Mul (constant-folded by ONNX Runtime when the session starts, so inference is float32). 352 MB (float32) -> 89 MB. Image embeddings vs the float32 graph: cosine 0.998-0.999 on the test photos; the top-5 labels are the same. 3. Split into 6 byte ranges (`.partN`) so every file is well under Cloudflare Pages' 25 MiB limit.