The model is public. Download the exact pack that ships in the app from HuggingFace and run it yourself with llama.cpp: weights, vision projector, and the decoding grammar included, with a quickstart in the model card.
Skin photos are about as private as personal data gets. If you're building an app that looks at them every week to track changes over time, "we upload your photos to our API" is not a privacy policy, it's a dealbreaker. So Sunny had one non-negotiable constraint from day one: the model runs on the phone, and photos never leave it.
One more constraint, just as hard: Sunny is not a diagnostic tool. It describes what a lesion looks like (type, colour, symmetry, borders, texture) so you can compare this month's photo against last month's. It must never assert a diagnosis, a benign/malignant verdict, or a risk score. That's not a disclaimer bolted on at the end; as you'll see, it's enforced at the decoding level.
Here's what that looks like in practice. A lesion photo goes in; a fixed six-field description comes out, always ending with the same non-diagnosis line:
The dead ends that shaped the design
The obvious first move was a big vision-language model. I fine-tuned one: a Gemma E4B pack that weighed 5.86 GB quantized. It worked, and it was completely unshippable. Nobody downloads a 6 GB app for skin tracking, and mid-range phones can't hold it in memory anyway.
The opposite extreme was tiny classifiers: a 6 MB MobileNetV3 and a 111 MB ConvNeXt-Tiny producing the schema fields directly. They were fast, but they plateaued in the mid-60s on exact-match field accuracy, the measured gain of the bigger one over the smaller was within noise, and a bank of classifier heads can't write the one thing users actually read: a coherent summary in plain language.
So the answer wasn't "bigger" or "smaller." It was a small language model fine-tuned into a specialist.
And fine-tuning is doing the heavy lifting, not the base model. Prompted zero-shot, the base VLM already produces the six-field format, but with essentially zero field accuracy and the safety line present only 7% of the time. After the LoRA fine-tune, every field jumps and the safety line becomes universal (colour is scored by Jaccard, an overlap score between the predicted and true colour sets):
The recipe
The base is SmolVLM-500M-Instruct, a 500M-parameter vision-language model. The pipeline:
- Data: PAD-UFES-20, a public dataset of 2,298 smartphone photos of skin lesions (real phone photos, not dermatoscope images, which is exactly the distribution the app sees). 2,292 survived preprocessing; split into 2,015 training and 277 validation images, grouped by lesion so no lesion appears in both sides.
- Fine-tune: three epochs of LoRA, a technique that trains a small add-on to the model instead of all its weights (r=16, alpha=16, lr 2e-4, effective batch 16), tracked in Weights & Biases. The target output is a fixed six-field schema: Lesion Type, Colour, Symmetry, Borders, Texture, Summary.
- Shrink: merge the adapter, then quantize, which means storing the weights in fewer bits: Q4_K_M for the language model, Q8_0 for the vision projector, packaged as GGUF. The whole phone pack is 412 MB: a 303 MB text model plus a 109 MB projector. That's 93% smaller than the E4B pack it replaced.
- Run: llama.cpp on Android behind a stable native interface (JNI). Google Play installs the model pack with the app; every artifact is verified against a pinned size and SHA-256 before it's copied into private storage. No sneaky post-install download, no silent cloud fallback. Cloud analysis exists but only fires on explicit, timestamped user consent, because that's the one case where a photo leaves the device.
The part I'd argue matters most: constrained decoding
A fine-tuned model is still a language model, and language models freelance. The fix is a GBNF grammar, a formal ruleset the decoder must follow, like a form with fixed boxes: the model is mechanically unable to emit anything except the six-field schema, capped at 160 tokens, ending with the exact non-diagnosis line.
On all 277 held-out validation images this gave 100% format compliance and 100% safety compliance. Not "we prompted it nicely and it usually behaves." The token sampler cannot produce a diagnosis, the same way a JSON parser cannot accept malformed JSON.
To be precise about what that claims and doesn't: it's a guarantee about output structure and safety, not about descriptive accuracy. Whether the description is dermatologically faithful still needs independent clinician-labelled validation, along with skin-tone and device coverage, on-device thermals and battery, and everything else on the release-gate list. Sunny is honest about being pre-clinical.
Why I think small models are changing the game
The lesson from Sunny generalizes well beyond skin tracking:
- Specialization beats scale for narrow jobs. A 500M model fine-tuned on 2,000 in-distribution images produces better structured output for this task than prompting a frontier model, and it does it in airplane mode.
- Capability per byte is the metric that matters on-device. 5.86 GB to 412 MB isn't an optimization, it's the difference between a product that can exist and one that can't.
- Grammar-constrained decoding turns an SLM into a reliable component. The reason people distrust LLMs in products is unpredictable output. Constrain the decoder and a small model becomes something you can build an interface on top of, with compliance you can measure at 100% instead of "usually."
- On-device flips the privacy default. When inference is local, "your data never leaves your phone" stops being a promise and becomes an architecture. Zero marginal inference cost is a nice side effect: no API bill scaling with users.
The frontier-model race is real, but there's a quieter shift underneath it: for a huge class of narrow, structured, privacy-sensitive tasks, a fine-tuned sub-1B model with constrained decoding is already the right engineering answer. Sunny is my existence proof.
Sunny lives at sunny.adityaps.work, and the model is on HuggingFace for you to download and experiment with — just remember the grammar file is part of the deal, not an accessory. If you're building on-device ML and want to compare notes: adipras1407@gmail.com.