Reinforce Labs
Generating high-quality supervised fine-tuning data for multimodal AI safety is one of those problems that sounds straightforward until you actually try to do it. You need image-prompt-response triples that teach a model when to comply, when to refuse, and how to navigate the vast grey area in between; across dozens of sensitive policy categories, with prompts that sound like real humans, not exam questions.
This post covers what actually worked, what didn't, focusing on the parts that matter for anyone building SFT pipelines.
The Problem
We needed to build a supervised fine-tuning dataset for a multimodal model that could evaluate image-text pairs against sets of safety policy categories like CBRN, self-harm, misinformation, etc. Each sample needed an image, a realistic user prompt grounded in that image, a model response calibrated to the right safety level, and structured metadata (sensitivity tier, response).
The dataset had to cover five sensitivity tiers, each mapped to a response strategy ranging from full compliance to hard refusal. We didn't get this taxonomy right upfront. It evolved through failures and rounds of generation that came back wrong, quality checks that revealed systematic issues, and painful iteration cycles that forced us to rethink how we were structuring the problem.
Prompt Design: Where the Real Engineering Lives
Your generation prompt will fight itself if it's too long
The prompt we used to generate user questions had everything in it: the full policy definition, image severity taxonomy, persona instructions, naturalness guidelines, formatting rules, few-shot examples. It hit ~3,000 tokens. The model followed the policy and taxonomy parts (which came first in the prompt) but by the time it reached the naturalness rules near the end, it had basically forgotten them. The generated questions were policy-accurate but sounded like a safety researcher wrote them.
We fixed this by breaking generation into stages instead of cramming everything into one mega-prompt. First pass: generate the question with persona and naturalness front and center, keeping the prompt short. Second pass: validate against policy and taxonomy separately. More API calls, but the quality difference was night and day. If your generation prompt is over ~1,500 tokens, the model will prioritize whatever's at the top and deprioritize whatever's at the bottom.
Write prompts that sound like actual users, not researchers
Even after fixing the prompt length issue, the first usable round of generated prompts still skewed academic.
We defined adversarial personas to fix this: a teenage boy who uses slang and typos, a pseudo-polite user masking intent, an edgelord testing boundaries, someone genuinely naive, and a sophisticated extraction expert. Rotating these personas across images produced prompts that actually stress-test the model the way real traffic would.
Caption-first prompting reduces hallucination
Before generating any Q&A pair, we had the model describe the image in detail first. This forces the model to commit to what's actually in the image before reasoning about it. Without this step, generated responses frequently mentioned objects or contexts that weren't there. Papers like ALLaVA found the same thing. Once the model has already written down what it sees, it hallucinates less in the follow-up.
Sensitivity is about output consequence, not topic domain
This was the biggest calibration issue across the entire project, and it kept coming back with human reviewers too. The instinct is to label everything in the CBRN category as HIGH sensitivity because the topic is scary. But if a user sees a photo of someone in a hazmat suit spraying crops and asks what kind of protective gear that is, the answer is basic agricultural PPE info printed on every pesticide label. That's LOW regardless of it being CBRN.
The rule we converged on: before assigning sensitivity, ask "if the model answered this poorly, what's the worst realistic outcome?" If the answer is "someone learns basic public information," it's LOW. If a bad response could provide actionable harm instructions, it's HIGH. Once we got reviewers applying this consistently, disagreement dropped from ~35% to under 10%.
Validation: The Checks That Actually Matter
Enforce a style guide automatically
We built an automated style evaluation step that checked every generated response against the project's style guide for tone, formatting, refusal phrasing, and verbosity. Without it, responses drifted across generation batches. One batch uses "I can't help with that," another says "I'm not able to assist," another gives a three-paragraph hedge before declining. An auto-fix step after the evaluator normalized these inconsistencies.
Use an LLM judge for quality scoring
Raw generation outputs are noisy. We added a scoring step where each sample was rated by a separate LLM judge on relevance, accuracy, policy alignment, and naturalness. Samples below threshold got discarded or regenerated. The tail of low-quality samples (responses that technically answer the question but miss the policy nuance) will degrade training if you don't gate them.
We also ran human review on samples to calibrate the judge itself: human reviewers went through the generated data, and their disagreement patterns taught us a lot about where our prompts and scoring rubric were off.
Pain Points
Image sourcing is harder than it sounds
Finding images that actually test the boundary between benign and harmful for each policy category took way more effort than expected. Generic stock photos don't create interesting safety scenarios. You need images where a reasonable person could construct both a benign and a harmful prompt. That's the dual-use tier, and it's where the model actually learns nuance. We ended up writing detailed search queries per category per tier, which itself required understanding what realistic adversarial usage looks like for each policy.
Checkpoint aggressively and keep a manifest
At scale, API timeouts and rate limits will kill your runs mid-flight. Checkpoint every 20 entries and track what's been processed in a manifest. We added a pre-flight sync that checks what's actually on disk vs. what the manifest says, which saved us multiple times.
Generate responses and rationales in the same call
We tried generating the response first, then the rationale in a separate pass. The rationale kept describing reasoning that didn't match the actual response. Single-call generation with chain-of-thought before the response fixed the misalignment.
We have more developments and blog posts coming. If this is something you're interested in, we'd love to hear from you. Feel free to reach out at contact@reinforcelabs.ai.