A fluent answer is not a finished reading.
What a small OCR study taught us about target selection, misleading perfect scores and the token limit that cut off a correct answer.
By Joseph Ayanda. AI-assisted development and evaluation on AMD Instinct MI300X VF, using ROCm and native Qwen vision-language models.
Matched baseline
31 / 33
Qwen3-VL-4B on the known-photo slice. Not an independent benchmark.
Completion fix
131 tokens
Required for one long sign. The old 96-token ceiling cut it off.
Measured memory
9,094 MB
Largest of 72 driver samples, not a continuous exact peak.
1. Define success before choosing the model
The task is not to write a plausible caption. It is to return the intended plate identifier or road-sign text. Reading a registration sticker instead of the identifier is a failure, even when every sticker character is correct. So is adding a nearby sign, dropping a short word or returning a convincing prefix of an unfinished answer.
Reference answers were stored separately from inference inputs. The input manifest contained only image identifiers, paths and hashes. Where a character or the intended target could not be resolved confidently, the example remained an unscored diagnostic instead of receiving a convenient reference after prediction.
A specialist pipeline reached 24/24 on its original development photographs. That number did not survive ordinary input changes: it scored 22/24 after a one-megapixel cap, and 19/24 under both JPEG-quality-35 compression and half-width/half-height resizing. The useful result was the failure pattern, not the original perfect score.
2. Measure the replacement, not its reputation
The same inputs and task prompt were passed to native Qwen3-VL-4B and Qwen3.5-4B on the allocated AMD GPU. The newer model frequently included state banners, secondary signs and background text.
| Evaluation slice | Qwen3-VL-4B | Qwen3.5-4B |
|---|---|---|
| 33 known photographs | 31 / 33 | 23 / 33 |
| Seven additional scored photographs | 6 / 7 | 7 / 7 |
| JPEG quality 35 variants | 23 / 24 | 19 / 24 |
| Half-resolution variants | 22 / 24 | 19 / 24 |
A separate localization-and-crop experiment processed 91 inputs. It rescued a missing plate prefix, but turned a correct zero into O elsewhere and dropped a short sign word. One rescue against two regressions was not an improvement. The earlier three-disagreement contrastive reranking experiment also corrected nothing. Neither became the default.
Practical rule: retain the whole-image baseline, count rescues and regressions separately, and promote an extra inference stage only after it earns its cost.
3. Fix delivery before buying more capacity
A new long warning sign revealed a different failure: the 4B model was reading correctly, but the 96-token cap stopped generation mid-sentence. The 8B model under the same cap was truncated too. Increasing the 4B allowance to 512 produced the complete 131-token answer.
| Configuration | Five scored references | Incomplete / all 11 inputs |
|---|---|---|
| 4B, 96-token cap | 4 / 5 | 2 / 11 |
| 8B, 96-token cap | 4 / 5 | 2 / 11 |
| 4B, 512-token cap + EOS check | 5 / 5 | 0 / 11 |
The six ambiguous examples stayed unscored. A rerun of all 91 earlier inputs produced exactly the same normalized answers as before. This fixes a delivery failure; it does not establish broad recognition accuracy or eliminate the remaining character ambiguities.
Try the completion rule
This is a local logic demonstrator based on the recorded token lengths. It does not invoke a model or fabricate a live GPU result.
ids = model.generate(**batch, max_new_tokens=512,
do_sample=False, max_time=23.0)
suffix = ids[:, batch.input_ids.shape[1]:]
if int(suffix[0, -1]) not in eos_ids:
raise TimeoutError("Incomplete transcription")
# Only a completed, on-time reading may be committed.max_new_tokens is a ceiling, not a target length. Short answers still stop at EOS. The Transformers time limit is cooperative, so a final deployment also needs a separate hard-deadline supervisor and atomic output commits. See the official generation documentation.
4. Measure the deployed resource, not just the tensor allocator
The selected 4B/512 policy was repeated on the 11-image slice while amd-smi sampled driver memory approximately every half second. All 72 samples were valid, the largest observed used-VRAM value was 9,094 MB in the tool's reported units, and all outputs matched the previous run.
This is a sampled high-water mark. It is not an instantaneous peak, nor proof that the final container passes the evaluator's memory and invocation rules. The cloud experiment used Torch 2.13.0+rocm10.0.0, Transformers 5.3.0 and an AMD Instinct MI300X VF. The AMD SMI CLI reference describes the telemetry tool.
Three completed engineering milestones
Revision-locked Qwen3-VL inference on real MI300X VF hardware, with exact-output, latency and input-hash records. No API response mocks were used for the neural results.
Seven comparison jobs covered 186 model-image evaluations, including overlapping compatibility inputs. Failed alternatives and paired degradation results were retained rather than hidden.
A token-budget ablation isolated truncation; an EOS gate prevents incomplete output from being labelled successful. The updated policy was regression-tested and rerun under driver-memory sampling.
Reproduce the guard, inspect the evidence
The downloadable adapter keeps the measured prompt and pinned 4B model revision. Its completion and snapshot-integrity tests run without a GPU. The adapter is an extracted deployment component, not the complete challenge container; its final warm-worker GPU integration has not been validated.
cd source
python -m pip install -r requirements-test.txt
python -m pytest -q tests
# CPU guard tests only. These do not measure OCR accuracy.Next: validate the complete container and command lifecycle, then evaluate a frozen policy on a larger, predeclared source-separated set. Public-photo pretraining overlap is unknown. Repeated variants must not inflate the number of independent scenes. No technical leaderboard rank is claimed here.