Model2026-08-03

Putting Chain-of-Thought Inside the Schema on a Small Language Model

#slm#qwen3#structured-output#constrained-decoding#chain-of-thought

A PropTech workflow automation project we worked on recently turned up a problem worth writing down. Everything from lease contracts to rent collection to facility tickets was already automated inside the app, but relatively few users knew how to reach the right feature at the right moment. The bottleneck was not the automation logic; it was the path to that logic. So we added a helper agent: say what you want in plain language, and it routes you to the feature and fills in the arguments.

The constraints were fixed up front. The agent could not call an external LLM provider's API, and it had to run on a small language model (SLM) installed on the same machine as the app. We picked Qwen3 1.7B. And in exactly that combination — a hybrid reasoning model plus constrained decoding — we hit a point where the two mechanisms collide head on. This is an account of why they collide, how we resolved it, and the design principles that came out of it.

Why a local SLM

Ruling out an external API was not a matter of performance; it followed from the customer's on-premise requirements.

Data boundary — the helper agent's input carries tenant details, contract amounts and arrears history along with the current screen context. The moment that data rides out of the customer's premises inside a prompt, it stops being a performance discussion and becomes a compliance one.
Cost structure — a helper is not an occasional feature; it is a UI element that sits there all day. Under per-call billing, always-on means an unpredictable variable cost, and that does not fit the pricing structure of a product delivered as an on-premise license.
Availability — a flaky office link or a provider incident must not take down the app's primary guidance feature.

That left us with little alternative but to evaluate a 1.7B-class Qwen3 model shipped alongside the app, and it is from here on that the substance of this article begins. A 1.7B model has to be handled differently from a frontier model — not because it fails to understand instructions, but because it fails to follow them consistently. A model that produces the right shape 97 times out of 100 is untrustworthy precisely because the other three are unpredictable. So we needed two corrective mechanisms: one to force the shape of the output, and one to raise the quality of the judgment behind it.

The output is an action record, not a sentence

What the helper agent ultimately has to produce is not prose to show the user. It is a structured action record: which screen to move to, which filter to apply, which draft to generate with which arguments. That record is translated directly into an internal API call.

Generating free text and pulling it apart with regexes is therefore not a good fit here, because the problem is not the failure rate itself but the shape of the failure distribution. A parser that breaks rarely, and in ways that are hard to reproduce, is the worst kind of defect to own in production. So we went with constrained decoding, which compiles the JSON schema into a grammar (GBNF and friends) and then masks the logits of any token the grammar disallows at every decode step. The important part is that this is prevention, not validation after the fact. A token that violates the grammar is not merely improbable — it cannot be selected at all. The parse failure rate does not drop; it goes to zero by construction. For putting a 1.7B model into real operation, that is effectively mandatory.

Turning on System 2: the hybrid reasoning model

The second corrective mechanism was to elicit reasoning. Qwen3 is a hybrid reasoning model that carries both a thinking mode and a non-thinking mode in a single set of weights. You switch between them with the enable_thinking argument in the chat template, or with /think and /no_think in the prompt itself; in thinking mode the model emits a <think> … </think> span before the final answer. The recommended sampling settings differ by mode as well — for thinking mode, temperature 0.6, top-p 0.95 and top-k 20 are advised.

Routing for this agent was not simple classification. An utterance like "show me what's overdue from last month" only lands on the right action once you weigh the current screen state, the interpretation of the time expression, and the fine distinctions between neighboring features. It looked like a problem where System 2 would earn its keep, so we enabled thinking mode — and that is where the trouble started.

The collision: one token stream, two authorities

The symptoms were scattered and hard to read at first. The cause, once found, was unambiguous, and in hindsight inevitable.

The state machine behind constrained decoding is live from the very first token of the assistant turn. If the root of the schema is an object, the only tokens allowed at that position are { and leading whitespace. <think> is not in the set the grammar admits, so it gets masked. In other words, the moment thinking mode is enabled and a grammar is attached, two different authorities are driving the same token stream. What you actually observe splits into two branches, depending on which one the runtime gives priority to.

When the grammar takes priority — with <think> masked, the model skips the reasoning process and goes straight to JSON. On the surface it looks like everything is working, but you are now sampling with thinking-mode parameters and getting System 1 output, and worse, the model has departed from the distribution it was trained on. Force an immediate answer from a model post-trained to reason first, and quality can land below what plain non-thinking mode would have given you. The classic symptom here is reasoning leakage: the deliberation has nowhere to go, so it spills into whichever string field the schema opens first.

{ "action": "First I need to work out what the user wants. The current screen is collections, and ..." }

When the reasoning parser takes priority — some runtimes interpose a reasoning parser and only apply the grammar after </think>. Formally the two now coexist, but at 1.7B a different problem appears, because a completely unconstrained region now exists in the reasoning span. Small models readily miss the closing tag and circle the same sentence, consuming tokens the whole time. The outcome is a response that either never reaches the JSON at all or is cut off partway through it, and latency variance becomes impossible to control in practice.

To put it plainly, a hybrid reasoning model assumes that reasoning lives out-of-band, in a span separated from the final answer, while constrained decoding requires the entire stream to be in-band and schema-conformant. The two premises cannot both hold. Which means nothing is resolved until one side is switched off.

Evaluating the workaround candidates

Define the problem that way and the options narrow to the following three.

Workaround	How it works	Why we passed
Split into two passes	The first call runs unconstrained in thinking mode; the second takes that result as context and decodes under the grammar	Prefill and decode both run twice, so latency roughly doubles. Not a fit for on-premise conditions, where the model shares a fixed compute budget with the app itself.
Parser-aware grammar	Activate the grammar only after `</think>`	Ties you to a runtime-specific feature, and the portability risk is high for a product that has to sit on whatever inference server the customer already runs. It also fails to address the underlying issue that reasoning length remains unbounded.
Repair and retry	Detect malformed output and try again	Makes latency worse without addressing the root cause. There is no guarantee the retry succeeds either.

Since each of these only partially manages the collision and none of them amount to a real fix, we had no choice but to look for another approach.

The fix: move the reasoning inside the schema

The decision itself was simple. Set enable_thinking: false to disable thinking mode, and fold the reasoning process into the output schema instead. This amounts not to giving up System 2 but to relocating where the reasoning happens. The justification for trying it lies in a property of autoregressive decoding: the model generates tokens left to right, and tokens produced for an earlier field become part of the context when a later field is decoded. If the fields holding the reasoning come before the fields holding the answer, the causal structure is identical to a <think> span. The idea rests on the observation that what makes chain-of-thought work is not the tag itself but the order things are arranged in.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["user_goal", "evidence", "rejected", "action", "params", "confidence"],
  "properties": {
    "user_goal":  { "type": "string", "maxLength": 120 },
    "evidence":   { "type": "array", "maxItems": 3,
                    "items": { "type": "string", "maxLength": 80 } },
    "rejected":   { "type": "string", "maxLength": 120 },
    "action":     { "enum": ["search_listing", "draft_contract",
                             "list_overdue_rent", "export_report",
                             "ask_clarification"] },
    "params":     { "type": "object" },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
  }
}

The first three fields are the reasoning span; the last three are the answer. Real output looks like this.

{
  "user_goal": "See which tenants did not pay last month's rent",
  "evidence": ["Current screen: Leasing > Collections", "'last month' means the whole of the previous month"],
  "rejected": "export_report generates a file rather than showing a list, so it does not match",
  "action": "list_overdue_rent",
  "params": { "period": "2026-07", "status": "overdue" },
  "confidence": 0.82
}

The grammar applies unbroken from the first token to the last, and the model still states the goal, gathers evidence and rules out candidates before committing to an answer. The two mechanisms that had been in conflict now coexist inside a single stream.

Four principles for designing the schema

In-schema CoT looks like adding a few fields, but in practice you only get the intended effect if you observe the following principles.

1. Confirm that field order can be pinned. Key order in a JSON object carries no meaning in the specification. The entire premise of in-schema CoT, however, is order. You have to confirm that whatever compiles your schema into a grammar enforces the declared property sequence. If the implementation permits arbitrary key order, the model can emit action first, and at that instant your reasoning fields degrade into after-the-fact justification. The check is easy: see whether output that puts the answer field first is still grammatically valid.

2. Control the allowance for reasoning time. maxLength and maxItems are not just guardrails here; they supply the one thing thinking mode never could — an upper bound on reasoning length. Generated token count is the dominant term in on-premise latency, and this design moves it out of the category of uncontrolled variables and into a range you set by design. At 30 tokens per second, 400 tokens of free-running deliberation is 13 seconds; reasoning capped at 80 tokens comes in under three.

3. Avoid free-form prose. Rather than one reasoning field that absorbs everything, it is better to decompose it into slots with clear jobs: the goal, the evidence, the reason for rejection. On small models, structure substitutes for capability to a meaningful degree, because filling in blanks is easier than writing on a blank page and the slot names themselves tell the model what it is supposed to check. A disconfirming slot like rejected is particularly effective at heading off premature conclusions.

4. Preserve the reasoning fields in your logs. The action record handed to the app has the reasoning fields removed. Throwing them away entirely would be a waste, though. They are observability data that tells you what the model was reasoning from when a routing decision went wrong, and they are raw material for evaluation sets and prompt revisions later. A <think> block gives you the same text, but only as an unstructured string that has to survive a parser, and it is never aligned with the action record itself.

What we gained, and what we gave up

To be straight about it, there is a cost. In-schema CoT does not use the model's own post-trained reasoning distribution. The slots we designed are a human-authored frame, not a trajectory the model learned. For tasks that need long search — multi-step math, involved code reasoning — genuine thinking mode still wins.

Our task was not that kind of task. It was routing over a finite action set and extracting arguments, and what it needed was not deep search but a checklist nothing falls off of. For problems in that shape, we confirmed that slotted reasoning was in fact more stable than free-running reasoning. We applied mitigations alongside it: naming the slots in vocabulary the model is comfortable reasoning in, and placing one or two examples in the system prompt that are exactly isomorphic to the schema.

Be careful not to fall into over-structuring, though, because every additional slot costs tokens and raises the risk that the reasoning becomes ritual rather than real. Adding one should require evidence that it actually reduces wrong answers, and keeping the schema stable matters in practice as well — a fixed schema is what lets you cache the compiled grammar and the prompt prefix, which cuts response latency and improves the experience accordingly.

Closing

Thinking mode on a hybrid reasoning model is an excellent feature, but if the output structure it presupposes conflicts with another constraint in your system, enabling it unconditionally cannot be the whole answer. The key point is that what we needed was never the <think> tag itself but reasoning that arrives before the answer, and the schema alone turned out to be enough to produce it. If you are building an on-premise or on-device execution environment for AI agents on an SLM, we hope the techniques and approaches set out here prove useful in your own implementation.

← Blog