System One models are designed around a simple but powerful idea: instead of generating an answer as text, evaluate a predefined set of possible answers and return a decision distribution.
But there is no single architecture for building such a system.
In this post, I look at four open-source approaches that illustrate some of the most interesting architectural choices:
- Laya — an encoder-based approach that scores candidate options at dedicated
[MASK]positions. - Strands Decider — a decoder-only LLM whose language-model head is replaced by a learned pointer head that scores candidate options.
- SemIf / OpenJev — a decoder-only LLM that directly reuses the model’s native logits for predefined answer tokens, followed by calibration.
- Contrastive Language Models (CLMs) — a representation-based approach that learns to match a state representation against embeddings of candidate actions using contrastive learning.
These models are particularly interesting to compare because they arrive at a similar System One-style interface through fundamentally different architectures.
At a high level:
Laya: score options at marker positions
Strands: learn a pointer to score option representations
SemIf/OpenJev: read the LLM’s own answer-token logits
CLM: compare state and action embeddings
There is also a second dimension that is easy to overlook: how much the underlying language model is adapted for the decision task. Laya fully fine-tunes its encoder backbone; Strands Decider adapts its Qwen backbone with LoRA; CLM trains its representation model with a contrastive objective; while SemIf/OpenJev demonstrates how a frozen causal LLM can itself be repurposed as a decision engine.
So rather than treating “System One model” as a single architecture, this post uses these models to explore four different ways of turning language-model representations into decisions.
Laya
HuggingFace: https://huggingface.co/convaiinnovations/laya
Laya is an encoder-based System One model, or Jev-alike open-source decision model.
At first glance, Laya looks relatively simple. But there are several interesting components worth looking at in more detail:
- Backbone
- Option-marker
- Act/Escalation head
- Router
Which backbone does it use?
Laya uses encoder backbones such as ModernBERT-large for English and mmBERT-base for multilingual.
Unlike decoder-only LLMs, these are bidirectional encoders: each token can attend to tokens on both sides of it, allowing the model to build a representation using the full input context.
What is the Option-Marker Scorer?
The Option-Marker Scorer is the component that turns the model into a variable-choice decision model.
Each candidate option gets its own marker, represented by a [MASK] token. The model produces a score at each option marker, and those scores are then softmaxed across the options belonging to that particular question.
This is important because the answer space does not have to be fixed when the model is trained.
What is the Act/Escalate head?
The Act/Escalate head is another component of Laya’s decision head.
Its purpose is different from simply predicting which option is most likely. It estimates whether the decision is reliable enough to be automatically acted upon or should instead be escalated for human review.
Instead of relying solely on simple, rigid confidence thresholding rules (e.g., “if probability is greater than 0.8, act”), the Act/Escalate head learns a non-linear decision boundary by combining semantic textual context with statistical properties of the option distribution.
What is the Router?
Laya actually consists of multiple specialized checkpoints built around different backbone models.
The Router is a pre-inference dispatcher that determines which checkpoint should handle a request. The project recommends using the Router in production; it detects language/script characteristics and dispatches the request to the appropriate checkpoint.
The current family includes:
- English Base (
convaiinnovations/laya): Built on a ModernBERT-large backbone (395M backbone + 26M decision head = 421M total parameters). Optimized for English text classification, email triage, and guardrails with a 512-token context window. - Multilingual (
convaiinnovations/laya-multilingual): Built on an mmBERT-base backbone (322M total parameters). Supports over 100 languages and non-Latin scripts, runs ~2.2x faster, and handles longer context windows (up to 8,192 tokens). - Typed-Decisions Specialist (
convaiinnovations/laya-typed-decisions): Also built on ModernBERT-large, but specialized and fine-tuned for specific structured workflow benchmark domains.
Is Laya really for zero-shot use?
This is where comparisons with Jev become interesting.
Because System One models accept the answer space at inference time, it is tempting to assume that they automatically provide strong zero-shot decision-making.
But the architecture alone does not guarantee that.
The underlying backbone still matters enormously. A roughly 400M-parameter encoder should not automatically be expected to have the same broad semantic capability as a multi-billion-parameter LLM.
Laya itself is quite explicit about this limitation:
Laya is a fast base to specialise, not a zero-shot decision engine.
Strands Decider
Github repo: https://github.com/strands-labs/strands-decider
Strands Decider takes a noticeably different architectural approach from Laya.
Rather than using an encoder, it uses a decoder-only causal LLM: Qwen3.5–2B-Base.
In a causal decoder, each token can attend only to previous tokens, rather than the bidirectional attention used by an encoder.
There are three components worth looking at:
- Backbone / LLM torso
- Pointer head
- LoRA adaptation
What is the backbone?
Strands Decider uses Qwen3.5–2B-Base as its backbone, or what the project calls the LLM torso.
The idea is to retain the Transformer layers that provide the model’s language understanding capabilities, while removing the standard language-model head responsible for generating the next token.
The Strands team describes the 2B model as a useful trade-off: small enough for local experimentation while still large enough to provide meaningful semantic capabilities.
What is the pointer head?
The core idea of Strands Decider is that we take a pre-trained LLM torso (Qwen3.5–2B), and remove the LM head, taking away its ability to generate text. The LM head is replaced with a pointer head which scores the answers offered by the torso for each option. It does this by scoring the hidden state at each option position against the hidden state at the <answer> position. This head is pretty small, just over a million total parameters.When people refer to the LLM Torso (or “Backbone Torso”) in the context of a decision model or “decider” system, it refers to the main stack of Transformer layers acting as a high-capacity feature extractor, while stripping away or bypassing the standard text-generation head.
The pointer head is very small — just over one million parameters — compared with the roughly 1.9B-parameter torso.
What is LoRA used for?
The Qwen3.5–2B torso is adapted using a rank-16 LoRA adapter.
The important point is that Strands is not simply taking an off-the-shelf Qwen model and asking it to choose an answer.
The torso is adapted specifically for the decision-model task. The dataset is also available on Github: https://github.com/strands-labs/strands-decider/tree/main/data.
Is Strands Decider zero-shot?
Potentially, but with an important qualification.
The model can accept new questions and candidate options without creating a new classifier head for every task. That gives it a form of task-level zero-shot flexibility.
However, that does not mean that every arbitrary domain will work well without additional adaptation.
The underlying Qwen3.5–2B model is substantially larger than Laya’s ~400M-parameter encoder, and Strands additionally trains it with decision-oriented data.
So it is reasonable to expect stronger generalization in some settings — but zero-shot performance remains task-dependent.
This distinction is important when comparing System One models:
Variable answer spaces make a model architecturally adaptable; they do not automatically make it universally (well-performing) zero-shot.
Did Strands Decider also use calibration?
Yes — and this is an important part of the design.
Calibration is not merely an implied property of the pointer head.
The Strands training recipe has a separate calibration stage after training and before evaluation/serving. The documentation explicitly states that the published reference checkpoint is already calibrated, while other checkpoints should be calibrated before serving.
SemIf / OpenJev
Unlike Strands Decider, which adds a learned pointer head to score arbitrary option representations, SemIf/OpenJev directly repurposes the LLM’s own logits over predefined answer tokens, followed by calibration. It is conceptually similar to NVIDIA’s LogitsProcessorZoo multiple-choice approach.
Contrastive Language Models (CLMs)
Contrastive Language Models (CLMs) are a new class of System One model trained with a contrastive learning objective that connects states and actions. Unlike earlier approaches, CLM uses embedding-based similarity to match an input state with a set of candidate actions.
A typed question is a state plus a closed set of candidate actions (the options and their descriptions); a softmax over CLM’s scores is the answer distribution, and the same call ranks best-of-N trajectories, routes tools, shortlists retrieval pools and answers typed decisions with no per-task setup.
To achieve this, CLM first trains separate state and action encoders on a large-scale dataset using a contrastive InfoNCE objective. During training, the model learns to pull each state representation closer to the representation of the ground-truth action while pushing it away from the representations of alternative actions.
No comments:
Post a Comment