Mubit's Banyan Sapling checks claims, says unsure 45% of the time

Mubit Labs released Banyan Sapling on Hugging Face this week and the launch thread called it perfect, which is exactly half true. The Apache 2.0 model is a 23B verification engine that reads a policy, evidence, and a claim and returns a verdict; on its own model card's 1,731-case evaluation, every one of the 779 calibrated answers was correct, but the model returned unsure on the other 952 cases.

Key facts

• The model card verifies Apache 2.0 weights on a Qwen3-30B-A3B base pruned to 23B stored parameters, MIT-bundled code, BF16 and FP32 adapters, and a 16,384-token input limit
• The card's eval table reports 94.22% raw accuracy over 1,731 cases, with 779 calibrated answers and zero false supported decisions
• The same table shows 952 cases returned unsure, a 45% answer rate, and 0.82% coverage on ContractNLI
• The core launch thread drew about 387 likes, 152 replies, and 203,000 views in its first day

What's driving the conversation

The founder's launch thread framed the result as "perfect on 779 of 1,731," and reply threads from followers including @gelorria split over whether that is a brag or a confession. One camp reads a 100%-precision calibrated verifier as the thing agent pipelines have been missing: a component that never falsely approves gets trusted with gates. The other camp does the division and asks what use a checker is that declines to answer more than half the time.

The substance

The model card discloses the trade-off the thread omitted. Banyan Sapling outputs supported, violation, missing, contradicted, or unsure, with calibrated probabilities, and its calibration is deliberately conservative: 779 of 779 calibrated answers correct, 952 of 1,731 cases returned unsure, and just 0.82% coverage on ContractNLI, where the domain gap is widest. Both framings in the discourse are accurate to a slice of that table. The precision claim is a vendor self-report from Mubit's own eval set, pending independent reproduction; the coverage number is the more honest headline and it comes from Mubit's own card.

Why builders are watching

The interesting signal is specialization. While October's flagship drops chase the frontier at 501B-816B parameters, this is the open-weights wave going the other way: a small, auditable, Apache 2.0 component, sized for one 48GB-class GPU, built to sit in an agent trust pipeline rather than to write code. The open-weight explainer framing holds, but the shape is new: purpose-built verifier rather than general assistant, closer kin to the small local models we covered in Liquid AI's LFM2.5-24B than to the frontier. Whether a 45% answer rate clears a production bar depends entirely on the pipeline's tolerance for unsure, and on whether independent evals reproduce the calibration. Watch for third-party benchmarks of the ContractNLI gap.