Put a short legal question to one small AI model, and on LegalBench, a test run by Vals AI, it scores 82%, against 89% for the best model tested. Give the same model a long research question on Vals AI’s Tax Agent Bench, and it scores 31%, against 52% for the best model tested there.
A model is the program that reads what you type and writes the reply. This one is Qwen 3.8 27B, an open model, meaning anyone may download it and run it on a machine that can hold it. The two tests differ in subject, tools and marking as well as length. So they show a different gap on two different tests. They do not show why, or where it would sit on your work.
A firm that runs a model on its own premises usually wants confidentiality: no outside AI supplier is handed the client’s papers. That firm still has to decide which work goes to the machine, and no test can decide for it. For each job, ask what a wrong answer would cost, who would catch it and how long that takes, then budget for the checking. Local Assistant provides private, local AI for business, so weigh this article accordingly.
LegalBench leans towards American law, the tax test is mainly US corporate tax, and no test built on UK tax was found. This is one model, and others of a similar size, older models among them, score more than ten points lower on LegalBench. The scores may come from a supplier’s servers, and neither test page says whether a compressed copy, which needs less memory, was the one tested, so a firm’s own copy may score differently. The tax score was earned with supplied tools, including web search, which leaves the premises. These scores, read on 5 November 2026 and rounded, will move.
A partner might fairly say that much of a practice’s daily work is short and that a few marks on a narrow question will rarely matter. That may be so, though neither test measured it, and a test score averages its errors where a client file does not.
The case that model output needs checking is made in The expensive coin flip, published under the company’s Orlovs Consulting name. Try the machine on closed files where you already know the answers only if the client would reasonably expect that use and whoever advises you on data protection agrees. That tells you something neither test can, though it proves no job safe.
A score of 82% sounds like a pass. These tests only rank models, though, and the pass mark for your clients’ work is one your firm has to write itself.
Drafted with AI assistance and reviewed by Ilja Orlovs before publication.