Context is king in code audits. We train our own models to read 524,000 tokens at once.

Most code scanners rent a model from someone else. We train ours — extending RoPE past its native range and supervising on real call graphs — measured answering across 524,000 tokens, double the base model's native context window.

Almost every AI code scanner on the market is a prompt wrapped around somebody else's API. That is a reasonable way to build a product — unless the product's promise is that your code never leaves Europe, and unless the thing you are selling depends on reading more code at once than anyone else can. We train our own models, on our own hardware, and we can tell you what they score.

In our most recent evaluation, one of them answered questions correctly across 524,000 tokens of code — twice the context window its base model ships with. This is why that number matters more than it sounds.

Context is king in a source code audit

Ask a human auditor how they work and none of them will say "one file at a time". They build a mental model of the whole system first, then look for the places where the model breaks.

That is because real vulnerabilities live in the gaps between files. A validation routine that looks airtight until you find the caller that skips it. An authorization check enforced three modules away from where you would think to look. A sanitizer applied on one path into a sink and not the other. None of those are visible in the file where the bug eventually bites.

A scanner that reads a file at a time is structurally incapable of seeing them. It sees syntax, matches patterns, and produces noise — the argument we have made before, and the reason our scanner loads entire clusters of related code instead.

But "read the whole cluster" is only a promise you can keep if the model can hold the cluster. Real clusters in real repositories run to hundreds of thousands of tokens. The context window stops being a spec sheet number and becomes the ceiling on how good an audit can be.

Why context windows break: RoPE

Transformers do not have an inherent sense of position. Modern models get it from RoPE — rotary position embeddings — which encode where a token sits by rotating its query and key vectors by an angle proportional to its position. Relative distance falls out of the geometry, which is elegant and works extremely well inside the range the model was trained on.

Outside that range it degrades. Positions past the training length correspond to rotations the model has never encountered, attention gets unreliable, and retrieval — the ability to find the one function definition that matters, 400,000 tokens back — is the first thing to go.

The common fix is to rescale at inference time. YaRN and related position-interpolation methods squeeze longer sequences into the rotation range the model already knows. It is free, it needs no training, and every "1M context" repack you find on Hugging Face is doing some version of it.

It also has a real limitation: the model still never trained on those positions. You have changed the arithmetic, not the weights. Precision past the native window drops, which is exactly where a code audit needs it most — long-range retrieval is not an edge case for us, it is the core operation.

So we train through the extension instead of bolting it on at runtime. The model sees genuinely long sequences at the rescaled positions during training, which makes those positions real rather than interpolated. There is a nice piece of evidence for how delicate this is: BFloat16 breaks down RoPE in long-context training — at long range, the rounding error in bf16 is large enough to corrupt relative position on its own.

Not every part of a model is equally sensitive to this. Precise long-range retrieval concentrates in a subset of the layers, and those are where the position extension has to hold up under real load. That is where we spend the effort.

Call graphs are a free, verifiable answer key

Training for long context is not the hard part. Knowing whether the model understood anything — rather than guessing well — is.

Code solves this for us. A call graph is exact machine-checkable truth: function A either calls function B or it does not. So we can generate unlimited questions with known answers, at any context length, from real repositories, and grade every one automatically.

It is also a question that cannot be answered without doing both things we care about simultaneously. Deciding whether A calls B across half a million tokens requires understanding the code and retrieving two definitions that may sit at opposite ends of the context. One question, both skills, and an answer key that ships with the source.

Two details keep it honest. We strip identifying names — including comments and docstrings — so the model cannot shortcut to an answer via a suggestive function name. And we require it to cite the call site, so a correct answer has to be grounded rather than lucky.

What it scores

Held-out repositories the model had never seen, 774 questions:

At 524,000 — double the base model's native context window — it still answers four out of five correctly.

One result genuinely surprised us: accuracy barely moves with distance. Whether caller and callee sit adjacent or at opposite ends of half a million tokens, performance is about the same. What degrades first is not correctness but grounding — as context grows the model stays right while getting worse at pointing to exactly where it found the answer. A more interesting failure than an accuracy cliff, and the thing we are working on now.

We are being deliberately careful about the claim. These are our numbers on our evaluation; the controlled comparison against the untrained baseline publishes with the next milestone. We would rather show the curve now and the controlled result when it is actually finished.

Why this is unusual

When we surveyed the field before starting, we could not find a single published trained extension past 262K for this model family. The "1M context" checkpoints on Hugging Face are configuration repacks — runtime interpolation, no training, and no evaluations published alongside them. The one genuinely long-context model in this family is API-only with a closed recipe.

That is the gap we are working in, and it is why we publish the numbers rather than a specification.

Why we do it at all

Training your own models is slower and more expensive than calling an API. Three reasons it is worth it.

It is the only honest version of data sovereignty. A scanner that ships your code to a US API is not keeping it in Europe, whatever the marketing says — and as we covered on sending source code to US AI services, even the residency options that do exist are opt-in, cost more, and have documented gaps. Owning the model removes the question rather than managing it.

We can build for the job. No general-purpose API is optimized for holding a code cluster in context and reasoning about which function reaches which. We can train precisely for that, because it is what our product does.

Models small enough to own are enough. This one runs on hardware you can rack — no frontier-scale cluster, no per-token meter. Not a compromise but the bet the company is built on, and the reason our on-premise appliance is a real product rather than a brochure item.

We start from open weights and everything after that — the data, the supervision, the position extension — is ours, runs on our own hardware, and never leaves Europe. Which base, at what size, and how it is trained is something we are happy to walk through under NDA; it is not something we hand to competitors in a blog post.

What comes next

Larger models on the same approach, the controlled baseline comparison, and pushing usable context further — because the clusters we actually want to audit are bigger than 524,000 tokens, and we would rather grow the window than go back to reading files one at a time.

We will publish the numbers either way. If you want that depth of reading applied to your own codebase, that is what a Codelight audit is.

Topics: #ai #data-sovereignty #benchmarks