A Chatbot Just Out-Searched the Engineers: LLM-Falsifier Finds Bugs on Its First Try

Somewhere in a quiet corner of the software-verification world, a group of researchers decided to hand the hardest part of a safety check to a chatbot. The result is both delightfully absurd and quietly profound: a large language model — the same kind of thing that drafts emails and writes poetry — can, on its very first try, find the exact moment a simulated car crashes past its speed limit, a feat that is "essentially impossible" for a traditional numerical optimizer, which starts its search blind.

On six of the twenty-one benchmark safety specifications tested, the LLM nailed a falsifying input on the very first simulation. Not the second. Not the third. The first. That is not a lucky guess at a multiple-choice question; it is the outcome, the researchers argue (ArjomandBigdeli et al., 2026), of giving the model something conventional optimizers have never had: a name for the thing it is trying to break, and a moment in time where the failure matters most.

This is the story of LLM-Falsifier, a tool that uses a large language model to find errors in cyber-physical systems — the software-and-mechanics hybrids that control everything from automatic transmissions to aircraft. It is a story about why "reasoning" and "optimizing" might be the same act, and why the things that make language models feel like cheating at math may actually make them superb engineers.

The Science

Before we get to the chatbot, we need to understand the problem it is solving, because the problem is genuinely hard — and genuinely important.

Cyber-physical systems (CPS) are everywhere: the controller in your car's automatic transmission, the flight software on a drone, the brake-by-wire system in a modern vehicle. Each one is a blend of continuous physics (speed, temperature, torque) and discrete logic (shift gear, open valve). These systems are built with specifications — formal requirements about how they should behave. For example: "The vehicle speed must never exceed 120 during the first 20 seconds."

Engineers need to verify these specifications. But rigorous formal verification of such systems is often computationally undecidable — it simply cannot be done with mathematical certainty for most practical cases (Henzinger et al., 1995). So the practical alternative is falsification: instead of proving a system always works, you aggressively hunt for a single scenario where it fails. Find one counterexample, and you have your bug (Kapinski et al., 2016).

The trick is that falsification needs a language for "how badly" a specification was violated. Enter Signal Temporal Logic (STL), a logic developed by Maler and Nickovic (2004) for specifying and reasoning about real-time signals. STL can be interpreted with a quantitative semantics called the robustness degree (Donzé and Maler, 2010; Fainekos and Pappas, 2009). Think of robustness as a margin of safety: a positive value means the system satisfied the specification with room to spare; a negative value means it violated it. The formula looks like this:

$$\rho (\phi ,\mathbf{x},0)=\underset{(x_0,u)}{\min }\rho (\phi ,\mathcal{M}(x_0,u),0)$$

For the automatic transmission example — call it the AT benchmark — the specification is ${\phi }_{AT1}={\mathbf{G}}_{[0,20]}(\text{speed}\le 120)$, meaning "globally, during the first 20 seconds, the speed stays at or below 120." The robustness of a trajectory is simply:

$$\rho ({\phi }_{AT1},\mathbf{x},0)=120-\underset{t\in [0,20]}{\sup }\text{speed}(t)$$

That is, the margin by which the peak speed stays below 120. Positive means safe; negative means you just found a bug.

The catch: this robustness function is non-smooth and non-convex, full of nested min/max operations that make it a nightmare for classical calculus-based optimization. So traditional falsification tools treat it as a black box and throw generic search algorithms at it — surrogate-based methods, Bayesian optimization, Monte Carlo tree search, simulated annealing. These tools work, but they are sample-hungry. Every candidate input must be run through a full simulation of the dynamical system, and simulations are expensive.

Now comes the interesting idea. A large language model is, at its core, a next-token predictor — but it turns out that, coupled with iterative prompting, LLMs are also surprisingly effective derivative-free optimizers. This insight, called Optimization by PROmpting (OPRO), was introduced by Yang et al. (2024), who showed that an LLM given a natural-language description of an optimization problem plus a history of past solutions and their scores can iteratively propose better candidates.

ArjomandBigdeli, Zhou, and Bak — all at Stony Brook University — asked a simple but audacious question: what if we make the LLM itself the falsifier? Not as an assistant to a traditional optimizer, but as the actual search engine?

What They Found

The core of LLM-Falsifier is a closed-loop workflow shown in Figure 1. The STL specification and the system files (say, a Simulink model) feed a static model summarization step, which extracts the natural-language names of inputs and outputs — names like "throttle" and "brake" and "speed." These populate a "meta-prompt." The LLM proposes a candidate input, the simulator runs it, and an STL monitor computes the robustness value. If it's negative, you've found your counterexample and the loop stops. If not, the sample and its feedback are appended to the prompt history, and the loop repeats.

The key design choices are threefold. First, the LLM gets to see the semantic names of the variables — words like "throttle position" and "brake" — rather than anonymous slot numbers $x_1,x_2,\dots ,x_{10}$. Second, it gets to see the output trajectories: the actual shape of the vehicle speed over time from past simulations. Third, and most subtly, it gets the critical time — the exact moment at which the robustness value is most sensitive to changes in the signal.

This critical-time witness turns out to be the secret sauce. For the AT1 example, the robustness of a trajectory is $120-{\sup }_{t\in [0,20]}\text{speed}(t)$, where the supremum is attained at a specific time $t^{\ast }$. Including the output signal value at that exact time gives the LLM a surgical clue about which part of its input was responsible for the problem — a piece of causal information that a purely numerical optimizer, which just sees a scalar score, can never extract.

The results are striking. On the ARCH-COMP 2025 falsification benchmarks, LLM-Falsifier outperformed existing falsification tools — spanning the full range of optimization paradigms, from surrogate-based and Bayesian optimization to search-based testing — on 14 out of 21 specifications when measured by the number of simulations required to find a counterexample (the primary cost in falsification, since every sample requires an expensive simulation).

And on six specifications, it found a counterexample on the very first simulation. The researchers are careful to note that this is essentially impossible for a numerical optimizer, which has no information before its first sample.

The paper includes a careful ablation study isolating the contribution of each enhancement. Starting from a "minimal" prompt variant (MP1) that gives the LLM only the problem description and past score, and moving through progressively enriched variants (MP2–MP4) that add natural-language names, output trajectories, and critical-time witnesses, the researchers show each addition meaningfully improves sample efficiency.

Why This Changes Things

Let's pause to appreciate how odd — and how promising — this is. Traditional falsification treats the problem as pure numbers: a vector of control points in, a scalar robustness out. The optimizer has no idea it's dealing with a car, or a throttle, or a speed limit. It's blind arithmetic.

The LLM, by contrast, understands what it's manipulating. When it sees a past trajectory where the vehicle speed crept up to 119.6 at time 20 — a near miss, robustness +0.42 — it can reason, "Ah, the critical moment is at $t^{\ast }=20$ seconds. To push speed above 120, I should apply more throttle earlier and keep it there, so the peak lands inside the window." That is not pattern-matching on a loss landscape; that is a form of causal reasoning about a physical system, expressed in a language the model was trained to wield.

To make the contrast concrete, consider the running example from Figure 2. Two candidate inputs: one applies full throttle at the first three control points and none afterwards, peaking at 119.6 within the window — a near miss. The other applies full throttle at all seven control points, reaching 120.2 at $t=20$ — a counterexample with robustness $-0.21$. A numerical optimizer stumbles toward this through random sampling and gradient-free heuristics. The LLM, given the semantic names "throttle" and "brake" and a description of the speed bound, can propose the full-throttle solution almost immediately because it can map the natural-language goal — "make the car go faster than 120 within 20 seconds" — directly onto plausible input values.

This matters beyond the novelty of "AI does engineering." Falsification is a safety-critical task. The whole point is to find bugs before they manifest in the real world. The automatic transmission benchmark isn't an abstraction — it's a model of a real vehicle control system, and a specification like "speed must not exceed 120 in the first 20 seconds" is the kind of requirement that, if violated in a real car, could mean an accident. Sample efficiency isn't just an academic metric; when each simulation is minutes of compute time, the difference between finding a bug on the first sample and finding it on the thousandth is the difference between a feasible safety check and an impractical one.

There's also a broader philosophical point lurking here. The researchers frame their work as connecting robustness optimization with LLM-based optimization — two literatures that rarely speak to each other. But what their results suggest is that the boundary between "numerical optimization" and "semantic reasoning" is more porous than engineers have assumed. When you can describe what you're optimizing in words — not just as a vector and a scalar — the optimizer can leverage a vast reserve of world knowledge about how cars, brakes, throttles, and speed limits actually behave. That is a genuinely new capability for the falsification toolbox.

The paper is careful about its limits. The LLM doesn't provide guarantees. It's probabilistic in a way that classical verification methods aren't. And the researchers acknowledge that different specifications and models may not always yield such dramatic early successes. But the direction is clear: the semantic richness that classical optimizers throw away is not noise — it's signal.

What's Next

The most exciting thing about LLM-Falsifier is that it's a first step, and its authors are explicit about the open questions.

The critical-time witness — the key idea that gives the LLM a pointer to the moment of failure — depends on the specific min/max-based robustness semantics used in this work. Stony Brook's team notes that alternative robustness formulations exist, such as arithmetic-geometric mean robustness (Mehdipour et al., 2019) or smooth cumulative semantics (Haghighi et al., 2019), which replace hard min/max operations with smooth aggregates suited to gradient-based optimization. Those might benefit LLM-driven falsification too, but the notion of a single critical time would need to be rethought if the robustness semantics change.

There's also the question of scale. The current work fixes the initial state and searches only over input signals, using a finite parameterization of control points. Real-world falsification problems often involve richer search spaces — uncertain initial conditions, variable time horizons, more complex models. Whether the LLM's semantic advantage survives that added complexity is an open empirical question.

And there's a deeper question about why this works. The researchers inspect the reasoning traces of the LLM to understand how it constructs falsifying inputs, but the inner workings of these models remain partly opaque. The "surprisingly effective optimizer" phenomenon is real, but our understanding of it is thin. The paper's ablation study — isolating each enrichment's contribution — is a step toward a mechanistic account, but the field has a long way to go.

Perhaps the most productive next direction is the most obvious one: hybrid systems. If an LLM can find a bug on the first sample but can't prove the absence of bugs, and a numerical optimizer can grind through a landscape the LLM might miss, why not combine them? The LLM as an informed explorer that seeds a classical optimizer with promising starting points; the classical optimizer as a fine-tuner that polishes the LLM's intuitions. The paper's own framing — that the LLM brings "semantic comprehension and causal reasoning" while classical tools bring rigorous search — practically begs for such a synthesis.

For now, though, the headline finding stands on its own. In the curious collision of chatbots and control systems, we've discovered that a language model can read a specification about a car, gaze at a trajectory, and point to the exact instant the car breaks the rules — and do it on the first day it shows up to work. The robots may not be ready to design our transmissions yet. But they're starting to be surprisingly good at finding the moments when those transmissions fail.