Benchmark methodology and results

A simulator is only useful if you can trust it. That is why Kmila's engine is verified systematically and automatically against GHDL, the open-source VHDL simulator used as a reference in industry and academia. The current result: 148 of 148 cases produce identical results, with not a single discrepancy.

What SimulatorBench is

SimulatorBench is Kmila's accuracy harness. For each case it runs the same problem through both simulators and compares their outputs sample by sample:

flowchart LR
    Spec["Problem<br/>specification"] --> Ref["reference/<br/>(VHDL + testbench<br/>from a public source)"]
    Spec --> Kmila["kmila.vhdl<br/>(independent<br/>implementation)"]
    Ref --> GHDL["GHDL"]
    Kmila --> Engine["Kmila engine"]
    GHDL --> Golden["golden.vcd"]
    Engine --> Out["kmila.vcd"]
    Golden --> Diff["per-port diff"]
    Out --> Diff
    Diff --> Result["PASS / FAIL"]

• golden.vcd — GHDL's output, taken as the golden reference.
• kmila.vcd — Kmila's engine output for the same stimulus.
• diff.json — the comparison of both traces.

How the comparison works

The comparison is designed to be fair and meaningful:

• Declared ports only. The bench compares only the design's declared out and inout ports. Internal signals and auto-generated clocks are ignored, because they are implementation details, not observable behavior.
• Identical stimulus. Each case defines a problem with its ports, its stimulus and its duration. If a case cannot be reduced to a (t, port, value) stimulus table, it is marked with an explicit skipReason rather than fabricating a substitute testbench.
• Sample by sample. It does not compare a summary: it contrasts each port's values across thousands of signal samples.

The honesty constraint

The most important aspect of the methodology is how the cases are written. The reference files come from public sources, and their origin (sourceUrl) and license are recorded. Kmila's own implementation (kmila.vhdl) is written from the problem specification, never by reading the reference VHDL.

This prevents the two implementations from "resembling" each other by copying, and makes their matching outputs a real signal of engine equivalence, not an artifact of the method.

Results and evolution

The corpus has grown alongside the engine's coverage. Over the harness's lifetime, roughly 26 cumulative simulator bugs have been fixed thanks to these comparisons.

Date	Cases	Result
2026-06-24	119 / 119	100%, 0 skipped
2026-07-28	136 / 136	100%, 0 skipped
2026-08-03	148 / 148	100%, 0 skipped

The most recent cases exercise demanding constructs: multi-hop concurrent alias chains, constant-array element access by dynamic and literal index, named-association aggregates, case-branch resolution, (others => '0') expansion at signal-construction time and inside clocked reset branches, the unsigned * unsigned width rule from numeric_std, indexed left-hand-side writes inside processes, direct entity instantiation, and correct inlining of function and procedure bodies with local variables and early return.

How to reproduce it

The bench is a command-line tool that verifies both GHDL and the Kmila API are reachable, discovers the cases and runs them:

dotnet run -- verify   # checks that GHDL and the Kmila API respond
dotnet run -- list     # lists the discovered cases
dotnet run -- run      # runs the whole corpus

How to read the result

148/148 means that, in every case of the corpus, Kmila's output matches GHDL's on all observable ports and across all samples. It is a measure of behavioral equivalence over the corpus, not a claim that Kmila implements the entire VHDL standard: coverage grows as cases are added. To see which constructs the engine supports, check Supported features.