Future directions¶

Candidate work that is not scheduled. The roadmap admits a milestone only once a concrete project supplies the use case, semantics, runnable example, test coverage, and performance peer. Nothing here has cleared that bar; this page records the analysis so the reasoning survives, in the same spirit as benchmarks/authoring_core/OPTIMIZATION_NOTES.md.

Two areas are covered: capabilities the framework lacks, and the verification corpus needed to judge whether it is actually better than the alternatives.

Align the scheduling semantics with cocotb¶

Note

Largely resolved. This section records the analysis as it stood before timing_backend and deferred_writes shipped. Both gaps described below are now closed by standard project configuration; see The write model for what cpptb does today. Only item 3 at the end – making the write model unconditional rather than a key – is still open. Read what follows as history, not as current behavior.

cocotb is the framework most people arriving here have used, and the trigger vocabulary already matches it deliberately: RisingEdge, FallingEdge, ReadOnly, ReadWrite, NextTimeStep, with drivers, monitors and scoreboards composing the same way. Two things then behave differently, and both are invisible until a test produces a wrong answer.

Writes apply immediately rather than being deferred. In cocotb an assignment is queued and applied at the next ReadWrite point, so writing straight after await RisingEdge(clk) cannot affect the edge just awaited. Here set() applies at once and co_await RisingEdge{} resumes before the design evaluates the edge, so the same shape drives into that edge and the transaction commits a cycle early. The symptom is a read-back returning the value just written, which reads as a design bug.

No project build supplies the phase waits. ReadWrite, ReadOnly and NextTimeStep are documented API, and Performance benchmarks four backends providing them, two of which pass the timing conformance suite. A default cpptb build provides none of them, and a testbench using those waits builds cleanly and fails at run time with a message naming --vpi. Putting --vpi in build.verilator_args makes the waits run, and the drive point it produces is right, but it does not give the documented contract: cpptb build links Verilator’s --binary main, which does not re-evaluate the design between the ReadWrite and ReadOnly callbacks, so two of the five timing-phase conformance checks fail with no diagnostic. Both contract-complete backends need a --cc --exe --build link against src/verilator_timing_main.cpp instead of --binary, which no cpptb.toml key can ask for. The generated SV-DPI backends are reachable only by matching design.defines against build.cxx_flags by hand, where defining CPPTB_SV_DPI_TIMING alone silently selects the inline pump that the same benchmarks record as invalid. Roadmap carries the worked examples and the observed output.

Three ways to close the gap, roughly in increasing order of commitment:

Name and validate the backend selection. A [build] timing_backend key threaded into code generation would replace a --vpi that does not deliver the contract, emit the link both contract-complete backends need, and reject the define combinations that currently produce wrong answers without a diagnostic. This is plumbing over machinery that already exists and is already measured.
Offer deferred writes. A queued write applied at the next settle point, alongside the immediate set(), would make a driver written the cocotb way correct on any backend rather than only where phases are available.
Make deferred the default after an edge wait. The most familiar, and the largest change: it alters what existing testbenches do and costs a scheduler round trip per write, so it would need the performance peer to justify it.

The first turns a flag that does not deliver the contract into a supported choice. The second is what actually makes a translated cocotb testbench correct. The third is the only one that removes the difference entirely, and is the one that needs the most evidence.

Items 1 and 2 have since shipped: timing_backend names the backend and emits the link both contract-complete backends need, and deferred_writes = true supplies the cocotb write model. Both are now standard project configuration – see The write model. Item 3, making that model unconditional rather than a per-project key, is what remains.

This has a concrete use case, a runnable example and a performance peer, so it clears the promotion bar in Roadmap if it is wanted. Found while porting Ibex’s dv/cs_registers testbench, where the cocotb-shaped driver was the first thing tried; see experiments/open_core_ports/ports/ibex_cs_registers.

Porting Ibex’s icache testbench put a size on what the convention costs, and it is larger than “drive off the falling edge” suggests. Once drivers call each other the rule has a second half: every task that drives has to be entered at a drive point, return at a drive point, and not open with a wait, because a task that opens with a wait re-anchors itself and issues its first write a clock late. Nothing states or checks either half, so each of that port’s driving tasks had to be traced edge by edge against the SystemVerilog clocking block it replaces before the port could be believed, and the rule survives only as a comment at the top of testbench.cpp. See experiments/open_core_ports/ports/ibex_icache_cpptb, whose per-item rates agree with the UVM baseline to about 1% on fetches/insn, the measure that would move first if a driver were mistimed.

Framework capabilities¶

Temporal assertions¶

The largest gap that no milestone currently covers. Every check today is procedural: expect_eq compares a value at one instant. There is no way to say “every request is followed by an acknowledgement within five cycles” except an ad-hoc coroutine with hand-written bookkeeping, and nothing distinguishes a property that passed from one that was never exercised.

This is the reason teams tolerate SystemVerilog. It is also the gap the coroutine model is best placed to close: implication, bounded windows, stability, and throughout all express naturally as awaitable expressions, and their pass, fail, and vacuity counts belong in TestContext alongside existing checks, feeding assertion coverage into cpptb/coverage.hpp.

It stays clear of the deliberate non-goals: no factory, no phases, no objections, no sequencer. It is a missing checking primitive, not imported architecture.

A second simulator¶

Already milestone 6, and worth restating as the gate on everything else. Verilator-only excludes encrypted vendor IP, gate-level netlists, power-aware runs, and accurate X propagation, which is most of what an industrial user needs. It is also the claim in the README that remains untested: transport is described as standards-based DPI, and one backend cannot demonstrate that.

Waveforms and transaction annotation¶

Milestone 7 already lists waveforms. Nothing in include/ or the code generator emits VCD or FST today. Debugging is where verification time is spent, and a failure currently yields logs and nothing else.

The valuable half is not the dump itself but the annotation: transaction recording already exists, so a scoreboard mismatch could place the offending transaction on the waveform at the right cycle. Combined with deterministic seeds, the deferred waveform-on-failure rerun becomes cheap and is worth promoting out of the backlog.

Coverage closure as a workflow¶

cpptb/coverage.hpp can merge, but merging is a primitive, not a methodology. Closure needs merging across seeds and runs, a report ranking holes by how reachable they are, and an answer to “what should I randomize next”.

This is deliberately not UCIS interchange, which remains correctly deferred: the value is the local closure loop, not a file format. Coverage as structured data that both C++ and Python tooling can consume plays to a strength SystemVerilog handles poorly.

Regression orchestration¶

Seeds already exist in the runner. The public CLI is build, list, and test. What is missing is the loop that multiplies everything else: many tests across many seeds, run in parallel, with failure clustering, a one-line reproduction command, and a trend over time. The backlog defers JUnit conversion, tag filtering, and reproduction-command presentation; individually small, together they are the difference between running tests and running a regression.

Simulation snapshot and restore¶

Long simulations pay their full cost on every run. A failure two million cycles into a boot sequence is debugged by replaying those two million cycles for every hypothesis, and a suite whose tests share one expensive initialization repeats it per test. The capability wanted is a snapshot of simulation state taken at a chosen point, and a later run that restores it and continues from there instead of starting at time zero.

The design half is tractable. Verilator generates save/restore support under --savable: VerilatedSave and VerilatedRestore round-trip the complete model state, and a cpptb simulator is one process built around one Verilated model, so design state, simulation time, and the generated wrapper’s calendar state all sit behind that one mechanism.

The testbench half is the real problem, and it is what shapes any honest design. A suspended C++ coroutine is a heap frame holding arbitrary locals, pointers, and a resumption address; there is no portable way to serialize one, so a snapshot cannot capture “the whole test, mid-await”. Two shapes survive that constraint:

Declared snapshot points. A test reaches a quiescent point — a settle point, where the deferred-write queue is empty by construction — and asks for a named snapshot alongside whatever testbench state it explicitly saves. A restore run reloads the design exactly and starts a fresh registered continuation coroutine there; suspended user coroutines do not come back. The contract is “the design resumes exactly, the testbench restarts deliberately”, which is also what vendor $save-style checkpoints deliver to UVM in practice. Most framework-owned state already serializes by construction: a random stream is a seed and derivation counters, coverage has a schema-1 JSON model, register-model mirrors and SparseMemory would need explicit save hooks. Clocks re-register through the existing time-zero query path.
Process-level checkpointing. fork() at the snapshot point captures everything, coroutine frames included: the parent holds the warmed state and forks one child per variant or hypothesis. This is cheap to prototype precisely because one process owns the whole simulation, and it covers the same-run cases — failure bisection, N seeds branching from one warmed reset — without any serialization design. It does not persist across runs; cross-run persistence via CRIU is Linux-specific and fragile against ASLR and library changes, and should not be the load-bearing mechanism.

The two compose: fork for exploration within a run, declared snapshots for resuming across runs. Promotion needs a concrete workload that hurts today — a long soak that fails late, or a directed suite replaying one boot per test — and the acceptance gate writes itself from machinery the repository already has: an identity gate, in the mold of make backend-equivalence-test, requiring that a restored run produce results and waveform identical to the uninterrupted run from the snapshot time onward.

Not recommended¶

Further scheduler performance work. Measured ratios sit between 0.76x and 1.08x against pure SystemVerilog, the optimization notes put the remaining architectural cost near 0.7%, and profiles attribute the rest to Verilator itself. Speculative protocol components should also stay deferred on the existing bar, since a component library built without a consumer is a maintenance liability.

A real verification corpus¶

The current benchmarks are framework-authored. Deciding whether cpptb genuinely improves on the alternatives needs testbenches somebody else wrote, for designs somebody else maintains, running workloads long enough to be meaningful.

What UVM on Verilator can and cannot do¶

Verilator 5.050 supports considerably more of the verification subset than it once did. Verified directly against the pinned version:

classes with rand members and constraint blocks solve and randomize;
constraint solving is delegated to an external z3 binary, so make z3-toolchain incidentally satisfies it;
covergroups with coverpoint and bins compile and sample.

Verilator’s own contributor documentation describes its current focus as “completing Universal Verification Methodology (UVM, IEEE 1800.2-2017) support”, so UVM is in progress rather than finished.

experiments/uvm_comparison/ already measured what that means in practice, on this same Verilator against Accellera UVM 1800.2-2017-1.0. UVM does run: one test passed three times out of three. The others did not. One segfaulted on a run, and both random tests executed while reporting scoreboard errors, so four tests produced two clean results between them.

The consequence for comparisons follows from that record rather than from an assumption. UVM is not yet a dependable performance peer on Verilator, because the tests that exercise randomization are the ones that fail, and a comparison is worthless if the baseline is not producing correct results. It is a legitimate ergonomics peer today: port its structure and compare the code. For performance the dependable peers on Verilator remain plain SystemVerilog and cocotb, which the four-mode harness already supports. This is worth re-testing periodically, since it turns on Verilator maturing rather than on anything in this repository.

Two later ports qualify that. experiments/open_core_ports/ports/core_ibex_uvm and .../ibex_icache_uvm run upstream UVM environments that somebody else wrote, and the icache one passes all ten of its own tests. Both needed work to get there, and ibex_icache_uvm records six Verilator defects with reduced cases, five of them in constrained randomization: dist applied as an equality against a pre-drawn sample, std::randomize ignoring dist weights, a solve ... before ... disabling every soft constraint in the class, a soft nested inside an if not being dropped when it conflicts, and a constrained randomize() over an inside range not being uniform over that range. The last of those is the whole of the stimulus difference between the icache UVM baseline and its cpptb port. So a UVM baseline can now be made to produce correct results on Verilator, and the stimulus it then produces still needs checking against what its own constraints describe.

A useful side effect of Verilator’s constraint support: pure-SystemVerilog twins can now use constrained-random classes rather than procedural stimulus, making them a more representative baseline than they were.

Candidates, verified as live and maintained¶

Project	Verification surface	Why it fits
lowRISC/ibex	`examples/simple_system` with `ibex_simple_system.cc`, plus `dv/uvm/core_ibex` and `dv/verilator`	Its Verilator harness is already hand-written C++, so a cpptb port is a like-for-like ergonomic comparison. Runs real RISC-V software.
alexforencich/verilog-ethernet	around 230 cocotb testbench directories	cocotb is coroutine-based, the closest methodological peer, and already a mode in the harness. Two of its modules are vendored already.
openhwgroup/cva6	`verif/` including `core-v-verif`, `env`, `regress`, `sim`, `tb`, `tests`	Application-class core with a full verification environment; the UVM half serves as an ergonomics peer.
pulp-platform/axi	`test/`	AXI is the protocol the component library lacks; APB is the only bus covered today.
lowRISC/opentitan	`hw/dv/verilator`	Chip-level with boot ROM and software. The most credible workload and the heaviest to adopt.
chipsalliance/caliptra-rtl	active, security root of trust	Realistic security-focused traffic; verification flow needs assessment before committing.

Suggested order¶

Ibex simple system. Smallest step with the clearest comparison, and it extends the existing PicoRV32 firmware workload to a maintained core whose Verilator flow is C++ already.
One verilog-ethernet cocotb testbench. Ported one-to-one, it measures cpptb against the peer methodology that most resembles it, on a design partly vendored already.
CVA6 or pulp AXI, depending on whether the goal is a larger workload or a wider protocol surface.
OpenTitan, once the earlier ports have established the porting pattern.

Vendoring should follow the precedent in benchmarks/framework_comparison/open_cores/: exact pinned files, upstream licence notices preserved in THIRD_PARTY_NOTICES.md, and only the RTL each workload elaborates.

What such a comparison must report¶

Performance alone would waste the exercise. Each port should record the semantic evidence both sides produce, the wall time and its validity under the existing environment guard, and the ergonomic measures that motivated the framework: lines of testbench code, how much is generated rather than written, what a failure reports before a debugger is opened, and how long a single test takes from edit to result.