Qanum L1: Public Testnet Benchmark Plan

This page reproduces docs/BENCHMARK-PLAN.md from qanum-core v0.9.0 (release tag). Links to files that are not published on this site are shown as plain text until the source repository is public.

This plan operationalizes the benchmark agenda of the consensus specification (§22.2): what will be measured on the public testnet, with which metrics, tools and cadence, and how each measurement feeds the decisions that must be frozen before v1.0. The testnet is the evidence base: every number published here ships with its method and raw data.

Reference hardware classes (results are always reported per class):

• Class S: x86_64 server/workstation (exact CPU, cores, RAM, storage reported with each run).
• Class P: Raspberry Pi 4 Model B, 4 cores, 8 GB RAM, SSD (the verified minimum platform, see TESTNET-RUNBOOK.md).

Item 1: Post-quantum verification costs -> definitive VERIFY UNITS

Question. Are the Verify-Unit costs right? Today (src/constants.rs): VU_MLDSA44_VERIFY = 100, VU_SLHDSA128S_VERIFY = 1600, VU_MLDSA87_VERIFY = 290 and VU_SLHDSA256S_VERIFY = 3600, against MAX_BLOCK_VERIFY_UNITS = 200 000. They come from the 2026-10-14 campaign below (worse ratio to ML-DSA-44 of the two cores, rounded up, with room for the spread between rounds).

Metrics. Per-verify time (median, p95) for the four suites: (a) isolated through the node’s verifier, single-threaded; (b) inside transaction and block validation, with the signature cache empty (cold) and full (warm); per machine. Derived: cost ratios normalized to ML-DSA-44 = 100.

Method/tools. examples/pq_bench (--release --locked), pinned to one core with taskset, one job on the machine: A/B/A/B/A rounds per suite family, blocks filled up to the first consensus limit, JSON Lines output with the commit, compiler, liboqs crate, CPU and kernel (README §Performance).

Decision rule. Final VU values ∝ measured CPU cost with a documented safety margin, proposed as a spec change through the AIP process.

Status (2026-10-14). First campaign on the build machine (aarch64, one Gold core at 2.42 GHz and one Silver core at 1.80 GHz; not one of the classes above, so reported as its own machine), development build: results and method. ML-DSA-87 measures 283-286 units and SLH-DSA-256s 3200-3492; SLH-DSA-128s measured 1466-1576. The constants are 290, 1600 and 3600, each above the highest ratio measured on this machine, with room for the 0.77 % spread between rounds. A block at the auth-bytes limit (415 ML-DSA-87 spends) validates cold in 152 ms (Gold) / 553 ms (Silver) on one thread; the verify-unit worst case (57 SLH-DSA-256s recoveries, measured at an earlier constant of 3500 VU; 55 fit at 3600) in 125 ms / 446 ms. Second run on the reference VPS (x86-64, AMD EPYC-Milan, KVM): results. Every check is faster there (worst blocks 75-86 ms cold) but the SLH-DSA ratios are higher (SLH-DSA-128s 2623-2687, SLH-DSA-256s 5601-5902): liboqs 0.16 runs ML-DSA with AVX2 code and SLH-DSA in portable C. The budgets bound time, so the constants follow the slowest core. Pending: class S and class P runs.

Item 2: TARGET_BLOCK_TIME: 75 s vs 90 s

Question. Keep 75 s (implemented: TARGET_BLOCK_TIME = 75, src/constants.rs) or move to 90 s? Spec §21 defers this to real testnet data.

Metrics. Orphan/stale rate (competing branches per unit time); reorg depth distribution; block propagation delay (peer-observed arrival time − block curtime); block-interval distribution vs the 75 s target; network hashrate context (getnetworkhashps) at measurement time.

Method/tools. Continuous passive collection on the seed nodes: getchaintips snapshots + Reorganize:/UpdateTip: log events, aggregated weekly. Propagation delays from peer-level timestamping in the seed logs.

Decision rule (proposed). Orphan rate ≤ 1 % sustained over ≥ 30 days at representative hashrate => keep 75 s. Sustained excess => evaluate 90 s with the same dataset (both intervals can be simulated from the observed propagation delay distribution before any consensus change).

Item 3: Mempool pressure under auth bytes

Question. How do real workloads fill blocks, given that throughput is dominated by MAX_BLOCK_AUTH_BYTES (3 MB) and VU (200 000), not by base block size (spec §22.1)?

Metrics. Observed auth/base byte ratio per block; block fill rate against both caps; mempool occupancy and feerate distribution (getmempoolinfo); rejection counters by reason (dust < 44 jomas, feerate below 1 joma/byte, caps).

Method/tools. Faucet activity plus volunteer load campaigns with documented transaction mixes (P2PQK payments, P2MULTI m-of-n, VAULT/HTLC custody flows; P2MULTI 2-of-3 spends alone are ≈ 7.7 kB of auth). Passive per-block stats from the seed nodes; weekly aggregation.

Decision rule. Input to spec §23.8 (fee market per auth byte / VU) and to any cap recalibration proposal.

Item 4: utxo_root and chainstate cost at scale

Question. What does the 256-level sparse Merkle tree commitment and the atomic chainstate snapshot cost as the UTXO set grows, the mainnet risk #1 (spec §22.1)?

Metrics. utxo_root recomputation time per block (with/without pruning); chainstate snapshot save/load time and bytes; bytes/UTXO; peak RSS; per-block triple-write cost (block+undo, chainstate, mempool).

Method/tools. The repo already ships the harnesses (ignored, run manually, minutes + RAM):

cargo test --release --test chainstate_scale_benchmark -- --ignored --nocapture --test-threads=1
# sizes via QANUM_BENCH_SIZES=250000,1000000,2500000 (default shown)

plus the sparse_tree_* benchmark family (incremental, reuse, partial-reuse, stability, stats), experimental_commitment_benchmark, import_session_benchmark / import_session_stress, node_session_benchmark and net_v2_ibd_stress_and_resilience. Live cross-check: IBD wall-clock on Class P (reference: full testnet chain ≈ 15 min as of ~23.5k blocks) and datadir growth (≈ 113 MB at that height).

Decision rule. Viability thresholds for a domestic full node on Class P (snapshot time, RSS, disk). If the incremental-commitment engine becomes necessary, the two ignored stress tests kept for it (pathological_block_tree, reindex_amplification) define the acceptance bar.

Item 5: ASERT stability under hashrate swings

Question. Does the every-block ASERT retarget (half-life 172 800 s = 2 days, ASERT_HALF_LIFE, src/constants.rs) stay stable under abrupt hashrate changes?

Metrics. Difficulty time series vs hashrate time series; oscillation amplitude after step changes; recovery time (blocks and seconds) to return within ±10 % of the pre-step interval; block-interval variance during and after the swings.

Method/tools. Controlled campaigns: coordinated miner on/off steps (≈ ×2 and ≈ ×0.5 hashrate) announced in advance; getnetworkhashps 120 <h> at historical heights for the work-over-time series; getmininginfo difficulty sampling; seed logs for interval statistics. The post-idle difficulty decay toward the pow limit (and the resulting fast-block burst) is a natural recurring experiment: it is expected behavior and will be measured as such.

Decision rule. No sustained interval deviation > 2× target outside the recovery window => half-life confirmed for v1.0; otherwise propose an adjusted half-life via AIP with the measured series attached.

Item 6: Bootstrap: full IBD vs QUSN snapshot install

Question. What wall-clock, disk and bandwidth does a trustless QUSN v2 snapshot bootstrap save versus a full IBD on the same machine and network, and does the saving hold once historical bodies are backfilled?

Metrics. Wall-clock from first accepted header to verificationprogress = 1.0 (node log timestamps); datadir size at tip; peak RSS; bytes downloaded. With -backfill-bodies=1: additional wall-clock and disk until getsnapshotinfo reports the backfill complete.

Method/tools. Two fresh datadirs on the same machine, same release tag and feature set:

1. Full IBD: start the node on fresh datadir A against the reference seeds; record time to tip.
2. Snapshot bootstrap: export the artifact once from an archive node (qanumd utxo-snapshot-export <path>; a serving peer via -snapshot-ibd=1 is equivalent), install it into fresh datadir B (qanumd utxo-snapshot-install <path>), start the node and record time to tip. Optionally repeat with -backfill-bodies=1 and record the backfill phase separately.

Both paths end at the same trustless verification: the artifact’s UTXO set must recompute to the utxo_root committed by a header chain that re-validates against the local genesis (see WALKTHROUGH-SNAPSHOT.md), so the comparison isolates cost, not trust.

Decision rule. Publish per-class ratios (time, disk, bandwidth) with exact tag, feature set and command lines; feed the decision of whether snapshot bootstrap becomes the advertised default onboarding path for domestic nodes (spec §22.1).

Status (2026-09-17). First campaign, measured with a development build that predates the first public release, on the earlier public test chain (host: KVM guest, Intel Xeon @ 2.60 GHz, 2 vCPUs, 2 GB RAM, Linux 6.1, x86-64; release + crypto-oqs; tip at measurement: ≈ 35 730). The current testnet is a new chain; the campaign will be repeated on it.

Bootstrap paths compared, fresh node to operational tip:

Speedup to a usable tip: ≈ 40-90× vs the estimated full IBD for the zero-touch path (measured ≈ 20-35 s on every host vs estimated 20-30 min), and every snapshot path degrades to full IBD automatically on any failure (availability, never safety).

Measured components behind the table (artifact: a published snapshot of that chain at height 32 686, SHA256 52e838ba...d6e22; P2P run served by both live seeds at height 35 726):

• verify, 32 687 embedded headers linked + PoW re-checked: 5.0 s
• install into a fresh datadir: 4.9 s (pin EMBEDDED)
• forward-sync 32 686 -> 35 704 (3 018 blocks): ≈ 4 min
• body backfill, 32 686 bodies: ≈ 13.5 min in the background while the node is already usable (+10.3 MB)
• -snapshot-ibd=1 breakdown: DNS discovery -> both seeds offered the artifact -> 5 × 2 MiB chunks downloaded in parallel from 2 peers, verified and installed in ≈ 13 s -> forward-sync to tip; provenance recorded as peer:<seed>:18445 (getsnapshotinfo)

• zero-touch path, multi-host check (same protocol, live seeds):

  Host	Binary	Start -> usable tip
  KVM guest, Intel Xeon @ 2.60 GHz, 2 vCPUs, 2 GB RAM, Linux 6.1 (x86-64)	dev build	17-33-35 s (three consecutive direct runs); the first ≈ 2 min measurement stands as a slow outlier
  Samsung phone: Qualcomm Snapdragon, 8 cores up to 2.84 GHz, 10 GB, Ubuntu 20.04 (aarch64)	dev build¹	29-30-30 s (three consecutive runs); one earlier run recorded 10 s and was not reproduced
  Raspberry Pi 4 Model B: Class P, the verified minimum platform (aarch64)	dev build¹	36 s
  VPS: KVM guest, AMD EPYC Milan, 6 vCPUs, 8 GB RAM, Linux 6.8 (x86-64)	dev build¹	26 s

  ¹ an earlier development build with the same snapshot code, so the measurements are comparable. All hosts were re-measured with the same direct command (1 s polling, no stability window); the spreads to note are run-to-run on the same machine (phone: 10 s outlier -> 29-30-30 s; dev VM: ≈ 2 min first run -> 17-33-35 s), dominated by the P2P phase (peer discovery/connect), not by CPU. Every host, including the minimum platform, lands in the ~17-35 s band; the original ≈ 2 min figure was a slow first-run outlier

• early-era IBD segment measured: 0 -> 10 976 blocks in ≈ 7.5 min (≈ 24 blocks/s); current-era segment: 32 686 -> 35 704 in ≈ 3-3.5 min (≈ 17.6 blocks/s)

Why the snapshot matters (what this campaign shows). The trust level is identical in every row: the artifact is never trusted, it must recompute the utxo_root committed by a header chain that re-validates against the local genesis, so the comparison isolates pure cost. At testnet scale the chain is light and the win is time-to-usable-state: a new node is validating live blocks ≈ 17-35 s after start (run-to-run spread is P2P-phase variance, not hardware class) instead of ≈ 20-30. At mainnet scale the disk/bandwidth dimension joins in: the artifact grows with the UTXO set, not with history, while IBD pays for every byte ever produced; the gap in the table widens, it does not shrink. Class P is covered by the Pi row above; the full-IBD number remains pending; per the reproducibility rule, ESTIMATED rows are labelled as such.

Cadence, publication, data retention

• Passive collection (items 2, 3, 5): continuous on seed nodes, weekly aggregation.
• Campaigns (items 1, 4, 5 controlled steps): at least once per release cycle, and before any v1.0 freeze decision.
• Publication: results (method, hardware class, raw series) published on qanum.org and summarized in the release notes of the following release; consensus-affecting outcomes enter the spec through the AIP process (spec §23).
• Reproducibility: every published number names the exact tag/commit, feature set (crypto-oqs or not), hardware class and command lines.