Benchmarking

Arc benchmarks are split by evidence level. Host results catch algorithmic regressions early. ESP32-S3 firmware results are the authority for target timing.

Host Regression Benchmarks

Run:

./tools/host-bench.sh

This benchmark runs Arc host-compilable primitives and standard-library baselines in one binary. It is a regression signal for queue shape, codec work, DSP kernels, and accidental overhead before firmware builds start.

For framework-adjacent host lanes, run:

./tools/framework-compare.sh

That script reports the checked-out ESP-IDF and Arduino-ESP32 source versions, then measures host-buildable Arc, raw ESP-IDF mbedTLS, and Arduino core paths where the same process can execute the compared code honestly.

The regular CI workflow runs the Arc host regression benchmark on firmware changes. The slower framework comparison is available through workflow_dispatch by enabling include_framework_benchmarks; it checks out and compiles Arduino-ESP32 sources.

ESP32-S3 Firmware Benchmarks

The on-device benchmark is:

./tools/ensure-frameworks.sh
cd examples/esp32s3/bench
. ../../../env.sh
idf.py build flash monitor

The firmware prints grouped arc-bench rows with an explicit surface column:

• arc for Arc primitives, codecs, DSP, service wrappers, and usage mixes.
• idf for raw ESP-IDF silicon or library calls measured in the same firmware image.
• arduino for optional Arduino-ESP32 component lanes when the local checkout is available.
• std for standard C/C++ baselines that are valid on the target.

Competitive Scope

The benchmark should compare real, like-for-like work:

• queue handoff, fan-in, latest-snapshot publication, and log lanes;
• stream framing and caller-buffer MQTT, WebSocket, and CoAP codecs;
• telemetry bridge, control tick, and protocol-bundle usage mixes;
• DSP dot, scale, MAC, FIR, SOS, state-space, and SIMD-adjacent kernels;
• raw ESP-IDF RNG, SHA, AES/GCM, NVS, OTA, temperature, async memcpy, and TWAI self-test paths;
• optional Arduino Print, frame emission, integer formatting, and base64 paths;
• Arc Text and format_to integer formatting lanes beside the Arduino print lane.

Do not publish fake external-bus numbers. SPI, I2C, UART, ADC, LCD, camera, and wired CAN comparisons need a documented fixture, attached devices, stable wiring, and captured evidence.

Reporting Rules

Every public benchmark report should include:

• Arc commit SHA;
• ESP-IDF commit/tag;
• Arduino-ESP32 commit/tag when enabled;
• ESP32-S3 board and flash/PSRAM configuration;
• exact command used;
• raw serial output, not only summarized winners.

That keeps benchmark claims reproducible and prevents host-only convenience numbers from being mistaken for silicon timing.