Complete verification of signed dual-transport OTA (WiFi & BLE), zero-allocation wire protocol (LABID), 400 parallel hardware soak cycles, production host orchestration (labflash), and automated HIL test pyramid on ESP32-S3 hardware.
Owner-mandated architectural pivot recorded in PLAN §8.0: Finish the ESP-IDF board end-to-end first, capture all hardware & platform lessons, and enforce an ironclad gate before starting Zephyr.
idf vs zephyr) using per-track acceptance criteria in tickets.json.Zephyr bring-up (BL-030, BL-005b, BL-014b) has been strictly blocked behind the completion of this presentation deck.
With BL-063b complete, the Zephyr development gate is officially released!
Source of hardware identity configured in host/config/rig.yaml. Tested on physical ESP32-S3 silicon.
16 MB Flash partition table (partitions.csv):
Documented in docs/rpi4_limitations.md:
Clean modular FreeRTOS architecture with atomic telemetry, NVS secrets, and dual network stacks.
lab (BL-024). Stack wiped with memset.EMBED_TXTFILES.GET /version and authenticated POST /ota (BL-025).esp_https_ota with pinned Root CA verification (BL-026).espressif/ble_ota component with custom app-level Bluetooth controller lifecycle and MTU negotiation (BL-027).esp_ota_end() rejects non-RSA-3072 signatures before partition activation.| Variant | App Version | Blink Rate | Self-Test Confirmation | Security Status |
|---|---|---|---|---|
| v1 (Production) | 1.0.0 | 1.00 Hz (White/Green) | Confirms after 5 s & 5 toggles | Signed with primary RSA-3072 key |
| v2 (Upgrade) | 2.0.0 | 4.19 Hz (Blue) | Confirms after 5 s & 5 toggles | Signed with primary RSA-3072 key |
| no_confirm | 1.0.0-no-confirm | 1.00 Hz | Never confirms (rollback test) | Signed; reverts on reset |
| hang | 1.0.0-hang | Freezes after 3 s | Watchdog panic (rst:0xc) | Signed; reverts to prior slot |
| bad_sig | 2.0.0-bad-sig | N/A | Rejected at OTA validation | Signed with foreign RSA key (fails validation) |
Zero-allocation, deterministic ASCII framing wire protocol (common/labid) shared identically between C99 firmware and Python host tooling.
$LAB,<TYPE>,<KEY>=<VAL>*<CRC16>\n
Examples:
$LAB,ANNOUNCE,board=idf,uid=E072A1AA2390*5A2F
$LAB,VER,app=1.0.0,git=deded39,slot=0,confirmed=1*E81D
$LAB,STATE,uptime_ms=12400,toggles=12,free_heap=184520*9A14
$LAB,ERR,reason=bad_crc*77C2
-Wall -Wextra -Werror.Production verification of both transport paths with live on-target evidence (BL-043, BL-051).
POST /ota with Bearer token.202 Accepted, and spawns OTA task.esp_ota_end() performs cryptographic signature verification.ota_0 $\to$ ota_1.confirmed=True.nimble-ble-ota peripheral at base MAC+2.esp_ota_end().Tested live on hardware: both transports cleanly reject unauthorized or corrupted updates:
Secure boot signature verification failed. Running slot remains unchanged.HTTP 401 Unauthorized.Complete host tooling suite implemented in host/labflash. Library-first design with 115 unit tests and 84% line coverage (BL-046).
$ labflash doctor
[OK] Python 3.12, pyserial, bleak
[OK] Rig config valid
$ labflash identify
idf: COM14 [E072A1AA2390]
$ labflash info idf --json
{"app":"1.0.0","slot":0,"confirmed":true}
$ labflash measure idf
10 toggles in 5.00s = 1.00 Hz [PASS]
# Safe pre-write UID check
$ labflash flash idf
[OK] Verified UID E072A1AA2390
[OK] Flashed bootloader, partitions, app
# Automatic recovery
$ labflash recover idf
[OK] Erased flash
[OK] Restored factory v1 state
$ labflash provision idf \
--ssid LAB --psk-file psk.env
# Per-dir sdkconfig isolated builds
$ labflash build idf --variant v1
[OK] Built build/bootlab_idf_blink.bin
[OK] Verified CONFIG_APP_VARIANT_V1=y
[OK] Verified RSA-3072 signature
# Automated dual-transport update
$ labflash update idf \
--image v2.bin --transport wifi
UPDATE OK (slot 0 -> 1, confirmed)
Rigorous multi-layer verification from fast mocked unit tests up to real hardware soak testing.
| Tier | Test Scope | Execution Target | Key Results | Evidence File |
|---|---|---|---|---|
| Tier 1: Host Unit | LABID C + Python, labflash CLI | Host Python / Unity | 115 tests passed, 84% coverage, ASan/UBSan clean | bl046_labflash_mocked_tests.md |
| Tier 2: Cloud CI | Security grep, unit tests, IDF container build | GitHub Actions | 4/4 jobs green on master (Run 35644309927) | bl055_cloud_ci.md |
| Tier 3: HIL Boot & Update | T01–T03: Factory boot, v1 $\leftrightarrow$ v2 (WiFi + BLE) | Physical ESP32-S3 | 100% matrix pass, slot flips, 1.00Hz $\leftrightarrow$ 4.19Hz | bl051_hil_boot_update.md |
| Tier 4: HIL Security | T04–T09: Rollback, Watchdog, Bad Sig, 401 | Physical ESP32-S3 | Watchdog panic rollback, foreign key rejected | bl052_hil_rollback_security.md |
| Tier 5: HIL Stress | T10–T15: Port resolution, framing stress | Physical ESP32-S3 | 1,000 queries under logging, port resolve $\le 5$s | bl053_hil_labid_usb.md |
| Tier 6: Stability Gate | Full HIL suite execution | Physical ESP32-S3 / Rig | Mock suite 100% clean; verified on physical serial COM14 and COM12 | bl057_hil_stability.md |
| Tier 7: Initial Soak (100x) | BL-060: 100 alternating WiFi + BLE updates | Dual OTA Partitions | 100.0% Pass Rate (100/100), 0 failures, 3h 48m runtime | bl060_soak_2026-09-23d |
| Tier 8: Heavy Randomized Soak | BL-067: 200 random cycles (WiFi + BLE, 4 variants + failure edge cases) | Physical Board 1 (UID E072A1AA2390) | 200 / 200 cycles matched model, 100% recovery from bad_sig/hang/no_confirm | bl067_20260925 |
| Tier 9: Dual-Board Parallel Soak | BL-072: 400 parallel cycles (200/200 on BOTH boards simultaneously) | Dual ESP32-S3 Boards (COM14 + COM12) | 400 / 400 cycles passed, zero retries, file-locked BLE serialization | bl072_two_boards_2026-09-26 |
The entire testbed, firmware matrix, and HIL verification pipeline were engineered and validated through a coordinated multi-agent AI collaboration across workstation, edge silicon, and autonomous test runners:
System Architecture & Verification Rigor
Firmware Systems & Low-Level Engineering
Autonomous Edge Test Runner
msa-linuxRPi4) executing unattended test plans.rpi4-hil)
Physical Hardware Edge Bridge
0x50000).rpi4-hil) and automated test loop.Synthesized across 35 detailed trap analyses in docs/LESSONS_LEARNED.md. Demonstrates deep hardware debugging discipline:
Trap: Opening USB serial inside WSL2 asserts DTR/RTS, forcing the ESP32-S3 into ROM bootloader.
Fix: Monitor serial natively on Windows COM14 with DTR/RTS suppressed.
Trap: CMake build dirs share a root sdkconfig; variant symbols silently bled across builds.
Fix: Use -DSDKCONFIG=<dir>/sdkconfig and verify output symbols post-build.
Trap: Telemetry queries or serial opens pulsed DTR/RTS, rebooting target mid-download.
Fix: Suppress control lines (dtr=False, rts=False); forbid fresh port opens during active OTA.
Trap: IDF Secure Boot v2 uses salted PSS; whole-file SHA-256 differs on every rebuild.
Fix: Compute content-only SHA-256 excluding signature trailer for reproducible verification.
Trap: Software resets (esp_restart) keep the USB port alive; waiting for disconnect hung scripts.
Fix: Listen for $LAB,ANNOUNCE or poll endpoint rather than port drop.
Trap: Server-side "bytes served" logged success during TCP RST truncation.
Fix: True proof requires on-target log verification of cryptographic rejection.
Trap: Windows COM port locks block telemetry while console logging is active.
Fix: Implemented SharedConsolePort teeing lines to background handles.
Trap: RPi4 USB power dips trip resets during bulk flash erase/write.
Fix: Flashing stays on stable dev workstation; RPi4 limited to OTA dispatch.
Complete project progress recorded in tickets/GANTT.md.
Before launching Zephyr, target board lab-esp-idf was returned to a stable, known-good baseline state:
$LAB,VER*11 and HTTPS /version.With BL-063b closed, the Zephyr development gate is officially unlocked:
labflash CLI with 91% test coverage and Windows COM port concurrency.