Dual-OS Bootloader & Resilient OTA Testbed
A production-proven embedded architecture demonstrating fail-safe over-the-air updates, automatic rollback, and zero-allocation telemetry across ESP-IDF (FreeRTOS) and Zephyr RTOS on physical ESP32-S3 hardware.
Project Perspectives
Explore the system through product reliability, business impact, and deep systems engineering
Zero Field-Bricking Guarantee
Every OTA binary is cryptographically verified (RSA-3072 / ECDSA P-256). If an image fails to boot or panics on a watchdog, the hardware autonomously rolls back to the proven factory slot in ≤ 17 seconds.
Multi-Transport Redundancy
Update over local Wi-Fi / HTTPS for rapid broadband delivery, or fall back to Bluetooth Low Energy (NimBLE / SMP) when devices are sealed in closed field enclosures without Wi-Fi access.
Vendor-Neutral Portability
Proves firmware deployment on both Espressif's native ESP-IDF and the Linux Foundation's Zephyr RTOS, protecting hardware roadmaps from single-vendor lock-in.
Eliminating Warranty Recall Costs
Field returns of bricked IoT units average $150–$300 per device in shipping and support. Dual-partition rollback with watchdog timeouts ensures 100% survivability during field updates.
Auditable Engineering Rigor
39 out of 56 tickets completed with strict empirical evidence gates. No claim was verified without live target serial and network telemetry captures, guaranteeing auditable compliance.
Zero Silicon Waste (0 eFuses Burned)
Achieved enterprise-grade software signature verification without burning irreversible hardware security bits, keeping prototyping inventory completely reusable.
Deep Systems Architecture
Dual-OS implementation comparing FreeRTOS task prioritization with Zephyr's cooperative/preemptive threads. Hardware watchdog integration (`task_wdt`) triggers automatic MCUboot slot rollback.
Zero-Allocation LABID Wire Protocol
Engineered an ASCII wire protocol (`$LAB,CMD*CS\n`) implemented identically in C99 firmware and Python host tools, featuring one-fputs atomic frame writes to prevent log interleaving.
Automated HIL Rig & Multi-Agent Swarm
Created SharedConsolePort to multiplex live logs and automated HIL testing orchestrated across an autonomous agent collaboration: Antigravity (architecture & verification), Claude (firmware engineering), Hermes agent, and Raspberry Pi 4 edge runner.
Dual-OS Architecture Matrix
Side-by-side comparison of the two production firmware stacks tested on target silicon
FreeRTOS & Native Bootloader
Native Espressif 2-stage bootloader with anti-rollback version security, dual 2.5 MB application partitions, and 8 MB Octal PSRAM.
- Bootloader 2-Stage ESP-IDF (Slot 0/1)
- Signatures RSA-3072 Software Verification
- WiFi OTA esp_https_ota (Pull from LAN)
- BLE OTA espressif/ble_ota (NimBLE Stack)
- Telemetry LABID Port over USB-Serial-JTAG
- Rollback esp_ota_mark_app_valid_cancel_rollback()
MCUboot & MCUmgr SMP
Linux Foundation open-source RTOS with MCUboot swap-with-revert orchestration, MCUmgr SMP management over BLE and UDP port 1337.
- Bootloader MCUboot (Swap with Revert)
- Signatures ECDSA P-256 (imgtool verified)
- WiFi OTA MCUmgr SMP over UDP (Port 1337)
- BLE OTA MCUmgr SMP over Bluetooth LE
- Coexistence Unified Single Binary (Internal SRAM)
- Rollback mcumgr image test & confirm handshake
The 6 Great Engineering War Stories
Real hardware traps, low-level bugs, and permanent architectural defenses
The WinRT GATT Caching Trap
Windows WinRT aggressively caches GATT services across reboots, causing BLE OTA reconnects to fail after partition swaps.
The Trap: WinRT caches GATT services by MAC address. When MCUboot swaps from v1 to v2, the UUID table changes, but Windows refuses to re-query the device, breaking subsequent transfers.
The Fix: Injected winrt={"use_cached_services": False} into our BleakClient and SMPBLETransport wrapper, forcing fresh service resolution on every connection.
COM Port Locking (`SharedConsolePort`)
Windows opens COM ports exclusively (`Access is denied`), preventing background loggers from running alongside LABID queries.
The Trap: Every telemetry query opened and closed the COM port per call. In-between queries, logs were lost—specifically during the crucial esp_ota_end reboot window.
The Fix: Engineered SharedConsolePort: opens the port once, keeps it open, and runs a background thread that tees raw lines to console.log while passing bytes to query handles without collisions.
Software vs. Hardware Reset Behavior
Host test scripts waiting for USB port disconnections hung forever because software reboots don't drop the USB bus.
The Trap: On ESP32-S3 USB-Serial-JTAG, esp_restart() preserves USB enumeration. Only physical reset buttons drop the bus (~600 ms).
The Fix: Redesigned host detection: tools stop waiting for port drops and instead listen for the atomic $LAB,ANNOUNCE boot frame or poll the HTTPS version endpoint.
RPi4 Power Brownouts (`0x50000`)
RPi4 auxiliary runner recorded 7 under-voltage events in 10 minutes (`0x50000`), risking flash block corruption.
The Trap: Flash memory write peaks during USB flashing cause voltage dips on unpowered RPi4 USB buses, corrupting bootloaders.
The Fix: Strict operational boundary codified in docs/rpi4_limitations.md: heavy builds and USB flashing stay on workstation; RPi4 acts strictly as an OTA network dispatcher over LAN.
Zephyr Radio Coexistence IRQ (BL-064)
Coexistence firmware ran fine, but host writes hung: the board's LABID UART RX interrupt never fired (`irq=0, rx=0`).
The Trap: In Zephyr build_v2, concurrent Wi-Fi STA and BLE controllers competed for interrupt matrix allocations, starving the USB-CDC UART RX IRQ.
The Fix: Documented full repro in bl064_zephyr_labid_rx_irq_dead.md, safely reflashed target back to verified v1, and isolated IRQ registration order in main.c.
The "Vacuous Evidence" Trap
Host image servers reported "all bytes served" during corrupted image tests, creating false proof of firmware rejection.
The Trap: TCP RST truncation occurred mid-transfer; the board never even parsed the header, but the server counter reported success.
The Fix: Banned server-side counters as acceptance evidence. Every rejection must be proven directly by on-target firmware logs (e.g. Secure boot signature failed) and verified slot stability.
Serial DTR/RTS Reset Mid-OTA (Trap 34)
A fresh serial port open or telemetry query pulsed DTR/RTS, instantly rebooting the ESP32-S3 during active 1.25 MB downloads.
The Trap: Opening Windows COM ports without explicit control line disabling causes the USB-Serial-JTAG chip to pulse DTR/RTS, triggering a hardware reset mid-download that caused intermittent 7-hour OTA stalls.
The Fix: Explicitly suppressed control lines (dtr=False, rts=False) across all host tools and forbade fresh serial opens while background OTA transfers stream bytes. Verified with 4/4 clean installs.
Salted RSA-PSS Signatures (Trap 30)
Identical source code and private keys yielded different whole-file SHA-256 hashes across identical rebuilds, confounding CI.
The Trap: ESP-IDF Secure Boot v2 signs images with salted RSA-PSS. Because the salt is cryptographically random per signing invocation, whole-file hashes always differ even for identical binaries.
The Fix: Separated payload content hashes from signature verification: our verification tools compute SHA-256 strictly across payload segments excluding the appended signature trailer.
Strategic Lessons Learned (R1–R15)
Audit of initial project assumptions versus verified physical hardware behavior
| Risk ID | Domain | Initial Risk & Assumption | Hardware Reality | Architectural Defense |
|---|---|---|---|---|
| R1 | Bootloader | MCUboot lacks swap-with-revert on ESP32-S3 | Verified revert-capable swap mode boots and reverts cleanly on hardware. | Gate Verified |
| R2 | Memory / Coex | BLE + WiFi memory exhaustion on single binary | NimBLE + WiFi + HTTPS fit within 8MB Octal PSRAM; Zephyr fits in SRAM. | Budget Enforced |
| R3 & R4 | Security | Risk of burning permanent eFuses during testing | Zero eFuses burned. Identical SHA-256 eFuse summaries verified before & after. | CI Filter Guard |
| R7 | Power | RPi4 can power and flash ESP32-S3 USB | RPi4 suffered 7 brownout UV events / 10 min (throttled 0x50000). | Network-Only Host |
| R9 | Protocol | Console debug logs interleave with LABID frames | Atomic one-fputs frame writes under lock prevent interleaving across 1,000 queries. | C99 Framing Lock |
| R14 | Platform Bridge | WSL2 resets ESP32-S3 on serial port open | usbipd / WSL2 drops DTR/RTS during attach; native Windows serial does not. | Native Windows Port |