Hosted Mender Incident History

1. resolvedAug 30 · 18:20 UTC

   This incident has been resolved.

Reported by Hosted Mender on their status page.

1. postmortemAug 28 · 08:24 UTC

   ## **Executive Summary** An issue was identified where microcontroller \(MCU\) devices, specifically those running Zephyr OS and `mender-mcu` on constrained hardware like the ESP32-S3, lost connection to the hosted Mender EU server \(`eu.hosted.mender.io`\). Specifically, devices failed during the initial TLS handshake with error `-0x2700`, preventing Over-The-Air \(OTA\) updates and inventory reporting. The investigation confirmed that the root cause was **insufficient client-side memory \(mbedTLS heap\)** when processing the server's RSA-4096 certificate chain, rather than a certificate validity or time-synchronization issue. To recover devices already deployed in the field without requiring a physical intervention, the engineering team deployed a **server-side mitigation by switching the production TLS certificate authority to Google Trust Services \(GTS Root R4\)**. This restored connectivity for affected field devices. ## **Impact** Customers using MCU devices targeting the EU instance were not able to perform updates from the 9th to the 15th of July. ## **Technical Root Cause** The hosted Mender EU endpoint utilized a cross-signed Let’s Encrypt certificate hierarchy ending in an RSA-4096 leaf and root certificate. From the 4th of June, letsencrypt \(EU\) started using a different chain from the same root CA, with one more intermediate. _Jan – early Jun 2026 │ leaf → R12/R13 → ISRG Root X1 --> 1 intermediate_ _Jun 4 2026 → now     │ leaf → YR1 → ISRG Root YR → ISRG Root X1 --> 2 intermediates_ This change landed at the beginning of July to hosted Mender EU, after automated cert-bot renewal, and then the mender-mcu clients lost connection. During the TLS handshake, mbedTLS requires continuous heap memory to simultaneously hold: * The TLS input and output content buffers \(`SSL_MAX_CONTENT_LEN`\). * The parsed peer certificate chain \(RSA-4096\). * The parsed trust anchors. * The temporary mathematical buffers required to execute an RSA-4096 signature verification. On constrained MCU devices, the default mbedTLS heap configuration \(`CONFIG_MBEDTLS_HEAP_SIZE=40960`, or ~40 KB\) was exceeded. Our tTesting revealed that processing this specific RSA chain required a minimum heap threshold of ~41.5 KB \(42496 bytes\). When the memory allocation failed internally during signature verification, mbedTLS surfaced the out-of-memory exception as a general certificate trust failure \(flag 0x08: `BADCERT_NOT_TRUSTED`\). A mitigation was published, to increase the `CONFIG_MBEDTLS_HEAP_SIZE`, but of course this can’t be applied to devices which lost the connection to the server. ## **Resolution & Recovery** ### **Immediate Field Recovery \(Server-Side\)** Because deployed devices with broken OTA connectivity cannot receive a firmware update to increase their local heap size, engineering deployed a server-side resolution: * Switched the production certificate chain for [eu.hosted.mender.io](http://eu.hosted.mender.io) from Let's Encrypt to **GTS Root R4 \(Google Trust Services\)**. * The GTS Root R4 chain is already bundled within the existing MCU client trust stores and requires significantly lower memory overhead during verification, allowing already-deployed devices to successfully complete the handshake and re-establish OTA connectivity without a client-side patch. * A previous attempt to force a lighter CA Chain with ECDSA root CA failed; although the required heap was less than the new RSA root, it was still heavier than the old RSA root, so our internal validation failed. ### **Long term solutions** * We are implementing an end-to-end failover strategy covering both TLS infrastructure and domain name \(DNS\) routing. * We implemented periodic tests to emulate a MCU device connecting to both US and EU instances to early catch this kind of issues. ‌ We sincerely apologize for the disruption this caused to your fleets, this was a serious failure, and we understand the operational burden it placed on you. We are committed to the improvements described above and to ensuring that changes in upstream certificate infrastructure never again silently break connectivity for constrained devices.

Reported by Hosted Mender on their status page.