Posted by François Baldassari

From a salt shaker with a built-in speaker to smart water devices that bring clean water to communities with weak infrastructure, connected devices are increasingly advancing into all areas of our lives. But more connectivity brings more possibilities for crippling issues that can impact product development, operations, and maintenance. IoT developers must consider how to plan for firmware architecture that leads to a better, stickier product.

Competition among connected device manufacturers is swelling in every corner of the industry, and user patience for clunky products won’t get the benefit of the doubt that developers might otherwise have had in the IoT’s nascent days. As users become more dependent on connected devices, consumer demands that those devices consistently function well - and securely - become the expectation. There remains, of course, work to be done: a quick Google search reveals stories like the Fitbit firmware update that destroyed the device battery, or the Tesla key fobs that could be overwritten and hijacked until a patch was rolled out.    

These stories underscore that the IoT ecosystem’s connected nature requires that hardware developers approach product development differently - and take firmware updates seriously. It used to be that developers could write static firmware for specific device use cases or commoditized products and, once released, have no further interaction or engagement with the product. That system no longer works. To have a successful product, IoT device manufacturers need to invest in design and in firmware development equally.

Whether it’s BLE on phones or LTE or Zigby and other mesh networks, IoT devices are connected, regularly transmitting sensitive and personal data to and from the cloud. The near limitless reach of modern connected devices across all areas of our lives, paired with the high price point of most IoT devices underscores that IoT developers must have a plan (and not an after-the-fact reaction) for firmware maintenance. Putting that plan in motion requires three considerations:

Device monitoring

Ubiquitous connectivity brings with it major challenges, but it also brings opportunities. Among other things, it allows automated device health monitoring. The typical process of releasing a product relies on users’ reporting a problem and requiring them to physically return the device to be evaluated, repaired, and returned. Simply put, this is a huge waste of money and time, and it also risks frustrating the customer to the point of losing them entirely. Using customers as your testers is simply a terrible business decision. (Maybe you could get away with it if you were the only game in town, but IoT device makers don’t have that luxury anymore). Automated device monitoring is the solution. By regularly analyzing the health of devices and flagging potential problems immediately, a monitoring system can help device makers catch and fix issues in hours that would have otherwise taken them weeks to root cause. Designing embedded systems with such capabilities gives critical observability into performance, stability, and overall health - either of a single device or of a fleet of millions. 

Repair

Shipping products that require an update or patch is inevitable for even the most talented and thorough teams. Just ask NASA. While no one can avoid updates entirely, it is possible to detect fleet-wide issues and solve them without burdening users. The key is to roll out updates incrementally, starting with a small number of devices and ramping up over time. This limits the impact of any new issues and insulates most of your users from the churn of getting a few bugfix releases in a row.  Another good option is to implement an A/B update system if you have enough flash memory. This allows your device to download an update in the background with no user impact and simply prompts the user to reboot once the update is ready. Fast and simple update flows like A/B updates are key to compliance, and prevent too much fragmentation across your fleet. Last but not least, it is important to pair regular updates with a monitoring system so you can quickly identify problems with the update, and rollouts can be paused or aborted altogether.

Building with security in mind

The ubiquity of IoT devices has accelerated customer demands for robust device security in lockstep, with regulatory bodies becoming more serious (and punitive) about security requirements and standards. For those building smart devices, I would offer these principles as table stakes for security: 

1. Devices must be updateable. 

2. Trusted boot is no longer optional. You need a chain of trust to control the firmware running on your device.

3. Rotate secrets and don’t use a master secret. Whether that means a set of encryption keys or other secrets to make devices functional, they must be dynamically changed, so the compromise of one device does not lead to the compromise of others. 

Software teams have long embraced iterative processes, and IoT device developers can learn much from this process. Focusing on firmware architecture that is responsive, observable, and proactive, lets device manufacturers ship a better product and create a happier customer base.

François Baldassari is the Founder and CEO of Memfault, a cloud-based observability platfrom for hardware devices. Prior to Memfault, François worked on developer infrastructure initiatives at Pebble and Oculus.