@ -26,6 +26,6 @@ Further, a feedback channel which provides information about the update status o
The remaining part of this paper is laid out as follows.
Section \ref{related} discusses related work.
Next, in Section \ref{env} we present the environment, our research is based on, while Section \ref{req} defines the requirements for the implementation of an OTA update mechanism in this environment.
Next, in Section \ref{env} we present the environment, our research is based on, while Section \ref{req} defines the requirements for the implementation of an \textit{OTA} update mechanism in this environment.
A concept for the implementation is presented in Section \ref{concept} and a reference implementation can be found in Section \ref{implementation}.
Finally, a conclusion and future work can be found in Section \ref{conclusion}.
The research presented in this paper was mainly driven by \textit{Magrathea Laboratories e.V.}\cite{maglab}, the local hackerspace in Fulda, in cooperation with researchers at the department for computer science at Fulda University.
The research presented in this paper was mainly driven by \textit{Magrathea Laboratories e.V.}\cite{maglab}, the local hackerspace in Fulda, Germany, in cooperation with researchers at the department for computer science at Fulda University of Applied Sciences.
Requirements were clearly defined by Magrathea Laboratories' demands to provide local and remote control over various sensors and actors in the foundations rooms to visitors and members.
Such components include door sensors, power sockets, temperature sensors, projectors and screens who are all managed by a home-automation controller, which is driven by the software \textit{home-assistant}\cite{HASS}.
It provides direct control over all existing components using a web UI and allows to define rules and automations on how these components interact.
@ -9,7 +9,7 @@ For the component's hardware, boards based on the \textit{ESP8266} micro-control
These boards feature a small and robust design, achieve very low power consumption and integrate WiFi without requiring any extra components.
It integrates a Tensilica L106 32-bit micro controller unit (MCU) with a maximum CPU performance of 160 MHz, 64 kB instruction memory and another 96 kB of main memory.
According to the manufacturer, the ESP8266 is among the most integrated WiFi-capable chips in the industry.
While at the beginning of this research, mostly \textit{ESP-01s}\cite{ESP-01s} boards in combination with self-developed power supplies and use-case specific hardware components were deployed, we quickly integrated wireless smart switches offered by Sonoff.
While at the beginning of this research, mostly \textit{ESP-01s}\cite{ESP-01s} boards in combination with self-developed power supplies and use-case specific hardware components were deployed, we quickly integrated \textit{Sonoff}\cite{sonoff} wireless smart switches product series offered by \textit{ITEAD}.
The firmware for all of the \textit{ESP8266}-based devices in the hackerspace is based on our common software platform, referred to as \textit{ESPer}.
\textit{Sming}, which in turn is based on the open-source SDK for \textit{ESP8266}, provides the base library for this framework.
@ -23,9 +23,9 @@ To prevent possible attackers from injecting malicious software into the embedde
New firmware only gets accepted by the device, iff the cryptographic signature of the downloaded firmware image can be verified.
\subsubsection{}\label{req5}
To reduce network load and aim for the maximum possible uptime of the device, the update process should only be done if a new firmware version is available.
To reduce network load and aim for the maximum possible uptime of the device, the update process should be done only if a new firmware version is available.
In contrast, on the release of new firmware, the roll-out to all devices should be performed as fast as possible.
%While checking for available updates and downloading such an update, the device should continue to work as usual.
While checking for available updates and downloading such an update, the device should continue to work as usual.
\subsubsection{}\label{req6}
For easy maintenance and monitoring, each device should provide detailed information about the currently installed firmware version and other details relevant for the update process.
\section{Concept for implementing OTA updates}\label{concept}
\section{Concept for implementing \textit{OTA} updates}\label{concept}
To implement \textit{OTA} updates under the given requirements, we first define a topology that integrates our build infrastructure, firmware repository, and controller with the IoT WiFi network, which the devices are connected to.
For our reference implementation, we particularly chose lightweight and common software projects to allow for easy exchangeability of the individual components.
The base topology, as well as the specific components used is shown in Figure \ref{fig:topology}.
@ -9,7 +9,7 @@ The base topology, as well as the specific components used is shown in Figure \r
\label{fig:topology}
\end{figure}
The source code of the \textit{ESPer} project is published into a \textit{Git} source code repository.
The source code of the \textit{ESPer} project is published into a \textit{Git}\cite{git}source code repository.
From there, the continuous integration system is responsible for automatically building and publishing the firmware image files, as soon as updated source code is available.
It is also in charge of assembling and publishing meta-information (version and signature) required for the update process.
Both systems are described in detail in the following section.
@ -21,7 +21,7 @@ Beside data like device type, chip and flash ID, the published data includes det
This allows administrators to find devices with outdated bootloaders and helps to find missing or failed updates.
\subsection{Common framework and build infrastructure}
The framework includes a build system, which allows to configure basic parameters for all devices, including, but not limited to, the Wi-Fi access parameters, the \textit{MQTT} connection settings and the updater URLs.
The framework includes a build system, which allows to configure basic parameters for all devices, including, but not limited to, the WiFi access parameters, the \textit{MQTT} connection settings and the updater URLs.
Each device requires to have the \texttt{UPDATE\_URL} option set to make the update work.
Skipping the option results in the exclusion of the code for update management during the build.
By sharing the same code, all devices ensure to have a common behavior when it comes to reporting the device status or interacting with the home-automation controller.
@ -36,9 +36,9 @@ By doing so, any device specific code is always linked to the latest version of
The according device type is provided as a string through a global constant at compile time and it must never be changed during operation.
Device specific code is organized in a sub-folder for each device type.
To build the software, a \textit{Makefile}\cite{make} is used, which provides a simple way for reproducible builds.
Whenever a new build process is started, the build system scans for all device specific folders and calls the original build process for each of them.
Whenever a new build is started, the build system scans for all device specific folders and calls the build process for each of them.
After the build of the firmware has finished, the build system also creates a file for each device type, containing the build version and cryptographic signatures of the corresponding firmware images.
To avoid interferences between different build environments, and to roll out new versions as quickly as possible, the code has been integrated into a continuous integration (CI) system.
To avoid interferences between different build environments, and to roll out new versions as quickly as possible, the code has been integrated into a continuous integration (CI) system which is also responsible for publishing the resulting firmware images to the firmware server queried during updates and notifies the devices to check for an update.
\subsection{Device setup and flash layout}\label{flashlayout}
@ -48,29 +48,29 @@ The memory mapping mechanism of the MCU allows only a single page of 1 MB of fla
\caption{The flash layout used for two ROM slots.}
\label{fig:memory_layout}
\end{figure}
As the image to download and flash possibly exceeds the size of free memory heap space, the received data must be written to flash directly.
In contrast, executing the code from the memory mapped flash while writing the same area with the downloaded update leads to unexpected behavior, as the executed code changes immediately to the updated one.
To avoid this, we chose the memory layout shown in Figure~\ref{fig:memory_layout}.
The flash is split into half to contain two firmware ROMs with different versions, one being executed and one which is being downloaded.
The flash is split into half to contain two firmware ROM slots with different versions, one being executed and one which is being downloaded.
This standby firmware also acts as a safety mechanism if the download fails or is interrupted as the previous version stays intact and can still be used (refer to requirement \ref{req2}).
In case of an error the old firmware is kept unchanged and will be used until the successful download of a newer firmware succeeds.
In addition to the two firmware ROMs, the flash provides room for the bootloader and its configuration.
In addition to the two firmware ROM slots, the flash provides room for the bootloader and its configuration.
\subsection{Cryptographically securing the firmware update}
To ensure only valid firmware is running on the devices, a cryptographic signature of the firmware images is calculated and checked as part of the update process.
For the calculation of the signatures, the \textit{sha256} hashing algorithm \cite{RFC6234} and an elliptic curve cipher based on \textit{Curve25519}\cite{bernstein2006curve25519} are used, which are both considered modern and secure methods for software signing (see \cite{barker2016nist, bsi}).
For calculating and verifying the signatures of a firmware image, the \textit{SHA-256} hashing algorithm \cite{RFC6234} and an elliptic curve cipher based on \textit{Curve25519}\cite{bernstein2006curve25519} are used, which are both considered modern and secure methods for software signing (see \cite{barker2016nist, bsi}).
The cryptographic signature for each of the two firmware ROMs is created by the continuous integration system during build time and is provided as meta-information along with the firmware images.
The cryptographic signature for each of the two firmware images is created by the continuous integration system during build time and is provided as meta-information along with the firmware images.
Therefore, the CI system must be equipped with the private key used to create the signatures.
In contrast, the micro controller only needs to know the according public key, to be able to verify the cryptographic signature.
For the same reasons as stated in Section \ref{flashlayout}, the signature of the new firmware image can not be verified before it is written to flash.
Therefore, the calculation of the sha256 checksum required for the signature is done while the update is downloaded and written to flash.
After the download has succeeded, the signature is verified and the bootloader only gets reconfigured if the signature is validated successfully.
Therefore, the calculation of the \textit{SHA-256} checksum required for the signature check is done while the update is downloaded and written to flash.
After the download has succeeded, the checksum is verified against the signature and the bootloader gets reconfigured iff the signature is validated successfully.
Otherwise, the bootloader will not be reconfigured and the system will not start the invalid firmware.
\subsection{Build infrastructure and automatic deployment}
The CI system, which is based on \textit{drone}\cite{drone} allows to execute commands, whenever a new version is published into the \textit{Git} repository.
The CI system, which is based on \textit{drone}\cite{drone} allows to execute commands, whenever a new version is published into the projects \textit{Git} repository.
A corresponding \textit{drone} configuration called \texttt{.drone.yml} exists beside the source code (Listing~\ref{lst:drone}).
Within this configuration file, settings relevant to the build process are provided to the build environment.
First, the \texttt{CONFIG=maglab} option lets the build system use an additional configuration file (\texttt{Configurion.mk.maglab}), which is stored inside the framework repository and provides environment specific information, such as the WiFi SSID.
To keep secrets like the WiFi password unexposed, it is not written down in the configuration file.
To keep secrets like the WiFi password and the private key unexposed, it is not written down in the configuration file.
Instead, to include secrets into a build process while allowing to keep the configuration public, \textit{drone} allows to encrypt these with a repository specific key.
Using this method, the password is stored as \texttt{.drone.sec} file inside the repository from where it is injected into the build environment.
Also noticeable in Listing~\ref{lst:drone} is the firmware version, which is configured to be the first 8 letters of the \textit{Git} commit hash uniquely identifying a version of the source code.
@ -33,7 +33,7 @@ publish:
\end{lstlisting}
For deployment, only the master branch is considered.
After a successful build, all distribution files (the firmware image and meta-information files) of all device types are copied to the repository server, where they are served by a HTTP server.
After a successful build, all distribution files (the firmware image and meta-information files) of all device types are copied to the repository server, where they are served by a \textit{HTTP 1.1}\cite{HTTP_1.1} server.
The configuration file (\texttt{Configurion.mk.maglab}) references this repository server as the source for updates.
Support for multiple devices of different type is implemented in both, the \textit{ESPer} framework itself and the build system.
@ -44,7 +44,7 @@ A single function \texttt{Device* getDevice()} must be defined exactly once in e
To implement this interface, a static instance of \texttt{Device} is created and returned.
Each \texttt{Device} is populated with device specific \texttt{Feature} instances.
While the \texttt{Feature}-API leverages common run time polymorphism to share functionality between features, the initial \texttt{Device} creation uses compile time polymorphism, which reduces the need for memory management and increases performance by avoiding virtual function tables.
Listing~\ref{lst:create_device_socket} shows the complete device specific code used for a simple power socket, which is mainly confined to the device type and its capabilities (i.e., the number of GPIO pins).
Listing~\ref{lst:create_device_socket} shows the complete device specific code used for a simple power socket, which is mainly confined to the device type and its capabilities (e.g., the GPIO pin numbers to use).
\begin{lstlisting}[caption={Device specific code for a socket driver.},
@ -67,13 +67,12 @@ The first one is a helper \textit{Makefile} built to accept a parameter for devi
In addition, the primary \textit{Makefile} scans a project subdirectory and uses each directory in there as a container for device specific code.
For each of these directories, the helper \textit{Makefile} is called and the subdirectories name is used as the value of the \texttt{DEVICE} parameter.
By splitting the build and recompiling the framework each time before intermixing it with the device specific code, the device type identifier can be used inside the shared framework code.
While building a devices firmware, the meta-information file used during updates is also created and stored beside the firmware image.
For development, each device can be build separately by using the device type identifier as \textit{Makefile} target.
In addition, the suffix \texttt{/flash} can be used to flash a specific firmware to the device.
While building the firmware images for a device, the build environment provides some constants which are baked into the resulting firmware image.
Beside the environmental configuration like the \textit{WiFi} credentials, \textit{MQTT} topics and other configurable tweaks, the current device and version identifiers are provided as compile time constants.
In addition, the public key used to verify firmware signatures during updates is derived from the private key and provided as an linker object which is linked into each firmware image (~\ref{lst:public_key_object}).
Beside the environmental configuration like the WiFi credentials, \textit{MQTT} topics and other configurable tweaks, the current device and version identifiers are provided as compile time constants.
In addition, the public key used to verify firmware signatures during updates is derived from the private key and provided as an linker object which is linked into each firmware image (Listing~\ref{lst:public_key_object}).
This allows to use all the information inside the code without any overhead while being configurable during build time.
\begin{lstlisting}[caption={Creating the linker object containing the public key.},
The build process will create the two firmware images, one for each ROM slot, and the meta-information file.
While building a devices firmware, the meta-information file used during updates is also created and stored beside the firmware image.
The meta-information file has a simple line oriented ASCII format, which is easy to generate and efficient to pars within the limited constrains of the embedded device.
To create the meta-information file, the current version identifier is written to the \texttt{.version} file.
After the build, the signatures for both firmware images are created and attached to the file.
The update mechanism is split into four main phases: checking for updates, reprogramming the device, calculating and verifying the cryptographic signature of the updated firmware, and reconfiguring the boot process to use the new firmware.
\subsubsection{Checking for updates}
In order to inform the IoT devices of the availability of a new firmware version, the update server provides a file for each device type containing meta-information about the latest available firmware version.
The meta-information file has a simple line oriented ASCII format, which is easy to generate and efficient to pars within the limited constrains of the embedded device.
It consists of the version identifier and the cryptographic signatures of both of the firmware binaries.
The version identifier can be an arbitrary string as the content is not interpreted semantically but only compared to the version identifier used during build time.
The other two lines in the meta-information file provide the hexadecimal representation of the cryptographic signatures, one line for each firmware binary file.
These meta-information files are provided by the update server using \textit{HTTP 1.1}\cite{HTTP_1.1}under the following path pattern: \texttt{\$\{DEVICE\}.version} (whereas \texttt{\$\{DEVICE\}} is the device type name).
These meta-information files are provided by the update server using \textit{HTTP 1.1} under the following path pattern: \texttt{\$\{DEVICE\}.version} (whereas \texttt{\$\{DEVICE\}} is the device type name).
Each device queries the update server regularly (initially at the end of the boot process and periodically once an hour) for the currently available firmware version.
It uses the \texttt{UPDATER\_URL} option to identify the update server.
@ -21,8 +19,8 @@ This allows faster roll-outs of updates and finer control for manual maintenance
\subsubsection{Reprogramming the device}
As the \textit{ESP-01s} is only equipped with 1 MB of flash, this means that the whole memory is mapped to a contiguous address space.
Therefore, the second ROM can not be re-mapped to have the same start address as the first ROM.
While the firmware is executed without any dynamic linking mechanism and the chip does not support position independent code, the addresses used in the ROMs are dependent to the offset at which the firmware is stored.
Therefore, the second ROM slot can not be re-mapped to have the same start address as the first ROM slot.
While the firmware is executed without any dynamic linking mechanism and the chip does not support position independent code, the addresses used in the ROM slots are dependent to the offset at which the firmware is stored.
This arises the need for building two firmware images, one for each target location.
To do so, a linker script for each of the two ROM slots was created, which is used to create two variations of the same firmware, only differing in ROM placement.
The two resulting firmware image files are both provided for download via \textit{HTTP 1.1} - which one to download depends on the target ROM slot and is selected by the device during the update process.
@ -41,16 +39,16 @@ irom0_0_seg :
\end{lstlisting}
\textit{rBoot}\cite{rBoot} has been choosen as it is integrated within the \textit{Sming} framework and allows to boot to multiple ROMs.
\textit{rBoot}\cite{rBoot} has been choosen as it is integrated within the \textit{Sming} framework and allows to boot to multiple ROM slots.
For configuration, an \textit{rBoot} specific structure is placed in the flash at a well-known location directly after the space reserved for the bootloader code.
This structure contains, among other things, the target offsets for all known ROMs and the number of the ROM to boot from on next reboot.
This structure contains, among other things, the target offsets for all known ROM slots and the number of the ROM slot to boot from on next reboot.
%The full memory layout of this approach is shown in Figure~\ref{fig:memory_layout}.
To calculate the origin of application data for each slot, the available memory of 1 MB is split in half and an offset of the size of the bootloader code and its configuration (0x2000 bytes) is added.
To calculate the origin of application data for each ROM slot, the available memory of 1 MB is split in half and an offset of the size of the bootloader code and its configuration (0x2000 bytes) is added.
For alignment and easy debugging, the second block is also shifted by the same amount ob bytes as the first block.
The unused gap of 8192 bytes is used by some applications to store data which can persist over application updates.
\begin{lstlisting}[caption={The flash layout used for two ROMs.},
\begin{lstlisting}[caption={The flash layout used for two ROM slots.},
@ -72,8 +70,8 @@ Using the same files for flashing and updating allows better debugging by elimin
Listing~\ref{lst:choosing_rom} shows the algorithm used to determine the download address and reconfigure the bootloader.
The update server provides these files in the exact same way as it provides the meta-information files, but the path pattern differs: the suffixes \texttt{.rom0} and \texttt{.rom1} are used to provide the firmware image files for the first and second slot respectively.
For installing a firmware update, the new firmware image file is downloaded using an HTTP GET request.
While the image is being downloaded each chunk received in the download stream is used to update the \textit{SHA256} hash before it is written to the flash.
For installing a firmware update, the new firmware image file is downloaded using an \textit{HTTP 1.1}\texttt{GET} request.
While the firmware image is being downloaded each chunk received in the download stream is used to update the \textit{SHA-256} hash before it is written to the flash.
When the write has been finished, the next chunk is received and the process continues until all chunks have been received.
After the download of a new ROM has been finished successfully, the calculated hash is checked against the signature provided in the meta-information file triggering the update and the public key baked into the running firmware. Only if the firmware is considered valid, the bootloader configuration is altered to boot into the new ROM slot and the device is rebooted.
After the download of a new firmware image has been finished successfully, the calculated hash is checked against the signature provided in the meta-information file triggering the update and the public key baked into the running firmware. Only if the firmware is considered valid, the bootloader configuration is altered to boot into the new ROM slot and the device is rebooted.
Over the last years, a rapidly growing number of IoT devices is found on the market, especially in the area of the so-called smart home.
These devices, which are deployed in vast numbers and frequently in use over many years, pose a risk to the internet as a whole if not provided regularly with security patches.
These devices, which are deployed in vast numbers are frequently in use over many years and pose a risk to the users privacy and to the internet as a whole if not provided regularly with security patches.
Hence, a fully automated process for large-scale software updates of such embedded devices must be considered.
In this article, we present an implementation of a durable and stable system for building and publishing cryptographically secure firmware updates for embedded devices based on \textit{ESP8266} microcontrollers.
This includes mechanisms to build the updates from source and automatically sign, distribute and install them on the target devices.
