esper-ota/4-1-update_mechanism.tex

\subsection{The update mechanism}

The implementation of the update mechanism consists of three parts which interact closely: checking for updates, reprogramming the device and reconfiguring the boot process.
This sections describes all three of these parts in detail.

The build-time configuration was extended to include a new option called \texttt{UPDATER\_URL}, which is the base URL used to query the update server.
Each device requires to have this option set to make the update work.
If the option is skipped, the code for update management is excluded during the build.


\subsubsection{Checking for updates}

Initially, each device queries the update server regularly for the current firmware version and initializes the update process if remote and local versions differ.
To do so, the update server provides a file for each device type containing the available version identifier, which is stored beside the firmware binary files.
These version identifier files are provided by the update server using \textit{HTTP 1.1}\cite{HTTP_1.1} under the following path pattern: \texttt{\${DEVICE}.version} (whereas \texttt{\${DEVICE}} is the device type name).
The version identifier can be an arbitrary string as the content is not interpreted semantically but only compared to the version identifier used during build time.

Each device tries to fetch the version identifier file once every hour.
After the version identifier file has been downloaded successfully, the whole file content is compared to the version identifier provided during build time.
If the version identifiers differ, the update process is initialized; in cases where the download has failed, the update server or the network connection was not available or any other error occurred, another attempt will be made at the next interval.

In addition to the interval, a special \textit{MQTT} topic shared by all devices is subscribed on device startup: \texttt{\${MQTT\_REALM}/update}.
Every time a message is received on this topic, a fetch attempt for the version identifier file is triggered.
This allows faster roll-outs of updates and finer control for manual maintenance.


\subsubsection{Reprogramming the device}

As the binary to download and flash possibly exceeds the size of free memory heap space, the received data must be written to the flash directly.
In contrasts, executing the code from the memory mapped flash while writing the same area with the downloaded update leads to errors, as the executed code changes immediately to the updated one.
To avoid this, the flash is split into half to contain two firmware ROMs with different versions, one being executed and one which is being downloaded.
This standby firmware also acts as a safety mechanism if the download fails or is interrupted as the previous version stays intact and can still be used.
In case of an error the old firmware is kept unchanged and will be used until the successful download of a newer firmware succeeds.

\begin{lstlisting}[language=,
                   caption={Linker script to build firmware for two ROM slots.},
                   label=lst:linker_script]
irom0_0_seg : org = ( 0x40200000         // The memory mapping address
                    + 0x2000             // Bootloader code and config
                    + 0x10               // Data offset after header
                    + 1M / 2 * ${SLOT}   // Offset for the ROM slot
                    ),
              len = ( 1M / 2 - 0x2010 )  // Half ROM size excl. bootloader
\end{lstlisting}

\begin{figure}[htbp]
\centering\includegraphics[scale=0.6]{flash_layout.pdf}
\caption{The flash layout used for two ROMs.}
\label{fig:memory_layout}
\end{figure}

Microcontroller boards based on the \textit{ESP8266} MCU are mostly following the same layout: the MCU is attached to a flash chip which contains the bootloader, firmware and other application data.
The memory mapping mechanism of the MCU allows only a single page of 1 MB of flash to be mapped at the same time\cite{ESP8266_Memory_Map} and the selected range must be aligned to 1 MB blocks.
As the \textit{ESP-01s} is only equipped with 1 MB of flash, this means that the whole memory is mapped to a contiguous address space.
Therefore, the second ROM can not be re-mapped to have the same start address as the first ROM.
While the firmware is executed without any dynamic linking mechanism and the chip does not support position independent code, the addresses used in the ROMs are dependent to the offset at which the firmware is stored.
This arises the need for building two firmware binaries, one for each target location.
To do so, a linker script for each of the two ROM slots was created, which is used to create two variations of the same firmware, only differing in ROM placement.
The two resulting firmware binary files are both provided for download via HTTP 1.1 - which one to download depends on the target ROM slot and is selected by the device during the update process.
Listing~\ref{lst:linker_script} shows the only difference between the two linker scripts, where \texttt{\${SLOT}} must be replaced with the slot number according to the current build.

In addition to the two ROMs, the flash must provide room for the bootloader and its configuration.
\textit{rBoot}\cite{rBoot} has been choosen as it is integrated within the \textit{Sming} framework and allows to boot to multiple ROMs.
For configuration, an \textit{rBoot} specific structure is placed in the flash at a well-known location directly after the space reserved for the bootloader code.
This structure contains, among other things, the target offsets for all known ROMs and the number of the ROM to boot from on next reboot.

The full memory layout of this approach is shown in figure~\ref{fig:memory_layout}.
To calculate the origin of application data for each slot, the available memory of 1 MB is split in half and an offset of the size of the bootloader code and its configuration (0x2000 bytes) is added.
For alignment and easy debugging, the second block is also shifted by the same amount ob bytes as the first block.
The unused gap of 8192 bytes is used by some applications to store data which can persist over application updates.

\begin{lstlisting}[caption={The flash layout used for two ROMs.},
                   label=lst:choosing_rom]
#define UPDATER_URL_ROM(slot) (( UPDATER_URL "/" DEVICE ".rom" slot ))

// Select rom slot to flash
const auto& bootconf = rboot_get_config();

// Add items to flash
if (bootconf.current_rom == 0) {
  updater.addItem(bootconf.roms[1], UPDATER_URL_ROM("1"));
  updater.switchToRom(1);
} else {
  updater.addItem(bootconf.roms[0], UPDATER_URL_ROM("0"));
  updater.switchToRom(0);
}
\end{lstlisting}

For installing a firmware update, the new firmware binary file is downloaded using an HTTP GET request.
The update server provides these files in the exact same way as it provides the version identifier files, but the path pattern differs: the suffixes \texttt{.rom0} and \texttt{.rom1} are used to provide the firmware binary files for the first and second slot respectively.
The firmware files provided on the update server are the exact same ones as used to initially flash the chip for the according version.
Using the same files for flashing and updating allows better debugging by eliminating errors related to the update process itself and makes development and initial installation very easy.
Listing~\ref{lst:choosing_rom} shows the algorithm used to determine the download address and reconfigure the bootloader.

After the download of a new ROM has been finished successfully, the bootloader configuration is altered to boot to the new ROM slot and the device is rebooted.


\subsubsection{Publish device information}
For monitoring and maintenance purposes, each device publishes a set of information to a well-known \textit{MQTT} topic after connecting to the network.
Beside already existing dates like device type, chip and flash ID, the information block has been extended with details about the bootloader, SDK and firmware version as well as relevant details from the bootloader configuration, like the currently booted ROM slot and the default ROM slot to boot from.
This allows administrators to find devices with outdated bootloaders and helps to find missing and failed updates.