Browse Source

proofreading by major

master
Sven Reissmann 7 years ago
parent
commit
586a451894
  1. 8
      1-introduction.tex
  2. 14
      2-environment.tex
  3. 15
      3-requirements.tex
  4. 54
      4-1-update_mechanism.tex
  5. 22
      4-2-multidevice_build_structure.tex
  6. 9
      4-3-automatic-deployment.tex
  7. 6
      4-implementation.tex
  8. 3
      5-conclusion.tex
  9. BIN
      index.pdf

8
1-introduction.tex

@ -5,9 +5,9 @@ On one side, it interacts with the hardware in a system specific way by implemen
On the other side, it provides use-case dependent functionality in interaction with general purpose hardware components.
Embedded systems are often thought as systems that never change their requirements or functionality.
But practical use shows that the environment in which these systems run do, in fact, change.
These changes include, but are not limited to, modifications of to the expected behavior or additions to it, reconfiguration of parameters related to communication with other systems and the users and correcting errors that have been reported after deployment.
In almost all cases, the requirements can be accomplished by changing the firmware and does no need any modification of the hardware.
However, practical use shows that the environment in which these systems run do, in fact, change.
These changes include, but are not limited to, modifications of to the expected behavior or additions to it, reconfiguration of parameters related to communication with other systems and the users as well as correcting errors that have been reported after deployment.
In almost all cases, the requirements can be accomplished by changing the firmware and do not need any modification of the hardware.
For updating the firmware on a system being deployed, the system must provide an interface for altering the firmware.
In addition, such an interface should provide mechanisms to check which firmware is currently installed and which configuration parameters are used.
@ -17,5 +17,5 @@ If a system is already able to communicate over a network interface, this can be
By reusing the existing communication channels, the dedicated update interface can be omitted which leads to smaller packaging and reduces production cost.
It also decreases the maintenance cost drastically because updates can be installed remotely.
\textit{OTA} updates allows the administrators to apply automation methods on the update process allowing to roll out updates in a controlled fashion. I.e. updates can be done on testing devices first, followed by security-critical deployments and subordinate ones can be delayed to times where the device is not utilized.
\textit{OTA} updates enables administrators to apply automation methods on the update process allowing to roll out updates in a controlled fashion. I.e. updates can be done on testing devices first, followed by security-critical deployments and subordinate ones can be delayed to times where the device is not utilized.
Information about the update status provided by the devices allows administrators to apply monitoring techniques ensuring all updates are installed and devices ar in the desired state.

14
2-environment.tex

@ -5,15 +5,19 @@ The home-automation projects developed by \textit{Magrathea Laboratories e.V.}\a
The different components available (like the door status, power sockets, projectors and screens, temperature sensors, etc.) are all managed by the home-automation controller driven by the software \textit{home-assistant}\autocite{HASS}.
It provides direct control over all existing components using a web UI and allows to define rules and automations on how these components interact.
% Make the paragraph sloppy to prevent the word ESP8266 to flow into the border
\begin{sloppypar}
The hackerspace has developed a common software and hardware platform for its home-automation projects called \textit{ESPer}\autocite{ESPer}.
For the hardware, boards based on the \textit{ESP8266} micro-controllers, mostly \textit{ESP-01s}\autocite{ESP-01s} boards, are used in combination with self-developed power supplies and use-case specific hardware components.
\end{sloppypar}
These boards provide a Microcontroller Unit (MCU) fast enough for all required scenarios and integrate WiFi without requiring any extra components.
The software is based on the \textit{Sming}\autocite{Sming} library, which in turn is based on the open source SDK for \textit{ESP8266} and integrates a lot of other software components for easy use.
To build the software, a \textit{Makefile}\autocite{make} is used and provides a simple way for reproducible builds.
To build the software, a \textit{Makefile}\autocite{make} is used, which provides a simple way for reproducible builds.
For communication with the controller, the \textit{MQTT}\autocite{MQTT} protocol is used.
It provides a lightweight messaging mechanism implementing the publish-subscribe pattern which allows devices to listen for commands and publish their current state to the controller and other interested parties.
The controller software has out-of-the-box support for this protocol which allows easy integration of all different device types using the same patterns.
It provides a lightweight messaging mechanism implementing the publish-subscribe pattern that allows devices to listen for commands and publish their current state to the controller and other interested parties.
The controller software has out-of-the-box support for this protocol, which allows easy integration of all different device types using the same patterns.
The components all share the same configuration in regard of the network access and the controller to communicate with.
The configuration is provided during build time which eschews the need for a configuration interface and reducing the management overhead thus minimizing security leaks.
The components all share the same configuration in regard to the network access and the controller to communicate with.
The configuration is provided during build time, which eschews the need for a configuration interface and reduces the management overhead, thus minimizing security leaks.

15
3-requirements.tex

@ -1,28 +1,29 @@
\section{Requirements}
The following requirements are defined as global project goals and has been refined during the work on the project multiple times.
The following requirements are defined as global project goals and have been refined during the work on the project multiple times.
\begin{itemize}
\item The systems should be able to perform updates on the release of new software releases without administrative interaction.
\item The systems should be able to perform updates on the release of new software without administrative interaction.
If a new version of the firmware is published, it should be prepared automatically for installation to the target devices.
All these devices should then download and install the new software version and start using it if no errors has occurred during the process.
All these devices should then download and install the new software version and start using it subsequently, if no errors have occurred during the process.
\item To ensure minimal maintenance effort, the update process should be insusceptible to errors as most as possible.
Even if the installation of an update fails in the middle of reprogramming the controller, the system should continue to work full functional immediately and after a reboot.
Even if the installation of an update fails in the middle of reprogramming the controller, the system should continue to work fully functional immediately and after a reboot.
\item Downloading the updated firmware should be done over the WiFi interface using the same network connection as used during normal operation.
Fetching the firmware should be done side-by-side with other traffic used during operation.
Fetching the firmware should be done side-by-side with other traffic produced during operation.
\item Reducing network load and aiming for the maximum possible device uptime is critical.
Therefore, the update process should only be done if a new version is available.
In contrast, the release of a new update should be rolled out to all devices as fast as possible.
During the checking for available updates and while downloading such an update, the device should continue to work as usual.
While checking for available updates and downloading such an update, the device should continue to work as usual.
\item For easy maintenance and monitoring, each device should provide detailed information about the currently installed firmware version and other details in relation to the update process.
\item For easy maintenance and monitoring, each device should provide detailed information about the currently installed firmware version and other details relevant for the update process.
\item Devices are categorized by types.
Each device type runs the same software and therefore provides the same functionality.
% What about the following sentence? "As ... something". Reads like something is missing, maybe a commata and then the next sentence?
As the device type is hardly coupled to the hardware.
It is provided through a global constant at compile time and it must never be changed during operation.
Updates must ensure that the correct firmware according to the device type is used while reprogramming.

54
4-1-update_mechanism.tex

@ -1,23 +1,23 @@
\subsection{The update mechanism}
The implementation of the update mechanism consists of three parts which interact closely: checking for updates, reprogramming the device and reconfiguring the boot process.
This sections describes all these parts in detail.
This sections describes all three of these parts in detail.
The build-time configuration was extended to include a new option called \texttt{UPDATER\_URL} which is the base URL used to query the update server.
Each device requires to have this option set to make the updates work.
The build-time configuration was extended to include a new option called \texttt{UPDATER\_URL}, which is the base URL used to query the update server.
Each device requires to have this option set to make the update work.
If the option is skipped, the code for update management is excluded during the build.
\subsubsection{Checking for update}
\subsubsection{Checking for an update}
Initially, each device checks the update server regularly for the current firmware version and initializes the update process if remote and local versions differ.
To do so, the update server provides a file for each device type containing the available version identifier which are stored beside the firmware binary files.
Initially, each device queries the update server regularly for the current firmware version and initializes the update process if remote and local versions differ.
To do so, the update server provides a file for each device type containing the available version identifier, which is stored beside the firmware binary files.
These version identifier files are provided by the update server using \textit{HTTP 1.1}\autocite{HTTP_1.1} under the following path pattern: \texttt{\${DEVICE}.version} (whereas \texttt{\${DEVICE}} is the device type name).
The version identifier can be an arbitrary string as the content is not interpreted semantically but only compared to the version identifier used during build time.
Each device tries to fetch the version identifier file once every hour.
After the version identifier file has been downloaded successfully, the whole file content is compared to the version identifier provided during build time.
If the version identifiers differ, the update process is initialized; if the download has failed, the update server or the network connection was not available or any other error occurred, another attempt will be made at the next interval.
If the version identifiers differ, the update process is initialized; in cases where the download has failed, the update server or the network connection was not available or any other error occurred, another attempt will be made at the next interval.
In addition to the interval, a special \textit{MQTT} topic shared by all devices is subscribed on device startup: \texttt{\${MQTT\_REALM}/update}.
Every time a message is received on this topic, a fetch attempt for the version identifier file is triggered.
@ -26,11 +26,11 @@ This allows faster roll-outs of updates and finer control for manual maintenance
\subsubsection{Reprogramming the device}
As the binary to download and to flash possibly exceed the size of free memory heap space, the received data must be written to the flash directly.
As the binary to download and flash possibly exceeds the size of free memory heap space, the received data must be written to the flash directly.
In contrasts, executing the code from the memory mapped flash while writing the same area with the downloaded update leads to errors, as the executed code changes immediately to the updated one.
To avoid that, the flash is split in half to contain two firmware ROMs with different versions, one being executed and one which is being downloaded.
To avoid this, the flash is split into half to contain two firmware ROMs with different versions, one being executed and one which is being downloaded.
This standby firmware also acts as a safety mechanism if the download fails or is interrupted as the previous version stays intact and can still be used.
In case of an error the old firmware is kept unchanged and will be used until downloading of a newer firmware succeeds.
In case of an error the old firmware is kept unchanged and will be used until the successful download of a newer firmware succeeds.
\begin{figure}[h]
\begin{lstlisting}[language=]
@ -53,25 +53,25 @@ irom0_0_seg : org = ( 0x40200000 // The memory mapping address
\label{fig:memory_layout}
\end{wrapfigure}
Microcontroller boards based on the \textit{ESP8266} MCU are mostly following the same layout: the MCU is attached to a flash chip which contains bootloader, firmware and other application data.
The memory mapping mechanism of the MCU allows only a single page of 1 MB flash to be mapped at the same time\autocite{ESP8266_Memory_Map} and the selected range must be aligned to 1 MB blocks.
As the \textit{ESP-01s} is only equipped with 1MB of flash, this means that the whole memory is mapped to a continuous address space.
Microcontroller boards based on the \textit{ESP8266} MCU are mostly following the same layout: the MCU is attached to a flash chip which contains the bootloader, firmware and other application data.
The memory mapping mechanism of the MCU allows only a single page of 1 MB of flash to be mapped at the same time\autocite{ESP8266_Memory_Map} and the selected range must be aligned to 1 MB blocks.
As the \textit{ESP-01s} is only equipped with 1 MB of flash, this means that the whole memory is mapped to a contiguous address space.
Therefore, the second ROM can not be re-mapped to have the same start address as the first ROM.
While the firmware is executed without any dynamic linking mechanism and the chip does not support position independent code, the addresses used in the ROMs are dependent to the offset at which the firmware is stored.
This arises the need of building two firmware binaries, one for each target location.
To do so, a linker script for each of the two ROM slots was created which are used to create two variations of the same firmware which only differ in ROM placement.
The two resulting firmware binaries files are both provided for download via HTTP 1.1 - which of these files to download is selected by the device during the download process depending on the target ROM slot.
Listing~\ref{lst:linker_script} shows the only difference between the two linker scripts, whereas \texttt{\${SLOT}} must be replaced with the slot number according to the current build.
This arises the need for building two firmware binaries, one for each target location.
To do so, a linker script for each of the two ROM slots was created, which is used to create two variations of the same firmware, only differing in ROM placement.
The two resulting firmware binary files are both provided for download via HTTP 1.1 - which one to download depends on the target ROM slot and is selected by the device during the update process.
Listing~\ref{lst:linker_script} shows the only difference between the two linker scripts, where \texttt{\${SLOT}} must be replaced with the slot number according to the current build.
In addition to the two ROMs, the flash must provide room for the bootloader and its configuration.
\textit{rBoot}\autocite{rBoot} has been choosen as it is integrated within the \textit{Sming} framework and allows to boot to multiple ROMs.
For configuration, a \textit{rBoot} specific structure is placed in the flash at a well-known location directly after the space reserved for the bootloader code.
For configuration, an \textit{rBoot} specific structure is placed in the flash at a well-known location directly after the space reserved for the bootloader code.
This structure contains, among other things, the target offsets for all known ROMs and the number of the ROM to boot from on next reboot.
The full memory layout of this approach is shown in figure~\ref{fig:memory_layout}.
To calculate the origin of application data for each slot, the available memory of 1 MB is split in half and an offset of the size of bootloader code and its configuration (0x2000 bytes) is added.
For alignment and easy debugging, the second block is also shifted by the same amount as the first block.
This unused gap of 8192 bytes is used by some applications to store persistent data which can persist over application updates.
To calculate the origin of application data for each slot, the available memory of 1 MB is split in half and an offset of the size of the bootloader code and its configuration (0x2000 bytes) is added.
For alignment and easy debugging, the second block is also shifted by the same amount ob bytes as the first block.
The unused gap of 8192 bytes is used by some applications to store data which can persist over application updates.
\begin{figure}[h]
\begin{lstlisting}
@ -93,16 +93,16 @@ if (bootconf.current_rom == 0) {
\label{lst:choosing_rom}
\end{figure}
For installing a firmware update, the new firmware binary file is downloaded via a HTTP GET request.
The update server provides these files in the exact same way as it provides the version identifier files but only the path pattern differs: the suffixes \texttt{.rom0} and \texttt{.rom1} are used to provide the firmware binary files for the first and second slot correspondingly.
These provided firmware file are the exact same one as used to initially flash the chip for the according version.
Using the same file for flashing and updating allows better debugging by eliminating errors related to the update process and makes development and initial installation easy.
Listing~\ref{lst:choosing_rom} shows the algorithm used to determine the download address and reconfiguring the bootloader.
For installing a firmware update, the new firmware binary file is downloaded using an HTTP GET request.
The update server provides these files in the exact same way as it provides the version identifier files, but the path pattern differs: the suffixes \texttt{.rom0} and \texttt{.rom1} are used to provide the firmware binary files for the first and second slot respectively.
The firmware files provided on the update server are the exact same ones as used to initially flash the chip for the according version.
Using the same files for flashing and updating allows better debugging by eliminating errors related to the update process itself and makes development and initial installation very easy.
Listing~\ref{lst:choosing_rom} shows the algorithm used to determine the download address and reconfigure the bootloader.
After the download of a new ROM has been finished successfully, the bootloader configuration is altered to boot to the new ROM slot and the device is rebooted.
\subsubsection{Publish device information}
For monitoring and maintenance purposes, each device publishes a set of information to a well-known \textit{MQTT} topic after connecting to the network.
Beside already existing dates like device type, chip and flash ID the information block has been extended with details about the bootloader, SDK and firmware version as well as relevant details from the bootloader configuration like the currently booted ROM slot and the default ROM slot to boot.
Beside already existing dates like device type, chip and flash ID, the information block has been extended with details about the bootloader, SDK and firmware version as well as relevant details from the bootloader configuration, like the currently booted ROM slot and the default ROM slot to boot from.
This allows administrators to find devices with outdated bootloaders and helps to find missing and failed updates.

22
4-2-multidevice_build_structure.tex

@ -4,19 +4,19 @@ The firmware for all \textit{ESP8266} based devices in the hackerspace are all b
\textit{Sming} provides the base library for this framework.
In addition, components and functionality shared by all devices has been identified and are providing a framework for the existing and possible further devices.
This framework provides a functional base for all devices and allows to reuse code providing functionality which is common in multiple devices.
The framework also includes a build system which allows to configure some basic parameters for all devices.
The framework also includes a build system, which allows to configure some basic parameters for all devices.
Including, but not limited to, the Wi-Fi access parameters, the \textit{MQTT} connection settings and the updater URLs.
By sharing the same code, all devices ensure to have a common behavior when it comes to reporting the device status or interacting with the home-automation controller.
This eases configuration and allows to collect information about all devices at a central point.
This eases configuration and allows to collect information about all devices at a central location.
Each device firmware exists as a separate project and includes a link to the framework.
As development on these devices happens in cycles, older projects are missing updates of the framework and there do not benefit from added features or fixed problems.
Updating the framework version and rebuilding the firmware would often result in an easy gain of these benefits but requires manual interaction.
More problems will arise if the application programming interface (API) of the framework has changed.
Then the device firmware must be updated to use the changed API which can be an unpleasant and complex task and leads to higher latency for firmware updates.
As development on these devices happens in cycles, older projects are missing updates of the framework and therefor do not benefit from newly added features or fixed problems.
Updating the framework version and rebuilding the firmware would often result in an easy gain of these benefits, but requires manual interaction.
More problems will arise if the application programming interface (API) of the framework changes.
In this situation, the device firmware must be updated to use the changed API, which can be an unpleasant and complex task and leads to higher latency for firmware updates.
To prevent these problems the device firmware of all devices in the space is now integrated with the framework into a larger project.
By doing so, the each device specific code is always linked to the latest version of the framework.
To prevent these problems the device firmware of all devices in the hackerspace is now integrated with the framework into a larger project.
By doing so, each device specific code is always linked to the latest version of the framework.
Device specific code is now organized as a folder for each device type.
The build system has been modified to scan for all device specific folders and call the original build process for each of them.
@ -24,7 +24,7 @@ The build system has been modified to scan for all device specific folders and c
\subsubsection{Framework integration}
The framework has been changed to keep control over the application life-cycle.
It ensures that the device unspecific code is executed at the right time and provide an API for device specific functionality.
It ensures that the device unspecific code is executed at the right time and provides an API for device specific functionality.
\begin{figure}[h]
\begin{lstlisting}
@ -46,11 +46,11 @@ Device* getDevice() {
\end{figure}
The framework specifies a simple interface which must be implemented by each device.
The framework specifies a simple interface, which must be implemented by each device.
A single function \texttt{Device* getDevice()} must be defined exactly once in each device specific folder.
To implement this interface, a static instance of \texttt{Device} is created and returned.
Each \texttt{Device} is populated with device specific \texttt{Feature} instances.
While the \texttt{Feature}-API leverages common runtime polymorphism to share functionality between features, the initial \texttt{Device} creation uses compile-time polymorphism which reduces the need of memory management and increases performance by avoiding virtual function tables.
While the \texttt{Feature}-API leverages common runtime polymorphism to share functionality between features, the initial \texttt{Device} creation uses compile-time polymorphism which reduces the need for memory management and increases performance by avoiding virtual function tables.
Listing~\ref{lst:create_device_socket} shows the complete device specific code used for a simple power socket.

9
4-3-automatic-deployment.tex

@ -2,7 +2,7 @@
The source code of the \textit{ESPer} project is published into a \textit{GIT} source code repository which is provided by the hackerspace.
To avoid interferences between different build environments on developers computers and roll out new versions as early as possible, the code has been integrated into a continuous integration (CI) system.
The CI based on \textit{drone}\autocite{drone}, provided as part of the hackerspace infrastructure, allows to execute commands on each version published into the \textit{GIT} repository.
The CI, which is based on \textit{drone}\autocite{drone} and provided as part of the hackerspace infrastructure, allows to execute commands on each version published into the \textit{GIT} repository.
Therefore a \textit{drone} configuration file as shown in Listing~\ref{lst:drone} has been added to the source code as \texttt{.drone.yml}.
\begin{figure}[h]
@ -30,14 +30,15 @@ publish:
\label{lst:drone}
\end{figure}
\begin{sloppypar}
As shown in the configuration Snippet, the build environment includes some special settings.
First, the \texttt{CONFIG=maglab} option let the build system use \texttt{Configurion.mk.maglab} instead of the default configuration file.
First, the \texttt{CONFIG=maglab} option lets the build system use \texttt{Configurion.mk.maglab} instead of the default configuration file.
This configuration file is stored inside the repository, too.
To keep the WiFi password secret, it is not written down in the configuration but must be specified in the environment.
To keep the WiFi password secret, it is not written down in the configuration, but must be specified in the environment.
To include secrets into a build process while allowing to keep the configuration public, \textit{drone} allows to encrypt these with a repository specific key.
Using this method, the password is stored as \texttt{.drone.sec} file inside the repository from where it is injected into the build environment.
At last, the firmware version is configured to be made out of the first 8 letters of the \textit{GIT} commit hash, which uniquely identifies a version of the source code.
\end{sloppypar}
For deployment, only the master branch is considered.
After a successful build, all distribution files (the binary firmware files and the version files) of all devices are copied to the machine running the home automation controller software into a directory served by a HTTP server.
The used configuration file references this server as source of updates.

6
4-implementation.tex

@ -1,11 +1,11 @@
\section{Implementation}
Implementing \textit{OTA} updates under the given requirements involves three different components which interact closely.
Implementing \textit{OTA} updates under the given requirements involves three different components, which interact closely.
The first component implements the update mechanism on the firmware running on the embedded device.
The first component implements the update mechanism in the firmware running on the embedded device.
It is responsible for checking and downloading the updates and installing them.
Second, the build system is in charge of building the firmware from source and publishing the built binary images.
At last the deployment provides infrastructure for downloading the binary firmware images and triggering the update on all devices.
Finally, the deployment provides infrastructure for downloading the binary firmware images and triggering the update on all devices.
\input{4-1-update_mechanism}
\input{4-2-multidevice_build_structure}

3
5-conclusion.tex

@ -2,7 +2,8 @@
The project has been successfully deployed in the hackerspace and is now an essential part of home-automation development and deployment.
The update infrastructure has been the crucial point for decisions to wards the framework for most members.
The update infrastructure has been the crucial point for decisions towards the framework for most members.
% Next one is a strange sentence ...
Allowing to do updates and the shared configuration and behavior results in a massive speedup when it comes to project deployment.
Before that, the cost for an update was estimated so high, that most projects tend to delay deployment until all required and wanted features are implemented.
Now, as the devices are deployed as soon as the hardware is considered stable, these devices start to provide functionality early and therefore the developers can get better feedback on the provided functionality.

BIN
index.pdf

Loading…
Cancel
Save