Unit Testing an Embedded System
Most developers use unit tests to create a better and more robust application. They act as a regression set and allow to modify parts of the application with good knowledge of where the modification has impact. If done properly, the unit tests will cause the various components of the application to be written without too much coupling, which makes implementing new functionality and other changes easier. But how about an embedded platform, think bare-metal on a microcontroller, like a Cortex-M4? How to write unit tests for code that does not run natively on the PC?
The trick here is: don’t. Don’t try to run them natively, but on the desktop PC instead. For example: we use the Google Test framework to create unit tests for our C++ (and a little C) application on our embedded Cortex-M4 board. They are build and executed on the PC and the reporting is done here as well. The end result is mostly the same: the unit tests allow for change to happen without much interference to the entire application.
But how? The Google Test framework does not run on the Cortex-M4, so how to use it here? For this some interfaces are created to split the low level peripheral drivers from the application code (like the SPI, UART and I2C drivers). This is roughly on the level of: “Write buffer X of size Y” and: “Read into buffer P with size Q”. By using mock objects the calls to/from this interface can be tested, even the contents. A similar thing is done for platform specific libraries, these will get a dummy implementation (for instance for a ‘delay_ms()‘ or an ARM specific ‘_NOP()‘). This leaves the application code to be tested – on smaller project with FreeRTOS and LoRa libraries, the coverage was ~50% of the ~5000 lines of code, the bigger the codebase the larger the portion that can be tested.
Unit tests are compiled and run on the PC platform – in our case Windows. This results in an executable which can be debugged, complete with all bells and whistles from within an IDE (like Visual Studio Code). This is an important step as it removes a mental barrier for developers for starting to write a unit test: once a few exist more tests will follow. The unit tests are build using CMake and Ninja – allowing it to run on a buildserver like Jenkins as well. The compiler (tool-chain) used is GCC v8.3.0. The unit test executable is 32 bit, the same as on the Cortex-M4, and has debug symbols to allow debugging with GDB (from within the IDE).
The application code is build and unit tested on the buildserver as well, here on Linux. The use of yet another OS may lead to some portability issues – so far they have been very sparse and relatively easy to resolve. One thing to note is these become easier to resolve if the compiler version is the same.
Typically ‘fragments of functionality’ are tested. In C++ this often means testing a single class, via an interface. New functionality (ideally) will get a unit test before it is added – both to act as regression test and as documentation of sorts. Creating a unit test will force the developer to think about his/her code making it more testable, thus creating more robust code.
For already existing code a different technique can be used. Often it means starting at the edges of an application or component and working your way inward to make things testable. Sometimes we can choose a larger separated block of functionality to refactor. As example: an application contained a block of functionality all related to NAND flash and storage: logging, recordings and some persistent data was handled via a single class. It also held some logic to interact with a bought-in component to deal with lower level read/write commands for the NAND flash. To make this large block testable it was split into clearly defined functionality (on paper first!) before it was done in code (using interfaces). Per split-off block of functionality an interface was created, which then received a bunch of unit tests. This resulted in looser coupling of these classes, easier and more thorough testing of the functionality and due to the testing some performance improvements could be made as redundant calls to methods were found and eliminated. A lot of uncertainty about the code was removed and replaced with tested confidence, which made testing the entire application (in an integration or system test) easier as this functionality could be left out when looking for issues (until otherwise proven they originated from this block).
Creating unit tests has a number of advantages:
- The code becomes better testable, and since it is better tested it also becomes more robust.
- The tests in itself become an example on how to use the code, sort of live and updated documentation.
- They form a safety net when restructuring code, functionality which should not change is spotted early on. It prevents changes to ‘ripple-through’ and affect unrelated code.
- Making code more testable creates a looser coupling between components. This makes making modifications (like adding features) easier.
- Making legacy code unit testable uncovers issues not found before.
- Usually not that critical, it also makes performance enhancements possible as the order of calls and the number of times a call is made can be tested.
- Tests can be added when a piece of functionality is changed or created. Instead of testing afterwards the tests are done at the same time or before adding functionality. They provide a check to see functionality is correctly implemented and act as a regression test to keep it so.
Some things like low level peripheral drivers requiring actual hardware cannot easily be unit tested. Typically these are tested manually, sometimes using an oscillator or logic analyser. They tend not to change much which makes this a feasible way of testing them. Time critical components are also hard to unit test – for these we can either test them manually, or use an automated system test which indirectly tests these functionality. A practical implementation would be to ‘force’ the system to make a recording for some time – preferably with known input signals (or let the device create simulated data) – and retrieve and validate this recording later. By stressing the system during this process potential timing issues are more likely to occur.
So much is possible, what is keeping you?