Firmware Testing with Renode and GitHub Actions | Interrupt

Blog
1

Automated firmware testing on real embedded hardware is what every hardware company strives to build. The mission is to repeatedly verify that changes to firmware being merged into master and shipped to customers will not cause catastrophic failures like power regressions or bootloops.

This is a companion discussion topic for the original entry at https://interrupt.memfault.com/blog/test-automation-renode
2

Hi Tyler,
What is the advantage (if any) of tests on emulated hardware over the unit tests (e.g. using CppUTest library) ?

Thank you

3

Difficult question. I believe that using CppUTest and unit tests on host will enable a complex firmware to achieve a test coverage in the 70-80% range, which is quite good.

Using an emulator such as QEMU or Renode will allow teams to test another layer or two of their software stack, including drivers written on top of the MCU peripherals and other devices connected through i2c, SPI, etc. Emulators more closely resemble the real firmware, so if it’s a bug on the emulator, it’s likely a bug on the real device.

I can see the following benefits of testing with an emulator:

  • Can compile in native 32bit instead of 64bit, which is becoming more common
  • Can test logic which directly deals with assembly code, registers, and MCU specific behavior
  • Can measure stack sizes in tests to confirm they align with expectations
  • With Renode, you can spawn multiple emulators/MCU’s and have them interact over various protocols. This seems to be the initial goal Renode was trying to achieve.

It’s worth speaking to the drawbacks as well

  • Slower iteration cycle for tests. Renode took about 1-2s per test, where as unit tests take milliseconds to run.
  • Non-parallel testing, and more complex CI setup to run tests. Debugging also becomes a little more difficult, but you still have access to a debugger at least.
  • Allows for poorly abstracted code that ties itself to a single MCU
  • Up-front work for benefits that may or may not be worth it, depending on the complexity of the project.

When we had ported the Pebble firmware to QEMU, we had to build a number of alternative drivers and service layers for use with QEMU. In the end, it was immensely valuable to be able to test an “almost-real” device, but it took a lot of work to get there. We put in the effort because it allowed third-party developers to build Pebble applications without actually having real hardware or a test board.

If the goal is just to run the same unit tests on the emulator that you’re currently running within CppUTest on the host, I’d say it’s not worth the hassle. If you plan on testing an extra layer or two with the in-emulator tests, then I would say it’s worth the investment. If you ultimately want to enable normal development using an emulator instead of needing hardware, then it’s definitely worth the time investment.