Embedded C/C++ Unit Testing Basics | Interrupt

Blog
1

Complex embedded projects have thousands and frequently tens of thousands lines of code. The majority of that code is entirely software (rather than “firmware”), and software in every industry is typically unit tested. However, in the embedded and firmware industry, unit testing is typically an after-thought or a task that is begun after working on a project for months or even years.

This is a companion discussion topic for the original entry at https://interrupt.memfault.com/blog/unit-testing-basics
2

Nice blog post, Tyler! One C unit test framework I enjoyed using at Vaunt which addresses the issue you mentioned of state leaking from one unit test to the next is Criterion (https://github.com/Snaipe/Criterion). It seamlessly runs each unit test in a separate process to help isolate them from each other, and it also supports pure C tests so no need for wrapping includes in externs. :grinning:

3

One C unit test framework I enjoyed using at Vaunt which addresses the issue you mentioned of state leaking from one unit test to the next is Criterion

Thanks for the tip! It looks quite good and would definitely help out with the state bubbling into other tests.

4

Hey Tyler, the examples relies on a x86 port of littlefs. I was thinking, isn’t this functionally equivalent to a “fake”? It’s a piece of code that compiles on x86 and has the same interface as what we use on the embedded target.

5

You are right. The x86 port of littlefs can be considered a fake. The port is mostly contained within the emubd module, which translates the littlefs API calls host filesystem calls.

6

Thanks for the clarification! Can you also give recommendations on how to organize the embedded target directory along side the test directory?

For the test directory, you suggested something like:

test
|— mock
|— fake
|— src
|— header_overrides
|— makefiles
|— (etc)

Should the embedded target code live in a neighbouring directory to test? E.g.

MyProj
|— embedded
| |— mutex // e.g. mutex/mutex.h from your examples
| |— src
| |— inc // e.g. kv_store.h from your examples
|
|— Test

Should there be a top level Makefile under MyProj? How can we elegantly handling using 2 different compilers in a project (e.g. arm-non-eabi-gcc for embedded and native x86 compiler for cpputest)?

7

The way I’ve seen it done in the past is to have separate Makefile’s for the ARM build and another for the unit test build. This would require the following then:

cd embedded && make -j4
# and
cd test && make -j4

If you did want a top level Makefile to control all of these sub Makefiles, something like the following would work, but I don’t usually do this…it makes wrangling all of the options and argument handling more difficult and confusing.

test:
    $(MAKE) -C test/

embedded:
    $(MAKE) -C embedded/

If you do opt for separate sub-directories and separate Makefiles, I usually use something like Invoke to wrap everything using the strategies discussed in another post of mine

8

I’m doing testing with CppUTest, but I’m encountering an issue with the device drivers, which try to call some assembly instructions and fail since we’re on an x86 machine rather than the MCU. What is the proper way around this? I need to keep the driver headers included since I use some of the typedefs in them.

9

Hey @danielhep! Great question.

What I’ve used in the past is to create a header_overrides directory where I have my mocks, stubs, and fakes directory. For reference, this section mentions it (briefly) Embedded C/C++ Unit Testing Basics | Interrupt.

Place any header in that folder, with the structure you are trying to match, and then include that directory before any of the include paths you are trying to override. This will ensure that CppUTest picks up your overridden header first before the MCU one.

For example, if you are trying to override a fake header that is included in the form of

#include stm32/inc/stm32f4xx.h

then make this the structure within header_overrides

$ tree header_overrides
header_overrides
└── stm32
    └── inc
        └── stm32f4xx.h

EDIT: I missed your last sentence:

I need to keep the driver headers included since I use some of the typedefs in them.

My solution has always been to redefine the typedefs in the override header. If things change, the unit tests will just break but that’s usually fine. The worst would be if the overridden header changed and things didn’t break.

10

Hi, i found the acronym of test driven development is incorrect:

Through proper use of unit tests, and especially while using practices from Test Driven Development "(TTD)"1

which should be “TDD”.

11

Woops, thanks for the report. Just pushed a fix. https://github.com/memfault/interrupt/commit/b64d3b0309b61f94894d04359f2aadf955a0500f

12

Hello.

First thanks for your articles on this subject. I’m an embedded developer recently found that I’m missing unit testing to make better code without bugs or the painful task of debug code.

I found your articles, and they are well explained, with good examples, and I was convinced.

However, I would like to make a small question. I found that your way of organize files was easy to understand and an excelent approach. However, on the cpputest site they point to a repo (GitHub - jwgrenning/cpputest-starter-project: gcc cpputest starter project, with instructions to help get your legacy code into cpputest for the first time) that is from the author of the renamed book on this subject. I found that file organization a little more hard to understand, and it does not seem to make use of separate makefiles for different tests, as you do.

I know I’m starting this up, and I’m reading that book, so maybe I’ll understand more later, but can you tell what advantage that way of structure has? Or do you find your way better?

Sorry for the long post, and reactivation of this topic.
Thanks for all your help.

13

I’m glad you liked the post! And no worries on commenting on an old post, this post continues to be updated as I find new information, as do all of our posts here. Comments are always encouraged.

I know I’m starting this up, and I’m reading that book, so maybe I’ll understand more later, but can you tell what advantage that way of structure has? Or do you find your way better?

The example repo you linked assumes that you’ll only need one compilation script and one compiled unit which runs all of your tests, which I’ve found to only handle the simplest of cases.

Let’s say you are testing a bunch of different pieces of your system, and we have this analytics module that is used in a number of places throughout the code base. In other words, our project is more than a single .c file.

You are going to want to stub out the analytics module in most of your tests, probably fake it out in a few of them, maybe use a mock in a few more, and actually use the real analytics.c subsystem in a single unit test. Since you can only compile in a single mock/fake/mock/implementation for a single module, you need to choose which one you include for which test.

To satisfy each of these requirements, you need to compile three different binaries and run them each separately. That is the primary reason for using multiple Makefiles.

I prefer the structure I’ve used in the Interrupt post example, as it’s the structure I’ve used at previous companies and used in complex projects with over 100 tests and 100+ modules as well.

14

Thank you for your quick answer.

That’s what I was thinking, I like your structure because you create a kind of a sandbox on a test. You can test each .c module with what stubs, fakes and mocks you want, that are enougth to test that module, without any kind of interference. The other structure seems that will always be a kind of integration test, that you can test any functions, but most of them with all other modules in the compilation.

I’ll dive more into this subject, but thanks for all your help.

15

@tyler At the beginning you made the distinction between testing ‘embedded software’ and ‘firmware hardware and drivers’. By ‘embedded software’ do you mean a project that involves some sort of kernel e.g. ChibiOS or linux? By ‘firmware hardware and drivers’ do you mean bare metal?
I’ve read recent Memfault posts that use Renode to test. Would you use unit test frameworks like CMocka for ‘embedded software’ and Renode’s Robot framework for ‘firmware and hardware drivers’? I’m a bit confused as to what should be used when.
Thanks

16

Hi, thank you for post.

How do you test static functions?

17

By ‘embedded software’ do you mean a project that involves some sort of kernel e.g. ChibiOS or linux? By ‘firmware hardware and drivers’ do you mean bare metal?
I’ve read recent Memfault posts that use Renode to test. Would you use unit test frameworks like CMocka for ‘embedded software’ and Renode’s Robot framework for ‘firmware and hardware drivers’?

It’s a good question, and is one that is generally a gray area. I’ll do my best to answer.

ChibiOS and Linux would be very hard items to test themselves, as they are built upon event loops, interrupts, likely close hardware interaction, and when you include a single source file of ChibiOS, you likely need to pull in the entire project.

What I typically mean by embedded software, is the layers on top of the hardware and OS.

  • This is your higher level flash driver that dispatches larges reads and writes to multiple individual flash sector (or subsector) reads and writes (where the flash chip itself is faked out).
  • Filesystem, and then any module built upon the filesystem (ideally with the filesystem faked out)
  • Graphics stack and Bluetooth communication protocols (what responses are expected given a certain request)

These modules that your unit testing should also have small number of dependencies. They should take in inputs and spit out outputs, and that is what is tested.

By firmware hardware, and drivers, yes, I typically mean bare metal in these situations. I have found this type of code very difficult to test in unit tests, and it’s usually more work than it’s worth. I’d probably even push for flashing specific firmware for on-device validation tests (simple ones though).

With Renode, it does open up the possibility to test lower level bits of your stack! I would still say it’s probably not worth it. Once an OS, driver, or hardware interaction module works, it works. It doesn’t contain more bugs, and the bugs that do remain always seem to be timing bugs and/or undocumented “features” from the manufacturer, which Renode or QEMU might not even help out much with.

With Renode or QEMU, I’d focus more towards integration tests, which pull in almost the entire firmware and run that as a whole with a few of the lower-level modules mocked/faked out (like the communication stack. Make it work over USB).

CMocka for ‘embedded software’ and Renode’s Robot framework for ‘firmware and hardware drivers’?

Yes, that sounds right.

1 Like
18

Oops, I think I meant to add this to the post!

I typically define something like the following and then use this instead of static in my codebase for those functions that need to be unit tested.

#if UNIT_TESTING
#define TEST_STATIC 
#else
#define TEST_STATIC static
#endif

Then, for unit testing, TEST_STATIC becomes a no-op and your functions become global, but for normal firmware builds, static is inserted.

19

Thank you for response. And what do you think about including C file with modules under test into module with test cases. For example:
https://github.com/whot/unit-tests-for-static-functions

20

Thanks for the link! I have never seen anyone try this this way.

Honestly though, looks like a huge hack (and is mentioned that it is one), and I am pretty confident you’ll fight it more than it helps when you work on larger projects.

The strategy I mentioned was used at Pebble when testing our firmware, and the project itself was hundreds of files. It worked well, and I wouldn’t try to fix something that isn’t broken.