Review: Test Driven Development for Embedded C, James W. Grenning

The TL;DR:

• Test Driven Development for Embedded C by James W. Grenning is an outstanding book. 
• The title says C, but if you work in C, C++, C#, Go, Objective-C, Java, Javascript, or anything else, this is worth reading.
• It says embedded, but if you work in embedded systems, front end web apps, mobile apps, desktop apps, backend servers, or anything else, this is worth reading.
• And it's not just TDD, it's all the concepts that go into good design.
• Get it, read it, USE it. You won't regret it.

Background

I first learned about XP (eXtreme Programming) concepts in 2007, when I was introduced to Kent Beck's Test-Driven Development: By Example. I used TDD (Test-Driven Development) to develop a major component on a server system. I learned more in 2013, when I read Michael Feathers' Working Effectively With Legacy Code. I used that to apply TDD to an existing server codebase.

Over the past 3 months, I've been on a reading binge, triggered by reading Robert C. Martin's 2017 book Clean Architecture: A Craftsman's Guide to Software Structure and Design. I have an hour-long commuter rail ride, so I have lots of time to read and work on my laptop, plus a little lunchtime reading, and I always have a book open at home.

I read his Clean Code: A Handbook of Agile Software Craftsmanship, The Clean Coder: A Code of Conduct for Professional Programmers, and am currently in the middle of his Agile Software Development, Principles, Patterns, and Practices.

I read Sandro Mancuso's The Software Craftsman: Professionalism, Pragmatism, Pride, and am in the middle of Mike Cohn's Agile Estimating and Planning, both from Martin's series.

I read Andrew Hunt and David Thomas' The Pragmatic Programmer: From Journeyman to Master, and am halfway through Pete McBreen's Software Craftsmanship: The New Imperative; Martin Fowler's Refactoring: Improving the Design of Existing Code is waiting on the shelf.

I've encountered bits and pieces of this material over the years, but this was a chance to go back to primary sources, get the full details and parts I've missed out on, and really understand them. I highly recommend it.

Review

But maybe you don't have time for all that. Maybe you'd like to cut to the chase and see how to apply their principles in practice.

Test Driven Development for Embedded C by James W. Grenning does that. It draws from many of those sources and more, showing you real-world examples to put them into practice.

Grenning is one of the original authors of the Agile Manifesto (as are Beck, Fowler, Hunt, Martin, and Thomas). He contributed the chapter "Clean Embedded Architecture" to Clean Architecture, and is the inventor of the Agile planning poker estimation method.

The book was published in 2011, so is now 7 years old, but it remains as timely as ever. That's especially true as IoT vastly expands the number of embedded systems that we rely on in our daily lives. Effective testing is critically important. For instance, see Testing Is How You Avoid Looking Stupid.

If you work on embedded systems in C, this is a must read.

If you work in a different language besides C, or on a different type of system than embedded systems, you may not think that a book on embedded C programming applies to you. But it's broadly applicable and worth reading.

The book is organized as an introductory chapter, the remaining chapters grouped into 3 parts, and appendices. I see it as three distinct portions, plus appendices: Chapter 1; Parts I and II (chapters 2-10); and Part III (chapters 11-15).

Throughout, Grenning addresses the common concerns people have with applying TDD to embedded systems. Embedded systems are a particular challenge, with particular target system constraints, so people might be skeptical.

This is a very hands-on, how-to book. I've included a number of lists from it, including those Grenning draws from other sources, because they illustrate the practical, pragmatic, disciplined approach. You can use this as a cheat sheet to remember them after you've read the book.

It might be tempting to think you can get by just with the information I've provided here and skip the book. But I've included it specifically with the hope that you'll realize you must read the book, and that it will be a worthwhile investment.

First Portion

This is the motivational portion, the appetizer. Grenning introduces TDD, its benefits in general, and the specific benefits for embedded systems.

He lists Kent Beck's TDD microcycle:

1. Add a small test.
2. Run all the tests and see the new one fail, maybe not even compile.
3. Make the small changes needed to pass the test.
4. Run all the tests and see the new one pass.
5. Refactor to remove duplication and improve expressiveness.

The microcycle is critically important to the technique, so Grenning reminds you of it several times as he works through examples. This is what makes TDD effective, and I know from my own experience is also what makes it fun and extremely satisfying. He has a sidebar titled "Red-Green-Refactor and Pavlov's Programmer", which is very apt. That Pavlovian drive to take the next step in the cycle draws you into the zone and keeps you cranking.

For embedded systems, in addition to all the benefits that apply to other types of software, the primary benefits include being able to develop tested, working code when the target hardware isn't available; being able to test off-target (i.e. not on the target embedded system), where you have all the benefits of a general-purpose system and none of the constraints of an embedded one, including speed of development turnaround cycle; being able to isolate hardware/software interactions; and decoupling software from hardware dependencies.

That last point is part of the Big Lesson (see below) from all this. TDD in general, for any type of software, results in testable and tested software. But more than that, it drives development in a way that improves the design significantly.

That improved design means a much longer and happier life for the software and the systems that use it. They will be able to adapt to changes much more easily. It's not just about getting V1.0 done. It's about getting to V10.0.

In Software Craftsmanship, Pete McBreen starts off with the origin of the term software engineering. It was coined by a 1967 NATO study group working on "the problems of software." A 1968 NATO conference identified a software crisis and suggested that software engineering was the best way out of that crisis. They were concerned with very large defense systems. McBreen gives the example of the SAFEGUARD Ballistic Missile Defense System, developed from 1969 through 1975.

He says, "These really large projects are really systems engineering projects. They are combined hardware and software projects in which the hardware is being developed in conjunction with the software. A defining characteristic of this type of project is that initially the software developers have to wait for the hardware, and then by the end of the project the hardware people are waiting for the software. Software engineering grew up out of this paradox."

McBreen is questioning the value of that style of large-scale software engineering in the development of commercial products, suggesting that a different approach is needed.

But doesn't that situation sound familiar? Doesn't that sound like the problem embedded systems developers face all the time, that Grenning is addressing? This was a situation where TDD and off-target testing could have significantly alleviated the software crisis.

Granted, it was more complicated, since they were also developing the very processors and programming languages they would use, while modern systems rely on COTS (Commercial Off The Shelf) processors and languages. But we see that this has been a pervasive problem for some 50 years.

All types of systems, from embedded to frontend mobile apps to high-scale backend servers, in all those languages, from C to C++, Objective-C, Go, Java, Javascript, etc., can benefit.

All that code can be removed from its normal production environment and run off-target, off-platform, in a unit test environment that allows you to exercise every code path you want easily and quickly. That includes the obscure dark corners of the code trying to handle unusual error cases that are hard to produce on the target system.

For some of my own experience testing off-target, see Off-Target Testing And TDD For Embedded Systems.

Second Portion

This portion is the meat of the book, applying TDD to real-world embedded development and going through the mechanics with practical examples.

Following the lead of Martin's book, Grenning makes restrained use of UML diagrams. While some people dislike UML because they associate it with the heavyweight BDUF (Big Design Up Front) software engineering methodologies that McBreen was talking about, this is a very effective use of it that communicates information quickly. Which is the whole point of UML.

Grenning presents two unit test harnesses, Unity and CppUTest (of which he is one of the authors). All of the material applies just as well to other test harness tools, such as Google Test/Google Mock. It's equally applicable to other languages and their language-specific test harnesses.

He uses Gerard Meszaros' Four-Phase Test pattern to structure tests:

• Setup: Establish the preconditions to the test.
• Exercise: Do something to the system.
• Verify: Check the expected outcome.
• Cleanup: Return the system under test to its initial state after the test.

The rubber meets the road in his five examples of using TDD to develop embedded code:

• LED driver
• Light scheduler for a home automation system
• Circular buffer
• Flash driver for ST Microelectronics 16 Mb flash memory device
• OS isolation layer (aka OSAL, OS Abstration Layer) for Linux/POSIX, Micrium uC/OS-III, and Win32 (this is actually an appendix and only covers thread control, but establishes the pattern)

Clearly, these have real hardware dependencies on both the processor I/O interface and the attached devices, as well as the system clock, and real OS dependencies. Those are critical concerns for the embedded developer. The LED driver is very simple behavior, so makes for a gentle introduction. The others are more complex.

Grenning discusses driver requirements, then shows the initial tests and code. Notice I said tests first. That's an important concept in TDD. You always write the test first, that uses the code in the way that you want the code to work. Then you write the code that satisfies that usage. He emphasizes the save-make-run cycle that you do repeatedly during this process. Then you repeat for the next test and bit of code. That's how you make fast progress.

The key concept is faking out portions of the system, so that the Code Under Test (CUT) can run as if it was running on the real system. That's critical for making TDD work off-target and off-platform. There are several strategies for doing this. In the case of the LED driver, he uses virtual registers to simulate memory-mapped I/O. This is simply a variable under the control of the test suite.

He also talks about test-driving the interface before test-driving the internals. That's another critical concept, integral to the whole design process. That's design-for-change. Because things will change. A product with a long, useful life, that represents an ongoing revenue stream for a company, will change over that time to adapt to changes in underlying technologies, user requirements, and usage. TDD means you can make changes without fear of breaking things (because you'll find and fix breakage as a result of performing the microcycle).

He talks about the strategy of incremental progress and refactoring as you go. This is in the heat of development. Final code does not flow directly from your fingertips. It evolves in incremental steps as you work. Did you ever look at someone's code and marvel at how clean and easy to follow it was, despite the complexity of the job it was achieving? You might think you could never do something that easily. This process results in that kind of code. Like a novelist in the heat of writing a scene, the first draft is never the final product, and the story arc evolves over time.

This is where he covers several important guidelines for driving the TDD process effectively. He lists Robert Martin's Three Laws of TDD:

• Do not write production code unless it is to make a failing unit test pass.
• Do not write more of a unit test than is sufficient to fail, and build failures are failures.
• Do now write more production code than is sufficient to pass the one failing unit test.

He describes Kent Beck's snappy acronym DTSTTCPW: Do The Simplest Thing That Could Possibly Work, which initially means just faking it (for instance, hard code a function to return false in order to get the test that uses it to pass). Then keep tests small and focused, and refactor on green (many unit test setups show a failing result in red, and a passing result in green).

As this evolves, the faked out code turns into real code (the hard coded false is changed to actual code that does something and returns true or false under the appropriate conditions). That builds out a verified test suite as it builds out verified code.

This leads to the TDD State Machine, which tells you what to do next. The guidelines above and the state machine take you through the mechanics of working in the TDD style. They answer the questions:

• How should you start?
• What should you do next?
• How do you know when you're done?

Whenever you write some production code, ask yourself, "Do you have a test for that?". If not, stop, go back, and write the missing test.

He also covers Dave Thomas and Andrew Hunt's DRY principle: Don't Repeat Yourself. This mantra helps drive the refactoring so that you keep the code lean and clean. I'll throw in additionally the DAMP principle: use Descriptive And Meaningful Phrases, a concept the book applies without calling out by name. This favors readable function and variable names that express intent over cryptic abbreviations and syntax. The result is code that reads with a narrative flow.

Keeping your code DRY and DAMP makes it easy for others to understand and modify (which might be you when you come back to it six months or a year later). This is the same as Beck's microcycle step 5.

To some degree this all turns TDD into a very mechanistic process. But that's a good thing. It's not a random, ad hoc process where you're constantly questioning yourself about what do to. Instead it's an orderly stepwise process that makes effective progress. You quickly see and appreciate the value.

It's also very fun and satisfying, because that mechanistic aspect actually drives your creativity. What's the next thing you can add to it? What's the next test, the next bit of functionality? When you finish, you feel like you've accomplished something, and you have the evidence to prove it. It's addicting.

That leads to Grenning's Embedded TDD Cycle, which starts with TDD on the development system, then advances to the target processor and eval hardware, then the actual target hardware:

• Stage 1: Write a unit test; make it pass; refactor. This is red-green-refactor, the TDD microcycle on the development platform.
• Stage 2: Compile unit tests for target processor. This is a build check that verifies toolchain compatibility.
• Stage 3: Run unit tests on the eval hardware or simulator.
• Stage 4: Run unit tests on target hardware.
• Stage 5: Run acceptance tests on target hardware. These are automated and manual tests of the integrated system.

This sequence gives you confidence in the code under test quickly, then you can address any hardware-dependent issues that start to arise, such as compiler, library, or primitive data type differences. Next you to start exercising the hardware-dependent code.

Testing separately on eval hardware and actual target hardware helps shake out hardware issues in the actual target, since the eval hardware is presumably known good. One of the challenges in embedded development is always trying to determine if problems are due to the software or due to the hardware, since both are in active development and haven't had much soak time to prove them out.

For the other TDD examples, Grenning goes through a progression of different collaborator strategies. These are the test doubles, the fakes, that are substitutable for real components. They stand in for those components to break the test dependencies and allow you to simulate and monitor interactions. An important point is that they are much lighter weight than full-scale simulators. Full simulators can themselves require significant development. These fakes have only enough behavior to support the tests (part of the DTSTTCPW mindset).

He uses these types of doubles:

• Spies
• Stubs
• Mocks
• Exploding fakes

He goes through the following substitution methods, showing how to do them and discussing when they are appropriate:

• Link-time substitution
• Function pointer substitution
• Preprocessor substitution
• Combined link-time and function pointer substitution

These are fully-worked-out examples, although he starts omitting intermediate steps as he progresses in the interest of brevity. All the code is available online.

Third Portion

This portion completes the meal, complementing the meat in the second portion. It addresses design issues. This is important because design for testability also means design for flexibility and long product life.

Grenning starts out with Martin's SOLID principles:

• S: Single Responsibility Principle (SRP)
• O: Open Closed Principle (OCP)
• L: Liskov Substitution Principle (LSP)
• I: Interface Segregation Principle (ISP)
• D: Dependency Inversion Principle (DIP)

He covers both how the previous chapters have incorporated these principles, and how to use them to guide the development process. TDD is closely intertwined with them.

Don't be put off by the apparent difference between non-object oriented and object-oriented languages. The specific language used is irrelevant. The syntactic mechanics may be different, but the concerns and concepts are all the same. C can be every bit as object-oriented as Java, it just takes a little more developer discipline. That means that all of the concepts of the various principles above apply.

He uses the SOLID principles in four module design models of increasing complexity, applicable in different embedded system design cases:

• Single-instance module: Encapsulates a module's internal state when only one instance of the module is needed.
• Multiple-instance module: Encapsulates a module's internal state and lets you create multiple instance of the module's data.
• Dynamic interface: Allows a module's interface functions to be assigned at runtime.
• Per-type dynamic interface: Allows multiple types of modules with the same interface to have unique interface functions.

You'll probably recognize more than one of these in the systems you work on. You may also recognize object-oriented concepts, and in fact he shows how to implement, use, and test a C++ virtual function table (vtable) in C.

Part of good design is adapting to change. He covers Martin Fowler's concepts of refactoring, both the code smells that point to things that need to be refactored, and the strategies for doing it with TDD. He describes a disciplined stepwise process that avoids burning bridges.

This then leads into Michael Feathers' concepts of working on legacy code (which Feathers defines as "code without tests"). He lists Feathers' legacy code change algorithm:

1. Identify change points.
2. Find test points.
3. Break dependencies.
4. Write tests.
5. Make changes and refactor.

He describes how to apply this to embedded systems. Two important types of unit tests during this process are characterization tests that establish how the legacy code behaves, and learning tests that help you learn how to work with third-party code.

The final chapter covers test patterns and antipatterns. This is useful for helping to build good, effective unit tests that are maintainable over the long term.

The Big Lesson

For embedded systems, working with the specific hardware is a critical detail. But as Martin points out in Clean Architecture, it's just a detail. For GUI-based mobile, web, and desktop apps, the GUI is just a detail. For either of these, as well as backend servers, the OS (or lack thereof on a bare metal system) is just a detail. The network is just a detail. The database or the filesystem is just a detail. The frameworks or third-party packages are just details.

All of those details, critical though they may be, can be isolated and segregated from the code that defines what it is your system is about. That code is called the business logic, which sounds a little too dry for me. But's it's the stuff that makes your system something that other people want to use. So it's the stuff that makes your system drive a meaningful business.

Your business logic interacts with all those details to make a functioning system. TDD allows you to test that logic, in all its happy, twisty, and unhappy paths, separated from its dependencies on the details. The details are represented by test doubles: dummies, stubs, spies, mocks, and fakes.

This is where the Gang of Four's concept of programming to an interface, not an implementation, stated in their book Design Patterns, comes into play. You write your business logic to work to an interface to accomplish the detail interactions. In the production environment, you use the real detail components, the real implementations, with a thin adaptation layer that conforms to the interfaces.

In the test environment you can substitute test doubles that conform to the interfaces; these are alternate implementations. Since you're in control of the test doubles, you can drive any scenario you need to in order to exercise the business logic.

That isolation also allows you to substitute in other versions of production details, so it's a design strategy, not just a testability strategy. Maybe you want to use some different hardware in your embedded system, or run your app on a different mobile device with a different GUI, or deploy the system on a different OS, or use a different database.

By defining your details as abstract data types or abstract services, you can drop in replacements, with just the effort of implementing the interface layers.