Review: Test Driven Development for Embedded C, James W. Grenning

The TL;DR:

• Test Driven Development for Embedded C by James W. Grenning is an outstanding book. 
• The title says C, but if you work in C, C++, C#, Go, Objective-C, Java, Javascript, or anything else, this is worth reading.
• It says embedded, but if you work in embedded systems, front end web apps, mobile apps, desktop apps, backend servers, or anything else, this is worth reading.
• And it's not just TDD, it's all the concepts that go into good design.
• Get it, read it, USE it. You won't regret it.

Background

I first learned about XP (eXtreme Programming) concepts in 2007, when I was introduced to Kent Beck's Test-Driven Development: By Example. I used TDD (Test-Driven Development) to develop a major component on a server system. I learned more in 2013, when I read Michael Feathers' Working Effectively With Legacy Code. I used that to apply TDD to an existing server codebase.

Over the past 3 months, I've been on a reading binge, triggered by reading Robert C. Martin's 2017 book Clean Architecture: A Craftsman's Guide to Software Structure and Design. I have an hour-long commuter rail ride, so I have lots of time to read and work on my laptop, plus a little lunchtime reading, and I always have a book open at home.

I read his Clean Code: A Handbook of Agile Software Craftsmanship, The Clean Coder: A Code of Conduct for Professional Programmers, and am currently in the middle of his Agile Software Development, Principles, Patterns, and Practices.

I read Sandro Mancuso's The Software Craftsman: Professionalism, Pragmatism, Pride, and am in the middle of Mike Cohn's Agile Estimating and Planning, both from Martin's series.

I read Andrew Hunt and David Thomas' The Pragmatic Programmer: From Journeyman to Master, and am halfway through Pete McBreen's Software Craftsmanship: The New Imperative; Martin Fowler's Refactoring: Improving the Design of Existing Code is waiting on the shelf.

I've encountered bits and pieces of this material over the years, but this was a chance to go back to primary sources, get the full details and parts I've missed out on, and really understand them. I highly recommend it.

Review

But maybe you don't have time for all that. Maybe you'd like to cut to the chase and see how to apply their principles in practice.

Test Driven Development for Embedded C by James W. Grenning does that. It draws from many of those sources and more, showing you real-world examples to put them into practice.

Grenning is one of the original authors of the Agile Manifesto (as are Beck, Fowler, Hunt, Martin, and Thomas). He contributed the chapter "Clean Embedded Architecture" to Clean Architecture, and is the inventor of the Agile planning poker estimation method.

The book was published in 2011, so is now 7 years old, but it remains as timely as ever. That's especially true as IoT vastly expands the number of embedded systems that we rely on in our daily lives. Effective testing is critically important. For instance, see Testing Is How You Avoid Looking Stupid.

If you work on embedded systems in C, this is a must read.

If you work in a different language besides C, or on a different type of system than embedded systems, you may not think that a book on embedded C programming applies to you. But it's broadly applicable and worth reading.

The book is organized as an introductory chapter, the remaining chapters grouped into 3 parts, and appendices. I see it as three distinct portions, plus appendices: Chapter 1; Parts I and II (chapters 2-10); and Part III (chapters 11-15).

Throughout, Grenning addresses the common concerns people have with applying TDD to embedded systems. Embedded systems are a particular challenge, with particular target system constraints, so people might be skeptical.

This is a very hands-on, how-to book. I've included a number of lists from it, including those Grenning draws from other sources, because they illustrate the practical, pragmatic, disciplined approach. You can use this as a cheat sheet to remember them after you've read the book.

It might be tempting to think you can get by just with the information I've provided here and skip the book. But I've included it specifically with the hope that you'll realize you must read the book, and that it will be a worthwhile investment.

First Portion

This is the motivational portion, the appetizer. Grenning introduces TDD, its benefits in general, and the specific benefits for embedded systems.

He lists Kent Beck's TDD microcycle:

1. Add a small test.
2. Run all the tests and see the new one fail, maybe not even compile.
3. Make the small changes needed to pass the test.
4. Run all the tests and see the new one pass.
5. Refactor to remove duplication and improve expressiveness.

The microcycle is critically important to the technique, so Grenning reminds you of it several times as he works through examples. This is what makes TDD effective, and I know from my own experience is also what makes it fun and extremely satisfying. He has a sidebar titled "Red-Green-Refactor and Pavlov's Programmer", which is very apt. That Pavlovian drive to take the next step in the cycle draws you into the zone and keeps you cranking.

For embedded systems, in addition to all the benefits that apply to other types of software, the primary benefits include being able to develop tested, working code when the target hardware isn't available; being able to test off-target (i.e. not on the target embedded system), where you have all the benefits of a general-purpose system and none of the constraints of an embedded one, including speed of development turnaround cycle; being able to isolate hardware/software interactions; and decoupling software from hardware dependencies.

That last point is part of the Big Lesson (see below) from all this. TDD in general, for any type of software, results in testable and tested software. But more than that, it drives development in a way that improves the design significantly.

That improved design means a much longer and happier life for the software and the systems that use it. They will be able to adapt to changes much more easily. It's not just about getting V1.0 done. It's about getting to V10.0.

In Software Craftsmanship, Pete McBreen starts off with the origin of the term software engineering. It was coined by a 1967 NATO study group working on "the problems of software." A 1968 NATO conference identified a software crisis and suggested that software engineering was the best way out of that crisis. They were concerned with very large defense systems. McBreen gives the example of the SAFEGUARD Ballistic Missile Defense System, developed from 1969 through 1975.

He says, "These really large projects are really systems engineering projects. They are combined hardware and software projects in which the hardware is being developed in conjunction with the software. A defining characteristic of this type of project is that initially the software developers have to wait for the hardware, and then by the end of the project the hardware people are waiting for the software. Software engineering grew up out of this paradox."

McBreen is questioning the value of that style of large-scale software engineering in the development of commercial products, suggesting that a different approach is needed.

But doesn't that situation sound familiar? Doesn't that sound like the problem embedded systems developers face all the time, that Grenning is addressing? This was a situation where TDD and off-target testing could have significantly alleviated the software crisis.

Granted, it was more complicated, since they were also developing the very processors and programming languages they would use, while modern systems rely on COTS (Commercial Off The Shelf) processors and languages. But we see that this has been a pervasive problem for some 50 years.

All types of systems, from embedded to frontend mobile apps to high-scale backend servers, in all those languages, from C to C++, Objective-C, Go, Java, Javascript, etc., can benefit.

All that code can be removed from its normal production environment and run off-target, off-platform, in a unit test environment that allows you to exercise every code path you want easily and quickly. That includes the obscure dark corners of the code trying to handle unusual error cases that are hard to produce on the target system.

For some of my own experience testing off-target, see Off-Target Testing And TDD For Embedded Systems.

Second Portion

This portion is the meat of the book, applying TDD to real-world embedded development and going through the mechanics with practical examples.

Following the lead of Martin's book, Grenning makes restrained use of UML diagrams. While some people dislike UML because they associate it with the heavyweight BDUF (Big Design Up Front) software engineering methodologies that McBreen was talking about, this is a very effective use of it that communicates information quickly. Which is the whole point of UML.

Grenning presents two unit test harnesses, Unity and CppUTest (of which he is one of the authors). All of the material applies just as well to other test harness tools, such as Google Test/Google Mock. It's equally applicable to other languages and their language-specific test harnesses.

He uses Gerard Meszaros' Four-Phase Test pattern to structure tests:

• Setup: Establish the preconditions to the test.
• Exercise: Do something to the system.
• Verify: Check the expected outcome.
• Cleanup: Return the system under test to its initial state after the test.

The rubber meets the road in his five examples of using TDD to develop embedded code:

• LED driver
• Light scheduler for a home automation system
• Circular buffer
• Flash driver for ST Microelectronics 16 Mb flash memory device
• OS isolation layer (aka OSAL, OS Abstration Layer) for Linux/POSIX, Micrium uC/OS-III, and Win32 (this is actually an appendix and only covers thread control, but establishes the pattern)

Clearly, these have real hardware dependencies on both the processor I/O interface and the attached devices, as well as the system clock, and real OS dependencies. Those are critical concerns for the embedded developer. The LED driver is very simple behavior, so makes for a gentle introduction. The others are more complex.

Grenning discusses driver requirements, then shows the initial tests and code. Notice I said tests first. That's an important concept in TDD. You always write the test first, that uses the code in the way that you want the code to work. Then you write the code that satisfies that usage. He emphasizes the save-make-run cycle that you do repeatedly during this process. Then you repeat for the next test and bit of code. That's how you make fast progress.

The key concept is faking out portions of the system, so that the Code Under Test (CUT) can run as if it was running on the real system. That's critical for making TDD work off-target and off-platform. There are several strategies for doing this. In the case of the LED driver, he uses virtual registers to simulate memory-mapped I/O. This is simply a variable under the control of the test suite.

He also talks about test-driving the interface before test-driving the internals. That's another critical concept, integral to the whole design process. That's design-for-change. Because things will change. A product with a long, useful life, that represents an ongoing revenue stream for a company, will change over that time to adapt to changes in underlying technologies, user requirements, and usage. TDD means you can make changes without fear of breaking things (because you'll find and fix breakage as a result of performing the microcycle).

He talks about the strategy of incremental progress and refactoring as you go. This is in the heat of development. Final code does not flow directly from your fingertips. It evolves in incremental steps as you work. Did you ever look at someone's code and marvel at how clean and easy to follow it was, despite the complexity of the job it was achieving? You might think you could never do something that easily. This process results in that kind of code. Like a novelist in the heat of writing a scene, the first draft is never the final product, and the story arc evolves over time.

This is where he covers several important guidelines for driving the TDD process effectively. He lists Robert Martin's Three Laws of TDD:

• Do not write production code unless it is to make a failing unit test pass.
• Do not write more of a unit test than is sufficient to fail, and build failures are failures.
• Do now write more production code than is sufficient to pass the one failing unit test.

He describes Kent Beck's snappy acronym DTSTTCPW: Do The Simplest Thing That Could Possibly Work, which initially means just faking it (for instance, hard code a function to return false in order to get the test that uses it to pass). Then keep tests small and focused, and refactor on green (many unit test setups show a failing result in red, and a passing result in green).

As this evolves, the faked out code turns into real code (the hard coded false is changed to actual code that does something and returns true or false under the appropriate conditions). That builds out a verified test suite as it builds out verified code.

This leads to the TDD State Machine, which tells you what to do next. The guidelines above and the state machine take you through the mechanics of working in the TDD style. They answer the questions:

• How should you start?
• What should you do next?
• How do you know when you're done?

Whenever you write some production code, ask yourself, "Do you have a test for that?". If not, stop, go back, and write the missing test.

He also covers Dave Thomas and Andrew Hunt's DRY principle: Don't Repeat Yourself. This mantra helps drive the refactoring so that you keep the code lean and clean. I'll throw in additionally the DAMP principle: use Descriptive And Meaningful Phrases, a concept the book applies without calling out by name. This favors readable function and variable names that express intent over cryptic abbreviations and syntax. The result is code that reads with a narrative flow.

Keeping your code DRY and DAMP makes it easy for others to understand and modify (which might be you when you come back to it six months or a year later). This is the same as Beck's microcycle step 5.

To some degree this all turns TDD into a very mechanistic process. But that's a good thing. It's not a random, ad hoc process where you're constantly questioning yourself about what do to. Instead it's an orderly stepwise process that makes effective progress. You quickly see and appreciate the value.

It's also very fun and satisfying, because that mechanistic aspect actually drives your creativity. What's the next thing you can add to it? What's the next test, the next bit of functionality? When you finish, you feel like you've accomplished something, and you have the evidence to prove it. It's addicting.

That leads to Grenning's Embedded TDD Cycle, which starts with TDD on the development system, then advances to the target processor and eval hardware, then the actual target hardware:

• Stage 1: Write a unit test; make it pass; refactor. This is red-green-refactor, the TDD microcycle on the development platform.
• Stage 2: Compile unit tests for target processor. This is a build check that verifies toolchain compatibility.
• Stage 3: Run unit tests on the eval hardware or simulator.
• Stage 4: Run unit tests on target hardware.
• Stage 5: Run acceptance tests on target hardware. These are automated and manual tests of the integrated system.

This sequence gives you confidence in the code under test quickly, then you can address any hardware-dependent issues that start to arise, such as compiler, library, or primitive data type differences. Next you to start exercising the hardware-dependent code.

Testing separately on eval hardware and actual target hardware helps shake out hardware issues in the actual target, since the eval hardware is presumably known good. One of the challenges in embedded development is always trying to determine if problems are due to the software or due to the hardware, since both are in active development and haven't had much soak time to prove them out.

For the other TDD examples, Grenning goes through a progression of different collaborator strategies. These are the test doubles, the fakes, that are substitutable for real components. They stand in for those components to break the test dependencies and allow you to simulate and monitor interactions. An important point is that they are much lighter weight than full-scale simulators. Full simulators can themselves require significant development. These fakes have only enough behavior to support the tests (part of the DTSTTCPW mindset).

He uses these types of doubles:

• Spies
• Stubs
• Mocks
• Exploding fakes

He goes through the following substitution methods, showing how to do them and discussing when they are appropriate:

• Link-time substitution
• Function pointer substitution
• Preprocessor substitution
• Combined link-time and function pointer substitution

These are fully-worked-out examples, although he starts omitting intermediate steps as he progresses in the interest of brevity. All the code is available online.

Third Portion

This portion completes the meal, complementing the meat in the second portion. It addresses design issues. This is important because design for testability also means design for flexibility and long product life.

Grenning starts out with Martin's SOLID principles:

• S: Single Responsibility Principle (SRP)
• O: Open Closed Principle (OCP)
• L: Liskov Substitution Principle (LSP)
• I: Interface Segregation Principle (ISP)
• D: Dependency Inversion Principle (DIP)

He covers both how the previous chapters have incorporated these principles, and how to use them to guide the development process. TDD is closely intertwined with them.

Don't be put off by the apparent difference between non-object oriented and object-oriented languages. The specific language used is irrelevant. The syntactic mechanics may be different, but the concerns and concepts are all the same. C can be every bit as object-oriented as Java, it just takes a little more developer discipline. That means that all of the concepts of the various principles above apply.

He uses the SOLID principles in four module design models of increasing complexity, applicable in different embedded system design cases:

• Single-instance module: Encapsulates a module's internal state when only one instance of the module is needed.
• Multiple-instance module: Encapsulates a module's internal state and lets you create multiple instance of the module's data.
• Dynamic interface: Allows a module's interface functions to be assigned at runtime.
• Per-type dynamic interface: Allows multiple types of modules with the same interface to have unique interface functions.

You'll probably recognize more than one of these in the systems you work on. You may also recognize object-oriented concepts, and in fact he shows how to implement, use, and test a C++ virtual function table (vtable) in C.

Part of good design is adapting to change. He covers Martin Fowler's concepts of refactoring, both the code smells that point to things that need to be refactored, and the strategies for doing it with TDD. He describes a disciplined stepwise process that avoids burning bridges.

This then leads into Michael Feathers' concepts of working on legacy code (which Feathers defines as "code without tests"). He lists Feathers' legacy code change algorithm:

1. Identify change points.
2. Find test points.
3. Break dependencies.
4. Write tests.
5. Make changes and refactor.

He describes how to apply this to embedded systems. Two important types of unit tests during this process are characterization tests that establish how the legacy code behaves, and learning tests that help you learn how to work with third-party code.

The final chapter covers test patterns and antipatterns. This is useful for helping to build good, effective unit tests that are maintainable over the long term.

The Big Lesson

For embedded systems, working with the specific hardware is a critical detail. But as Martin points out in Clean Architecture, it's just a detail. For GUI-based mobile, web, and desktop apps, the GUI is just a detail. For either of these, as well as backend servers, the OS (or lack thereof on a bare metal system) is just a detail. The network is just a detail. The database or the filesystem is just a detail. The frameworks or third-party packages are just details.

All of those details, critical though they may be, can be isolated and segregated from the code that defines what it is your system is about. That code is called the business logic, which sounds a little too dry for me. But's it's the stuff that makes your system something that other people want to use. So it's the stuff that makes your system drive a meaningful business.

Your business logic interacts with all those details to make a functioning system. TDD allows you to test that logic, in all its happy, twisty, and unhappy paths, separated from its dependencies on the details. The details are represented by test doubles: dummies, stubs, spies, mocks, and fakes.

This is where the Gang of Four's concept of programming to an interface, not an implementation, stated in their book Design Patterns, comes into play. You write your business logic to work to an interface to accomplish the detail interactions. In the production environment, you use the real detail components, the real implementations, with a thin adaptation layer that conforms to the interfaces.

In the test environment you can substitute test doubles that conform to the interfaces; these are alternate implementations. Since you're in control of the test doubles, you can drive any scenario you need to in order to exercise the business logic.

That isolation also allows you to substitute in other versions of production details, so it's a design strategy, not just a testability strategy. Maybe you want to use some different hardware in your embedded system, or run your app on a different mobile device with a different GUI, or deploy the system on a different OS, or use a different database.

By defining your details as abstract data types or abstract services, you can drop in replacements, with just the effort of implementing the interface layers.

Off-Target Testing And TDD For Embedded Systems

I've recently started reading things by James Grenning (Wingman-sw.com), one of the authors of the Agile Manifesto. My interest in his work relates to Test-Driven Development (TDD) for embedded systems.

A copy of his book Test Driven Development for Embedded C is currently winging its way to me. His site links to a webinar he gave last summer, Test-Driven Development for Embedded Software, that makes a great introduction to the topic.

I found one of his answers on Quora interesting. The question was: Can I perform a unit test when creating C firmware for ARM Cortex-M MCUs? The answer, of course, is yes. Specifically, testing can be done off-target (i.e. not on the target embedded system).

I wrote a long comment on the answer, and decided it might make an interesting blog post. So the remainder of this post reproduces it substantially as it appears there, with some cleanup. He very kindly asked if I would be interested in adding it to his Stories From The Field.

My Three Stories

I can offer three anecdotes that show why I give a big thumbs up to off-target testing. Off-target testing puts you on target!

The first case was back in 1995. I had recently transferred to the DEChub group at Digital Equipment Corporation to work on networking equipment.

They had a problem with their popular DECbridge 90 product, an office- or departmental-scale stackable Ethernet bridge running an in-house custom RTOS on Motorola 68K, all written in C. It would run for weeks or months at a customer site, then suddenly crash. That would throw the LAN into a tizzy as it went through a Spanning Tree Protocol (STP) reconfiguration event. Then the LAN would do it again once the bridge came back up and advertised its links.

So it could be very disruptive to the local network and everyone running on it, completely unpredictable. No one had been able to reproduce the problem in the lab.

I was tasked with finding and fixing it. This platform had very little in the way of crash dump and debug support, and software update was done by burning and replacing an EPROM. It did have an emulator pod, so that was how most debugging was done.

The problem here was the long run time between failures. That made trying to collect useful information from repeated test runs, either real or via emulator, impractical to the point of impossibility.

The one clue we knew from the crash log was that it was an OOM condition (Out Of Memory). The question was why. Other than supporting STP, which is a bit of complex behavior, a bridge is a pretty simple device, just L2 forwarding. Packet comes in, look it up in the bridge tables, forward it out the appropriate interfaces.

The key dynamic structure was the MAC address table. A bridge is a learning device in that it learns which MAC addresses are attached to which links. It builds up the table as it runs, learning the local network topology and participating in STP. So this table was certainly a prime suspect, but it had capacity for thousands of entries, yet it was crashing in LANs with only tens or hundreds of nodes.

The table used a B-tree implementation that was public-domain software from some university. We speculated that it was a memory leak in either the B-tree itself, or our interfacing to it.

So I pulled out the B-tree code and built a test program for it that would go through tens of thousands of adds and deletes in various simple patterns. This is similar to the type of test fixture that Brian Kernighan and Rob Pike later talked about in their book The Practice Of Programming.

I ran this off-target, on a VAX/VMS. VMS supported a simple Ctrl-T key in the terminal interface that would show the process memory consumption, similar to what the Linux ps command shows. The turnaround time on playing with this setup was minutes, build and run, with the full support of an OS to help me chase things down, including good old printf logging and customized dumping of data structures (VMS also had a debugger similar to gdb).

With this I could see that under some patterns, memory consumption was monotonically increasing. So yeah, a memory leak. Further exploration allowed me to home in on the right area of the code.

It was right there in the B-tree memory release code: it would free the main B-tree nodes (the main large data element being managed), but not the associated pointer nodes that it used for bookkeeping. So on every B-Tree node release, it would leak 8 bytes of memory.

This was a case of a very slow memory leak, that only manifested with lots of table changes. In a customer environment, it could take a long time to chew through the memory. In the lab running on-target, it was even slower, since we didn't know what the cause was, so we didn't know how to trigger and reproduce it.

Off-target, it took less than an hour to find. Code change: add two lines to free the pointer nodes. This was after many man-weeks of effort for all the people involved in trying to reproduce and chase down the problem, plus all the aggravation caused at customer sites. Ten minutes to code, build, and verify the fix.

The second case was just recently. I implemented an FSM based directly on the book Models to Code: With No Mysterious Gaps, by Leon Starr, Andrew Mangogna, and Stephen J. Mellor (backed up by Executable UML: A Foundation for Model Driven Architecture, by Stephen J. Mellor and Marc J. Balcer, and Executable UML: How to Build Class Models by Leon Starr; I highly recommend the trio of books). Thank you, Leon, Andrew, Stephen, and Marc!

I was also reading Robert C. Martin's Clean Architecture: A Craftsman's Guide to Software Structure and Design and Clean Code: A Handbook of Agile Software Craftsmanship at the time, which heavily influenced the work (and finally motivated me to fully commit to TDD; Grenning contributed the chapter "Clean Embedded Architecture" to Clean Architecture). Mellor and Martin are both additional Agile Manifesto authors.

A product of all this reading, the FSM was a hand-built and -translated version of the MDA (Model -Driven Architecture) approach, in C on a PIC32 running bare metal superloop.

The FSM performed polymorphic control of cellular communication modules connected via a UART. The modules use the old Hayes modem "AT" command set to connect to the cell network and perform TCP/IP communications.

It was polymorphic because it had to support 4 different modules from 2 different vendors, each with their own variation of AT commands and patterns of asynchronous notifications (URC's, Unsolicited Result Codes).

If you think LANs and WANs are squirrelly, just wait till you try cellular networks. I could hardly get two test runs to repeat the same path through the FSM. Worse, there were corner cases that the network would only trigger occasionally.

It was horribly non-deterministic. How can I be sure I've built the right thing when I can't stimulate the system to produce the behavior I want to exercise?

The solution: build a nearly-full-coverage test suite to run off-target. I built a trivial simulator with fake system clock and UART ISR that I ran on an Ubuntu VM on my Mac. That gave me full support for logging and gdb.

This wasn't quite TDD, but it was one step away: it was Test-After Development, and instead of Google Test/Mock or some other framework, I built my own ad-hoc fakes and EXPECT handling.

With this I was able to create scenarios to test every path in the FSM, for all the module variants. Since I had control of the fake clock and ISR, I could drive all kinds of timing conditions and module responses. It did help that the superloop environment was pure RTC (Run To Completion, which coincidentally is required for Executable UML state machines), rather than preemptive multitasking/multithreading. But I could have faked that as well if necessary.

I was able to fix several bugs that would have been hell to reproduce and debug on-target. In all cases, just as with the B-tree, the code changes were trivial. The time-consuming and hard part is always the debug phase to figure out what's going wrong and what needs to be changed. Doing the actual changes is usually simple.

That debug phase is where non-TDD approaches run into trouble, especially when they have to be done on-target. It can consume unbounded amounts of development time. The time required to do TDD is far shorter, and for a significant number of problems can either completely eliminate the debug phase, or narrowly direct it to the specific code path of a failing test.

The third case was this past week, when I did my first true TDD off-target thing for some embedded code. The platform is embedded Linux on ARM, so full OS, with cross-compiled C++.

I built the code in my Ubuntu VM and used Google Test/Mock, mocking out the file I/O (standard input stream and file output) and system clock. The code wasn’t particularly complex, but it did have a corner case dealing with a full buffer that represented the greatest bug risk.

I used very thin InputStreamInterface, OutputFileInterface, and ClockInterface classes as the OSAL (Operating System Abstraction Layer) to provide testability (thank you, Robert and James!).

It was gloriously wonderful and liberating to build it TDD style, red-green-refactor, and I knew I had all the paths in the code covered, including the unusual ones. That instills great confidence in what I did. No more worrying that I got everything right. I was able to demonstrate that I did.

Did it take a little extra time? Sure, mostly because I’m still on the learning curve getting into the TDD flow. But if I hadn’t used TDD and this code had produced failures, it would take me longer after the fact to chase down the bug. Plus I was able to avoid the impact on all the other people in the organization affected by the development turnaround cycle.

And just today, I added more behavior to that component using the TDD method. I was able to work fully confident that I wasn't breaking any of the stuff I had already done, and just as confident in the new code.

So I'm definitely a believer in off-target testing, and from now on I'll be doing it TDD.

Another benefit of this off-target TDD model? Working out of that Ubuntu VM on my Mac, I'm totally portable. I can work anywhere, at a coffee shop, on the train commuting, at the airport, on a plane, at home. I can be just as productive as if I had my full embedded development environment in front of me. Then once I'm back at my full environment, I have tested, running code ready to go.

For reference, these are the books that taught me TDD while in different jobs, both highly recommended:

• Test Driven Development: By Example, by Kent Beck (yet another Agile Manifesto author). I was introduced to the book and TDD in general by new coworker Steve Vinoski in 2007, whose cred in my eyes went way up when I noticed his name in the acknowledgements of James O. Coplien's Advanced C++ Programming Styles and Idioms.
• Working Effectively With Legacy Code, by Michael C. Feathers. Amazon tells me I bought this in 2013. At the time I used it to start adding unit test coverage to our codebase at work. What makes this book particularly useful is the fact that nearly all software development requires working with legacy code to some degree, even on brand new projects. It also helps you avoid creating a legacy of code that future developers will curse. 

Review: Make: Electronics and Make:More Electronics

Charles Platt's Make: Electronics and Make: More Electronics.

Amazon links:

• Make: Electronics: Learning Through Discovery 2nd Edition
• Make: More Electronics: Journey Deep Into the World of Logic Chips, Amplifiers, Sensors, and Randomicity

If you're interested in learning electronics, I highly recommend these two books by Charles Platt. They are hands down the best books I have ever seen on the subject, spectacular resources for the beginner.

Rather than focusing on theory, Platt jumps right into hands-on experimentation. The books are organized as a series of experiments and circuit-building projects that build knowledge incrementally.

He calls it "learning by discovery". He then follows up with just enough theory to explain what's going on. This is an extremely effective method that avoids getting bogged down.

I first learned electric theory in high school science. I learned Ohm's Law and basic circuit layout, including the equations for computing series and parallel resistance. I learned further details in college physics.

But these didn't really cover the practical details of electronics. They didn't address detailed circuit design, combining components into useful projects.

I started to learn some of those details from the books of Forrest M. Mims III and George Young's book Digital Electronics: A Hands-on Learning Approach. The latter introduced me to integrated circuit chips (IC's) and digital logic, as well as breadboard experimentation.

That constituted the bulk of my electronics knowledge for the past 35 years. But there was still a lot missing, particularly an intuitive understanding of electronics and all those other random parts surrounding the IC's.

Then I found Platt's books. Platt has a real gift for explaining things at an intuitive level in just a few concise paragraphs and clear diagrams.

He delves deep into the practical details. No detail is too small. For a beginner trying to learn from a book, this is critical.

He explains the most basic things so that you know how to wire up a breadboard and check things with a meter. He shows how things work internally, both mechanically and electrically, so a component isn't just an opaque black box.

The color diagrams are outstanding. One thing I really like is the way he steps from a circuit schematic diagram, to a breadboard-friendly schematic, to a breadboard component and wiring diagram, to a component value diagram, to an under-the-covers diagram illustrating all the electrical paths in the wiring and the breadboard connections hidden beneath.

He takes several projects from breadboard to final soldered board built into a simple enclosure. This shows how you can turn your experiment into a completed useful or fun gadget.

The diagrams and project builds really show where other books fall short. Most books show a schematic, and maybe a completed breadboard or a completed wired-up project. But they don't show the stepwise process to get from the start to the end.

That process is not always obvious and is full of opportunities for mistakes, so having it laid out in detail is a huge benefit. He also covers some of the things that can go wrong and how to diagnose and fix them.

The clarity of the diagrams and overall layout make the books very readable. This is another improvement over other books.

If you send a registration email to Platt, he'll add you to his email list for a bonus project and book updates.

Microcontrollers

Platt doesn't emphasize microcontrollers in these books. In an age where Arduinos, Raspberry Pi's, and other microcontrollers allow you to solve nearly any problem with a little embedded software, he mostly shows you what you can do without them. One experiment does cover using an Arduino.

He also discusses the pros and cons of replacing the discrete components with microcontrollers in several experiments. This is actually very useful from an engineering standpoint, giving you choices in how to implement things.

That also ensures that if you do incorporate microcontrollers into your projects, you understand how to integrate them with external components. There are many books on getting started with microcontrollers, but they tend to gloss over the details of those other components, assuming you already understand them. Which you will if you read these books!

Component Kits

Component kits for all the experiments in the first book are available online. You can certainly gather parts on your own, but the kits offer you one-stop shopping of the correct parts.

I used ProTechTrader, the supplier he recommends in his email, and I recommend them highly based on my experience. Make sure you get the kits for the 2nd edition.

The kits are available for the best price directly from the ProTechTrader website. They offer 3 kits, covering experiments 1-11, 12-24, and 25-34. Each is available in regular and deluxe versions. The deluxe versions add things like a digital multimeter, soldering iron, 9V power supply, and upgraded magnet.

I purchased the regular version of each kit, since I already had most of the deluxe items, with free economy 3-10-day shipping. The kits arrived in 3 days.

While you pay a little extra per part for convenience vs. buying everything separately, it was well worth it. The parts are extremely well organized. They're bagged and labeled by value, stored in compartmentalized containers, and identified by experiment.

Don't underestimate the value of the labor that went into that. Platt dedicates several pages in his book to organization of workspace and parts. That's key to efficient work. Rummaging around in a box of loose parts will make you tear your hair out.

Additional Books


Platt's Make: Tools and 3-volume Encyclopedia of Electronic Components.

Platt has several additional books that make useful companions to this pair:

• Make: Tools: How They Work and How to Use Them
• Encyclopedia of Electronic Components Volume 1: Resistors, Capacitors, Inductors, Switches, Encoders, Relays, Transistors
• Encyclopedia of Electronic Components Volume 2: LEDs, LCDs, Audio, Thyristors, Digital Logic, and Amplification
• Encyclopedia of Electronic Components Volume 3: Sensors for Location, Presence, Proximity, Orientation, Oscillation, Force, Load, Human Input, Liquid Light, Heat, Sound, and Electricity

The first book (no, it's not about how to make tools, it's just part of the Make: series) covers the basic hand and small power tools you'll find at home centers and hardware stores, showing how to use them to build small projects. It feature Platt's usual deep attention to practical details.

The book contains a number of simple projects in wood and plastic. The methods for working with plastic are particularly noteworthy, because while there are many books about woodworking, there aren't many about plastic.

These are the skills you need to build different styles of enclosures and stands for your electronics projects, and can also be applied to other mechanical aspects such as robotics.

The remaining books are a 3-volume encyclopedia of electronic components. This is all the information that he didn't have room for in the other books, plus more. Where those books were written as tutorials, this is a reference set.

He's compiled a vast trove of information culled from manufacturer data sheets, tutorials, reference books, and other sources to create a centralized, practical one-stop resource.

Need to know pinouts, sample circuits, voltage levels, alternative packages? You can find them here, in Platt's signature level of detail.

How To Ace Calculus

This is the method I used to ace 3 semesters of calculus. Also linear algebra, differential equations, and a semester of physics. It should work for any math or science class, at earlier or later level.

When I say ace, I mean getting a grade of 100 on most homework, quizzes, tests, midterms, and finals. In all cases, the top grade in the class. Yes, I was the one breaking the curve.

Sounds arrogant? Well, I didn't start off in that lofty position. So before I give you the recipe for success, let me give you the recipe for failure.

The Recipe For Failure

In 1978, I entered Northwestern University, in Evanston, IL, as a mechanical engineering major. My dream was to work for NASA.

This was a major in the Technical Institute, requiring calculus, physics, and mechanics (statics and dynamics).

The recipe:

1. Show up for all classes and pay attention.
2. Complete all reading assignments on time.
3. Complete all assigned homework on time.
4. Study for tests, reviewing homework.

This was the recipe that had gotten me through high school, where I was usually able to do most of the homework in the last 5 minutes of class allocated for that purpose. Then just a few minutes in "study hall" period or at home to complete it, and another 10 or 15 to read the next section.

Sounds like a pretty good plan, right? Sounds like a good student, right?

The problem was that the material in college was more difficult and faster paced. Doing just the assigned problems was barely enough to keep your head above water, sometimes not even enough for that. It didn't give you enough practice thinking through and performing the work.

Back in algebra, problems were simple, they had one procedure to follow to the solution. Calculus wasn't like that. There were multiple procedures depending on the style of the equation. A good portion of the battle was classifying the equation to determine what approach to bring to it.

Physics was similar. Both topics required a more analytical approach. That meant building up a problem database in your mind so you could pick the approach. That meant experience doing lots of problems.

The result of following that recipe? C's, D's, and finally, an F in physics. Where I had prided myself on my math and science abilities, my favorite subjects, I had failed. Distraught, I dropped out of Northwestern.

The Recipe For Success

About 5 years later, I started part-time classes at Richland College, part of the Dallas County Community College District.

Oh sure, you may say, community college. That's easy, it's not a real college.

Negative. Richland used exactly the same textbooks as Northwestern, just the next editions. So it was exactly the same material. And I had Ralph Esparza as instructor for calculus I and III. Ralph was feared among students as a tough math teacher, who cares if it's community college or Ivy League.

I was determined to repeat all those classes in my favorite subjects, and do well in them. Somewhere in hindsight, I had realized the need to do more than the minimum.

The recipe:

1. Show up for all classes and pay attention.
2. Complete all reading assignments on time.
3. Complete all assigned homework on time.
4. Complete all remaining odd-numbered problems in the section and check against answers in the back.
5. Complete all remaining even-numbered problems in the section.
6. Study for tests (regular tests, midterms, and finals), redoing all problems that the tests cover.

This is a great way to work in study groups, too. When someone in the group has difficulty, everyone can contribute to helping them understand it. Or maybe the one person in the group who understands it is able to help everyone else.

This boils down to doing every problem in every section of the textbook at least twice, more like three or four times.

You may say, that's a lot of work. Yes, it is.

In Nike ads, athletes show how tough they are. Just do it. Be tough.

The result? Redemption.

Sodoto: See One, Do One, Teach One

Here's a useful strategy on this learning path: see one, do one, teach one. Sodoto.

Sodoto is a learning method and a teaching method rolled into one. It's the cycle of knowledge.

I'm familiar with it from medicine. My wife is a surgical nurse, and this is the traditional method of teaching in surgery. Obviously, safety concerns mean that you don't just watch a brain surgeon at work and then go try it yourself.

But this forms a useful pattern of mentoring and learning and passing knowledge along. It applies to any kind of knowledge- or skill-based activity.

It works with a single student at a time, or a whole group. You don't have to be a formal teacher.

See one: watch someone do a procedure.

Do one: do what you saw.

Teach one: show it to someone else.

Once you learn a procedure, you're primed to teach it. That's how knowledge spreads.

Here's the real kicker: the teaching step is actually a powerful learning step for you as the teacher. It locks the knowledge into your brain.

You have to have sorted out what you're talking about in order to teach it. You can't just vaguely know it and wave your hands in the air glossing over details. Your students will be annoyed and you'll feel stupid.

The process of getting ready to teach and then doing the teaching forces you to organize your thoughts and chase down details, because you don't want to look stupid, and you want to be prepared for questions.

That motivates you to dig deeper. As a result, you end up learning more yourself.

There are two keys to making this work: background knowledge, and the experience of doing it.

Background Knowledge

Background knowledge applies at each stage of see, do, and teach. Note that "see" can mean live and in person, or on video.

Whatever the subject, medicine, coding, climbing, sailing, scuba diving, physical fitness, martial arts, building anything from woodworking to electronics, any knowledge you have before seeing the procedure will help you understand it.

You can bet that surgeon learning how to do brain surgery brought a huge amount of background knowledge.

Some things take minimal background, just the random skills and knowledge you already have from life. But more difficult subjects benefit from whatever time you can invest beforehand. Videos, books, blogs, and articles, in print and online, are all good resources, as well as online forums.

That establishes the background knowledge you'll bring to seeing the procedure.

Once you've seen the procedure, as you prepare to do it yourself, it's useful to go back to your resources. Now that you know better what to look for, you can get more details. You can reinforce what you saw.

That expands the background knowledge you'll bring to doing the procedure.

Once you've done the procedure, as you prepare to teach it, go back to your resources again. As a result of doing, there will be details you want to fill in, and you may understand the material better. You may have run into some things that you wished you knew more about. You may anticipate additional questions from your students.

That further expands the background knowledge you'll bring to teaching the procedure.

Experience Of Doing

The experience of doing the procedure is critical. That's where you have the opportunity to work through mistakes and see what works and doesn't work for you. That's where you start to lock it into your brain.

Don't be afraid to make mistakes! Mistakes are great learning opportunities. As long as there's no injury and no damage, there's no harm done. And a little blood on the deck isn't an injury.

This is also where you can work out your own changes to the procedure. Just because you saw it done one way doesn't mean that's the only way to do it. That was one way. You can use it as your starting point, and add your own tweaks.

Or maybe you'll realize what you saw really was a good way and you shouldn't mess with it.

You might need to do the procedure more than once before teaching it. Some procedures take practice before you feel confident teaching them to someone else.

The experience of teaching the procedure will be different from the experience of doing it for yourself. Your students may have questions or difficulties that force you to think about things in different ways.

Plus there's the pressure of performing for an audience. But as you gain experience teaching, that will get easier. It's just a different kind of doing.

The experience of teaching is where you finish locking it into your brain.

Teamwork

Sodoto is a great method for dividing up a project. Whether at work, at school, or with your friends, you can divide up the project and have each person take on a part.

They go off and see how it's done, do it themselves until they feel ready, then bring it back to the team to teach everyone else.

What if you can't agree how to divide it up because multiple people want to do the same thing? Fine! Let them!

Each person will have their own take on the experience and teach it slightly differently. That helps explore all the possibilities in the procedure.

More C+-

In The Case For C+-, I talked about writing quick tools in a simple C style, but taking advantage of the C++ standard library, primarily the dynamic data structures. It ends up being C++ without any (or just a few) user-defined classes, so is something of a lightweight object-oriented approach (yes, yes, I'm sure OO purists are barfing at the thought). The main benefit is fast coding.

There I showed as an example the msgresolve tool, which I used to resolve messages logged by an IOT device (the client) and its server. This is a lot of string processing and cross-indexing, with logs containing potentially thousands or tens of thousands of messages.

Shortly after I had completed msgresolve, I needed to have a tool to help me sift through large text files of server logs, logging the TCP connections made by clients and their subsequent activity. I was chasing down a problem where some of the connections were shutting down sooner than expected.

I wasn't sure what was causing the early shutdowns, and wasn't even sure initially which connections had experienced it, so I wanted to be able to gather all the lines for a given connection and list them out for tracing through, for each connection.

That would help me identify the ones that were live at the end of the log sample vs. the ones that had ended early. The log entries for hundreds of connections were all intermixed.

Armed with the methods I had used in msgresolve.cpp, conceptual design was easy. I wanted an ordered list of connections, and associated with each one, the sequential list of log entries associated with the connection.

There were also connections with some internal addresses I wanted to ignore. I could have done this filtering with grep, but it was easy enough to build the capability into the program so that it could stand alone. That also helped me explore some additional string processing functions.

Given that architecture, the data structure I needed was a std::map that mapped a string (the connection identification) to a std::list of strings (the log lines for the connection).

I had the program working in less than an hour. Then I spent at least another hour screwing around with the timestamps in the log entries, figuring out how to process them and deciding what to do with them. Then a little more time on refactoring and cleanup.

Throughout, I used a sample log file that had entries for several connections, including addresses I wanted to skip. I used that as a simple unit test to exercise the code paths.

The resulting code provided the impetus for a simple generalized string processing module, which I'll cover in another post. But you can see some clear patterns emerging in this code.

Doing quick tools like this is fun and very satisfying. It makes your serotonin flow. You have a problem you need to deal with, so you sit down and spew a bunch of code in a short time, refine it, and use the results.

This is actually quite different from long-term product development. That kind of work has its intense coding phases, but once the initial version of the product is out, a lot of the work is much smaller surgical changes.

Even fitting a major new feature in often involves many small bits of code scattered throughout the larger code base, integrating the tendrils. Getting that to work has a different kind of satisfaction.

Design Choices

These tools also give you a chance to think about different approaches. You can balance the variables of memory consumption, CPU consumption, I/O consumption, time, and code complexity (that is, ease of writing and maintaining the code, and compiled code space consumption, not algorithmic complexity) for a given situation.

For instance, the log files I was dealing with had over a million lines of data, some 200MB worth covering hundreds of connections.

That meant I had several choices:

1. I could load all the data into memory and then print it out in an orderly manner. This is a single-pass solution, that consumes large amounts of memory.
2. I could scan the file once, identifying all the individual connections, then for each connection, scan the file from beginning to end to read and print their lines. This a multi-pass solution that requires little memory but significant file I/O.
3. I could scan the file once, and for each identified connection, track the file position of the first and last line, then for each connection, just scan that range of the file. This is a multi-pass solution that reduces the total file I/O for a negligible increase in memory.
4. I could do the same thing, but instead of tracking just the first and last line file positions, build a list of the file position and length of each line, then on each pass, just skip directly to the locations of the lines. This is still multi-pass, but significantly reduces the total file I/O because it only visits each file position twice, requiring a bit more complexity and a bit more memory.

The decision on which choice to use is system-dependent. If memory is cheap and plentiful, and file I/O is relatively expensive, either in terms of time or charges to transfer data over a data link (maybe the data is remote, accessed over a cellular link), then the single-pass solution is best, choice #1.

On a small-memory system, a multi-pass solution is better, and you just have to live with the extra I/O. In that case, choice #4, which is the most complicated code, has the best compromise of low memory and low I/O consumption.

Although if you're really pressed for code space, the simplest multipass solution that iterates over the entire file for each connection is the better choice, #2.

Realistically though, you don't run tools such as this locally on small systems. Where possible, you offload the data to a larger system and run the tools offline.

In this case, I'm running on a Mac with 16GB of memory. Slurping up a 200MB text file and holding everything in memory is nothing. So the single-pass solution is the way to go.

Loading The Data

The log file may just be a chronological sample of all the activity logged by the server, so some connections will already exist at the start of the log, and some will remain at the end. Meanwhile, connection starts and terminations will appear in the log.

The program parses the lines from the log to find connection lines (i.e. some activity for a given connection). It identifies them by looking for a connection ID, which consists of an IPv4 address/port pair (a remote socket ID), and may optionally include a hexadecimal client ID.

It uses just the socket ID as the connection ID, which is the key to the connection map. When it finds a new connection ID, it adds it to the map with an empty list. For each connection line, it finds the connection map entry and appends the line to the list of lines for that connection.

As it loads connection lines, it filters them against the set of IP addresses to skip (these skip address are due to logging of other types of connections besides the client connections). That helps reduce the noise from a large log file.

Printing The Data

The program prints the connections by first iterating through the map and printing out a summary of each one: the connection ID, the number of lines, the duration of the connection data found in the log, and how its lifetime relates to the overall log. Since the map is an ordered structure, printing connections is always in sorted order by connection ID (though string sorted, not numeric sorted).

Then it makes a second pass through the connection map, iterating through each line for each connection and printing it out, with separators between connections. It prints a header and summary line before and after the activity lines for each connection, showing where the connection starts and ends relative to the start and end of the log.

As a last quick change, I added a threshold time value to clearly identify connections that ended at least 60 seconds before the end of the log. This would be a good candidate for a command-line parameter to override the default threshold.

All the output lines have unique greppable features or keywords so you can use other tools for additional postprocessing or extraction. For instance, I could grep out the end summary line of each connection, and maybe the last couple of activity lines before it, to see how each connection ended up. I could use the "threshold exceeded" indication to identify the ones that had ended early.

Some Design Evolution

This program adds the isMember() function, which determines whether a string is a member of a set of strings. Since my usage here was intended to deal with a small set and I had other functions that had similar iterative structure, in the heat of battle I quickly coded it as a linear search of a vector of strings.

That worked fine here, but as I pulled a bunch of this code out into a general string processing module, I realized that was a bad choice, because it's an O(N) search.

That became especially bad when I wanted an overload that took a vector of strings and determined if they were all members. That meant an O(M) repetition of O(N) searches: an O(M*N) or effectively O(N^2) algorithm.

That gets out of hand fast as M and N get larger. Meanwhile, the std::unordered_set is perfect for this, an O(1) algorithm for single searches, and an O(M) algorithm when repeated for M items.

I've left the original isMember() implementation here as an example of the evolution of a concept as you generalize it for other uses.

I also threw in a few overloads that I didn't end up using, but that set the stage for running with the concept in the string processing module. More discussion of that in the post containing the module.

The Result

This program turned out to be another fun exercise in string processing once I had built the basic support functions and could see the problem in those terms. It felt more like working in Python, and in fact just as the dict structure from msgresolve.cpp was inspired by Python, so are the split() and join() functions here.

The funny thing is that it took doing stuff in Python to make me see this approach. That points out one of the advantages of working in multiple different languages: you start seeing opportunities to apply some of the common idioms from one language in another language.

Here's logsplit.cpp:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399

// Usage: logsplit <serverLog> // // Splits a server log file by IPv4 connection. Prints a // summary list of the connections, then the log lines for // each separate connection. // // This is an example of a C++ program that is written mostly // in plain C style, but that makes use of the container and // composition classes in the C++ standard library. It is a // lightweight use of C++ with no user-defined classes. // // 2018 Steve Branam <sdbranam@gmail.com> learntocode #include <iostream> #include <iomanip> #include <sstream> #include <vector> #include <list> #include <map> enum ARGS { ARGS_PROGNAME, ARGS_SERVER_LOG, ARGS_REQUIRED, ARGS_SKIP = ARGS_REQUIRED }; enum SERVER { SERVER_DATE, SERVER_TIME, SERVER_THREAD, SERVER_SEVERITY, SERVER_FUNC, SERVER_CONN, SERVER_TIME_LEN = 16, SERVER_TIMESTAMP_LEN = 28 }; enum CONN { CONN_IP, CONN_PORT, CONN_CLIENT_ID }; enum { END_TIME_THRESHOLD = 60 }; typedef std::string String; typedef std::vector<String> StringVec; typedef std::list<String> StringList; typedef std::map<String, StringList> ConnMap; typedef std::pair<String, StringList> ConnMapEntry; const char *timeFormat = "%Y-%m-%d %H:%M:%S"; StringVec skipIps; size_t lines = 0; size_t skipped = 0; String firstTimestamp; String lastTimestamp; ConnMap connections; StringVec split(const String& str, const char* delim) { char buffer[str.size() + 1]; StringVec strings; strcpy(buffer, str.c_str()); char *token = std::strtok(buffer, delim); while (token != NULL) { strings.push_back(token); token = std::strtok(NULL, delim); } return strings; } String join(const StringVec& strings, const String& sep, size_t start = 0, size_t end = 0) { String str; if (!end) { end = strings.size(); } for (size_t i = start; i < end; ++i) { str.append(strings[i]); if (i + 1 < end) { str.append(sep); } } return str; } bool isMember(const String&str, const StringVec& set) { for (size_t i = 0; i < set.size(); ++i) { if (str == set[i]) { return true; } } return false; } typedef int (*CharMatch)(int c); bool isToken(const String& token, CharMatch isMatch) { if (token.empty()) { return false; } else { for (size_t i = 0; i < token.size(); ++i) { if (!isMatch(token[i])) { return false; } } } return true; } bool isToken(const StringVec& tokens, CharMatch isMatch, size_t start = 0, size_t end = 0) { if (!end) { end = tokens.size(); } for (size_t i = start; i < end; ++i) { if (!isToken(tokens[i], isMatch)) { return false; } } return true; } bool isNumeric(const String& token) { return isToken(token, isdigit); } bool isHex(const String& token) { return isToken(token, isxdigit); } bool isNumeric(const StringVec& tokens, size_t start = 0, size_t end = 0) { return isToken(tokens, isdigit, start, end); } bool isIpv4Address(const String& str) { StringVec tokens(split(str, ".")); return ((tokens.size() == 4) && isNumeric(tokens)); } bool isIpv4Port(const String& str) { return ((str.size() <= 5) && isNumeric(str)); } bool isIpv4Socket(const StringVec& strings) { return ((strings.size() >= 2) && isIpv4Address(strings[0]) && isIpv4Port(strings[1])); } time_t getTime(const String& strTime, const char* format) { std::tm t = {}; std::istringstream ss(strTime); ss >> std::get_time(&t, format); return mktime(&t); } time_t getTime(const String& field) { // Skip opening and closing brackets. return getTime(field.substr(1, SERVER_TIMESTAMP_LEN - 2), timeFormat); } size_t getDuration(const time_t& start, const time_t& stop) { size_t seconds(difftime(stop, start)); return seconds; } size_t getDuration(const String& start, const String& stop) { return getDuration(getTime(start), getTime(stop)); } size_t getDuration(const time_t& start, const String& stop) { return getDuration(start, getTime(stop)); } size_t getDuration(const String& start, const time_t& stop) { return getDuration(getTime(start), stop); } bool isServerTime(const String& str) { if (str.size() == SERVER_TIME_LEN) { for (size_t i = 0; i < str.size(); ++i) { if (!isdigit(str[i]) && (str[i] != ':') && (str[i] != '.') && (str[i] != ']')) { return false; } } return true; } return false; } bool isConnId(const String& str) { StringVec fields(split(str, ":")); return (isIpv4Socket(fields) && (fields.size() < CONN_CLIENT_ID + 1 || isHex(fields[CONN_CLIENT_ID]))); } bool isServerConn(const StringVec& fields) { return ((fields.size() > SERVER_CONN) && isServerTime(fields[SERVER_TIME]) && isConnId(fields[SERVER_CONN])); } bool loadServer(const char* fileName) { FILE* file = std::fopen(fileName, "r"); if (file) { char buffer[1000]; while (std::fgets(buffer, sizeof(buffer), file) != NULL) { String line(buffer); StringVec fields = split(buffer, " \t"); if (isServerConn(fields)) { ++lines; lastTimestamp = line.substr(0, SERVER_TIMESTAMP_LEN); if (firstTimestamp.empty()) { firstTimestamp = lastTimestamp; } strncpy(buffer, fields[SERVER_CONN].c_str(), sizeof(buffer)); StringVec conn = split(buffer, ":"); if (isMember(conn[CONN_IP], skipIps)) { ++skipped; } else { String key(conn[CONN_IP]); key.append(":"); key.append(conn[CONN_PORT]); ConnMap::iterator match; match = connections.find(key); if (match == connections.end()) { connections.insert(ConnMapEntry(key, StringList())); match = connections.find(key); } match->second.push_back(line); } } } std::fclose(file); if (connections.empty()) { std::cout << "No connections found" << std::endl; return false; } return true; } std::cout << "Failed to open server file" << fileName << std::endl; return false; } void printSeparator() { std::cout << std::endl << "=-=-=-=-" << std::endl << std::endl; } void listConnections() { std::cout << connections.size() << " connections " << firstTimestamp << "-" << lastTimestamp << " " << lines << " lines, " << getDuration(firstTimestamp, lastTimestamp) << " sec:" << std::endl; if (skipIps.size()) { std::cout << "(skipped " << skipped << " connections with " << join(skipIps, ", ") << ")" << std::endl; } std::cout << std::endl; for (ConnMap::iterator curConn = connections.begin(); curConn != connections.end(); curConn++) { String conn(curConn->first); StringList connLogs(curConn->second); std::cout << conn << "\t" << connLogs.front().substr(0, SERVER_TIMESTAMP_LEN) << "-" << connLogs.back().substr(0, SERVER_TIMESTAMP_LEN) << " " << connLogs.size() << " lines, " << getDuration(connLogs.front(), connLogs.back()) << " sec" << std::endl; } printSeparator(); } void logConnections() { time_t timeFirst = getTime(firstTimestamp); time_t timeLast = getTime(lastTimestamp); for (ConnMap::iterator curConn = connections.begin(); curConn != connections.end(); curConn++) { String conn(curConn->first); StringList connLogs(curConn->second); size_t duration(getDuration(connLogs.front(), connLogs.back())); std::cout << "Connection " << conn << " " << connLogs.front().substr(0, SERVER_TIMESTAMP_LEN) << "-" << connLogs.back().substr(0, SERVER_TIMESTAMP_LEN) << " " << connLogs.size() << " lines, " << duration << " sec:" << std::endl << std::endl; size_t seconds = getDuration(timeFirst, connLogs.front()); std::cout << firstTimestamp << " Starts " << seconds << " sec after start of log." << std::endl; for (StringList::iterator curLog = connLogs.begin(); curLog != connLogs.end(); curLog++) { std::cout << *curLog; } seconds = getDuration(connLogs.back(), timeLast); std::cout << lastTimestamp << " " << connLogs.size() << " lines, " << duration << " sec. Ends " << seconds << " sec before end of log."; if (seconds > END_TIME_THRESHOLD) { std::cout << " Exceeds threshold."; } std::cout << std::endl; printSeparator(); } } int main(int argc, char* argv[]) { if (argc < ARGS_REQUIRED || String(argv[1]) == "-h") { std::cout << "Usage: " << argv[ARGS_PROGNAME] << " <serverLog> [<skipIps>]" << std::endl; std::cout << "Where <skipIps> is comma-separated list " << "of IP addresses to skip." << std::endl; return EXIT_FAILURE; } else { if (argc > ARGS_REQUIRED) { skipIps = split(argv[ARGS_SKIP], ","); } if (loadServer(argv[ARGS_SERVER_LOG])) { listConnections(); logConnections(); } else { return EXIT_FAILURE; } } return EXIT_SUCCESS; }

The sample log file, conns.log:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[2018-04-03 13:16:29.469659] [0x00007fb3ff129700] [debug] start() 10.1.180.206:30450 TCP socket receive_buffer_size=117708 [2018-04-03 13:16:29.469678] [0x00007fb3ff129700] [debug] start() 10.1.180.206:30450 TCP socket send_buffer_size=43520 [2018-04-03 13:16:29.867381] [0x00007fb3ff129700] [debug] set_idle_send_timeout() 10.1.180.206:30450 set idle send timeout to 60 seconds [2018-04-03 13:16:29.867394] [0x00007fb3ff129700] [debug] set_idle_receive_timeout() 10.1.180.206:30450 set idle receive timeout to 120 seconds [2018-04-03 13:16:29.867450] [0x00007fb3ff92a700] [info] handle_connected() 10.1.180.206:30450 remote connected [2423/8951] [2018-04-03 13:16:29.959877] [0x00007fb3ff129700] [debug] install_connection() 10.1.180.206:30450:00003bd4 linkType 0 [2018-04-03 13:16:29.966599] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0x4a2137a6, 231 bytes [2018-04-03 13:16:29.966935] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0xa11f878a, 35 bytes, msg type 3 [2018-04-03 13:17:29.967117] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x1c35386e, 29 bytes, msg type 1 [2018-04-03 13:17:29.967228] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds [2018-04-03 13:17:30.086722] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0xf6e3f3bf, 29 bytes [2018-04-03 13:17:30.086813] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17 [2018-04-03 13:17:40.086722] [0x00007fb3ff129700] [debug] receive() 10.2.80.206:3050:00000bd4 RX sum 0xf6e3f3bf, 29 bytes [2018-04-03 13:17:40.086813] [0x00007fb3ff129700] [debug] send() 10.2.80.206:3050:00000bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17 [2018-04-03 13:17:30.139377] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0xa0f8a8e1, 78 bytes [2018-04-03 13:18:29.867494] [0x00007fb3ff129700] [debug] handle_idle_receive_timeout() 127.0.0.1:32450:00003bd4 60 seconds [2018-04-03 13:18:29.967315] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds [2018-04-03 13:18:30.086988] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x8fcf53f6, 29 bytes, msg type 1 [2018-04-03 13:18:30.087101] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds [2018-04-03 13:18:30.197029] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0x515f1d4e, 29 bytes [2018-04-03 13:18:30.197120] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x6827d190, 29 bytes, msg type 17 [2018-04-03 13:18:30.249027] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0xda66c5c7, 78 bytes [2018-04-03 13:19:30.087189] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds [2018-04-03 13:19:30.139486] [0x00007fb3ff129700] [debug] handle_idle_receive_timeout() 10.1.180.206:30450:00003bd4 60 seconds [2018-04-03 13:19:30.197322] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x812afb16, 29 bytes, msg type 1

Sample run:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

$ g++ logsplit.cpp -o logsplit $ ./logsplit conns.log 127.0.0.1,3.3.3.3 2 connections [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 25 lines, 181 sec: (skipped 1 connections with 127.0.0.1, 3.3.3.3) 10.1.180.206:30450 [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 22 lines, 181 sec 10.2.80.206:3050 [2018-04-03 13:17:40.086722]-[2018-04-03 13:17:40.086813] 2 lines, 0 sec =-=-=-=- Connection 10.1.180.206:30450 [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 22 lines, 181 sec: [2018-04-03 13:16:29.469659] Starts 0 sec after start of log. [2018-04-03 13:16:29.469659] [0x00007fb3ff129700] [debug] start() 10.1.180.206:30450 TCP socket receive_buffer_size=117708 [2018-04-03 13:16:29.469678] [0x00007fb3ff129700] [debug] start() 10.1.180.206:30450 TCP socket send_buffer_size=43520 [2018-04-03 13:16:29.867381] [0x00007fb3ff129700] [debug] set_idle_send_timeout() 10.1.180.206:30450 set idle send timeout to 60 seconds [2018-04-03 13:16:29.867394] [0x00007fb3ff129700] [debug] set_idle_receive_timeout() 10.1.180.206:30450 set idle receive timeout to 120 seconds [2018-04-03 13:16:29.867450] [0x00007fb3ff92a700] [info] handle_connected() 10.1.180.206:30450 remote connected [2423/8951] [2018-04-03 13:16:29.959877] [0x00007fb3ff129700] [debug] install_connection() 10.1.180.206:30450:00003bd4 linkType 0 [2018-04-03 13:16:29.966599] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0x4a2137a6, 231 bytes [2018-04-03 13:16:29.966935] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0xa11f878a, 35 bytes, msg type 3 [2018-04-03 13:17:29.967117] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x1c35386e, 29 bytes, msg type 1 [2018-04-03 13:17:29.967228] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds [2018-04-03 13:17:30.086722] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0xf6e3f3bf, 29 bytes [2018-04-03 13:17:30.086813] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17 [2018-04-03 13:17:30.139377] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0xa0f8a8e1, 78 bytes [2018-04-03 13:18:29.967315] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds [2018-04-03 13:18:30.086988] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x8fcf53f6, 29 bytes, msg type 1 [2018-04-03 13:18:30.087101] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds [2018-04-03 13:18:30.197029] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0x515f1d4e, 29 bytes [2018-04-03 13:18:30.197120] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x6827d190, 29 bytes, msg type 17 [2018-04-03 13:18:30.249027] [0x00007fb3ff129700] [debug] receive() 10.1.180.206:30450:00003bd4 RX sum 0xda66c5c7, 78 bytes [2018-04-03 13:19:30.087189] [0x00007fb3ff129700] [debug] handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds [2018-04-03 13:19:30.139486] [0x00007fb3ff129700] [debug] handle_idle_receive_timeout() 10.1.180.206:30450:00003bd4 60 seconds [2018-04-03 13:19:30.197322] [0x00007fb3ff129700] [debug] send() 10.1.180.206:30450:00003bd4 TX sum 0x812afb16, 29 bytes, msg type 1 [2018-04-03 13:19:30.197322] 22 lines, 181 sec. Ends 0 sec before end of log. =-=-=-=- Connection 10.2.80.206:3050 [2018-04-03 13:17:40.086722]-[2018-04-03 13:17:40.086813] 2 lines, 0 sec: [2018-04-03 13:16:29.469659] Starts 71 sec after start of log. [2018-04-03 13:17:40.086722] [0x00007fb3ff129700] [debug] receive() 10.2.80.206:3050:00000bd4 RX sum 0xf6e3f3bf, 29 bytes [2018-04-03 13:17:40.086813] [0x00007fb3ff129700] [debug] send() 10.2.80.206:3050:00000bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17 [2018-04-03 13:19:30.197322] 2 lines, 0 sec. Ends 110 sec before end of log. Exceeds threshold. =-=-=-=-