Testing Is How You Avoid Looking Stupid

This is a presentation I gave at the IOT With The Best online conference on October 14, 2017: Testing Is How You Avoid Looking Stupid.

The SlideShare includes an embedded YouTube video recording of my original presentation (I typically watch things like this at 1.5 or 2x speed, selectable from the Settings menu in the YouTube window, which helps me maintain focus).

The abstract:

As IOT products become more pervasive, they have an increasing ability to adversely affect the lives of their users and those around them. Testing is the due diligence that closes the engineering loop to verify proper behavior. Steve will present an introductory overview to testing for IOT products, covering the IOT triad: embedded IOT devices, backend servers, and frontend apps. He'll talk about the consequences of inadequate testing for companies and individual contributors, and levels and types of testing.

Testing is not an absolute guarantor of quality, and you need to have worked out requirements and design to test against, but without doing it, you'll look stupid.

Skimping on testing also means you'll make life miserable for someone. Maybe even kill them.

Books

Doing this presentation turned out to be a bit expensive, because it set me off on a book-buying binge. Fortunately, there's a robust online market in used books.

This went down three paths. First, I wanted to reference the Toyota unintended acceleration problem as a case study. I was familiar with it from reading Risks Digest (my source for all things safety, reliability, security, and usability).

What I found was Professor Philip Koopman at Carnegie Mellon University. He was a plaintiff's expert witness in one of the lawsuits, and had put together a nice presentation on the problem.

But it also turned out he had written a book on embedded systems entitled Better Embedded System Software (available from his site at half off). I ordered the book and read it immediately. It turned out to be a great overview of a broad range of topics on improving embedded system software.

It also listed a number of other books as recommended reading at the end of each chapter. The thing I like about that is these are curated recommendations, helping select which books to read from the vast ocean of books available and raising awareness of obscure areas.

Off to Amazon! And then of course those books had additional recommended reading as I started working my way through them, so more books...

He also has some good videos at his company website, Edge Case Research (he uses Vimeo for his video; I use the Vimeo Repeat And Speed Chrome extension for watching on Vimeo at 2x speed).

Second, a name that leapt out at me on the speaker's list for the conference was Stephen Mellor. Learning the Ward-Mellor method back in the late 80's was an absolute watershed moment for my career. I've applied parts of it informally ever since.

Three minutes into watching his recorded presentation he mentions that he has a new book out on how to take models directly into code for embedded systems. Stop! Google! Book ordered!

And of course as I started reading that one, it referenced others... These books cover Executable UML, which looks like an excellent follow-on to the Ward-Mellor method (unfortunately, I completely missed the boat on Schlaer-Mellor, but xUML also builds on that). One of the benefits I see in xUML is that it imposes rules and discipline on general UML that provide simplifying structure on what is already an extremely complex endeavor.

Third, there were several titles in the many Amazon recommendations as I placed orders that looked interesting, especially having been sensitized to some of the topics by the other books.

It'll take me a while to complete all these, but so far they've been well worth reading, an excellent addition to my bookshelf and another watershed for my career. There will probably be more.

Here's the full list if you're interested in further reading, organized by reference source:

• Philip Koopman:
• Better Embedded System Software, 2010, Philip Koopman.
• Security Engineering: A Guide to Building Dependable Distributed Systems, 2008, Ross Anderson.
• Software Security: Building Security In, 2006, Gary McGraw.
• Writing Secure Code: Practical Strategies and Proven Techniques for Building Secure Applications in a Networked World (Developer Best Practices), 2004, Michael Howard, David LeBlanc.
• Software Architecture: Perspectives on an Emerging Discipline, 1996, Mary Shaw, David Garlan.
• Systems Architecting: Creating & Building Complex Systems, 1991, Eberhardt Rechtin.
• Stephen Mellor:
• Models To Code: With No Mysterious Gaps, 2017, Leon Starr, George Mangogna, Stephen Mellor.
• Executable UML: A Foundation for Model-Driven Architecture, 2002, Stephen J. Mellor, Marc J. Balcer.
• Executable UML How to Build Class Models, 2002, Leon Starr.
• Amazon recommendations:
• Embedded Software Development for Safety-Critical Systems, 2016, Chris Hobbs.
• Real-Time Software Design for Embedded Systems, 2016, Hassan Gomaa.
• Secure Coding in C and C++ (2nd Edition) (SEI Series in Software Engineering), 2013, Robert C. Seacord. This is particularly interesting because Seacord is the author of the CERT C Secure Coding Standard.
• Software Fundamentals: Collected Papers by David L. Parnas, 2001, Daniel M. Hoffman, David M. Weiss, editors.
• Software Architecture in Practice (2nd Edition), Len Bass, Paul Clements, Rick Kazman. I chose this over the 3rd edition due to the case studies listed.
• Documenting Software Architectures: Views and Beyond (2nd Edition), 2011, Paul Clements, et al.
• While I'm here, four other relevant books that I already had and highly recommend:
• Computer-Related Risks, 1994, Peter G. Neumann. A compendium of people and companies looking stupid, from the first decade of Risks Digest.
• Engineering a Safer World: Systems Thinking Applied to Safety, 2012, Nancy Leveson. The page includes a link to a free PDF download of the book under the "Open Access Title" heading, so there's no excuse for not reading it. This has some absolutely hair-raising case studies, and gives a pragmatic approach to understanding how and why systems fail. You'll never again blame it on "human error".
• Real-Time Concepts for Embedded Systems, 2003, Qing Li, Caroline Yao. This is an excellent broad introduction for anyone new to embedded systems, as well as operating systems concepts for multithreaded systems.
• Practical UML Statecharts in C/C++: Event-Driven Programming for Embedded Systems, 2008, Miro Samek. This book is just freakin' brilliant, applying the concepts of UML statecharts in the context of different classes of real-time systems, using the concepts outlined in Li and Yao's book. This could serve as the manual model compiler for xUML.

The book of Parnas' writings deserves special mention because Parnas is one of the greats of the field, like Knuth and Dijkstra. Many important concepts in software engineering can be traced to him; references to these papers pop up all over throughout decades of texts (such as in Gomaa's book!).