tag:blogger.com,1999:blog-64700648537846860942024-03-28T05:02:42.062-07:00Flink And BlinkPointing the way forward and back.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-6470064853784686094.post-82604069199388713182021-08-06T15:36:00.001-07:002021-08-06T15:36:21.838-07:00New Location For Blog Posts<p>I'm blogging these days at <a href="https://www.embeddedrelated.com/blogs-1/nf/Steve_Branam.php" rel="nofollow" target="_blank">EmbeddedRelated.com</a>. See my <a href="https://www.embeddedrelated.com/blogs-1/nf/Steve_Branam.php" rel="nofollow" target="_blank">latest posts</a>.</p><p>Thanks for reading, and I'll see you there!</p><p>Steve</p>Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-29053439653909270362019-11-03T17:15:00.001-08:002019-11-19T12:20:00.088-08:00Review: Clean Agile, by Robert C. Martin, and More Effective Agile, by Steve McConnellThis started out as a review of McConnell's book, but Just-In-Time, my pre-order of <a href="https://blog.cleancoder.com/">Uncle Bob's</a> book arrived Friday. Ah, sweet serendipity! I read it yesterday, and it fits right in.<br />
<br />
I have no idea what the two authors think of each other. I don't know if they're friends, enemies, or frenemies. I don't know if they shake their fists at each other or high-five. But as a software developer, I do believe they're both worth listening to.<br />
<br />
I've read most of the books in Martin's <a href="http://www.informit.com/martinseries">Clean Code series</a>. I'm a big fan. He was one of the original signatories of the <a href="https://agilemanifesto.org/">Agile Manifesto</a>.<br />
<br />
A <a href="https://embeddedartistry.com/blog/2018/12/6/a-look-at-ten-hardware-startup-blunders-part-4-summary-and-recommended-reading">recent post</a> by Phillip Johnston, CEO of Embedded Artistry, set me off on a path reading some of <a href="https://stevemcconnell.com/">Steve McConnell's</a> books and related material. I've become a big fan of his as well.<br />
<br />
Week before last, I read McConnell's <a href="https://amzn.to/2PnJ5sz">Software Estimation: Demystifying the Black Art</a>, 2006. Last week, I read his new book <a href="https://amzn.to/2WhbNN2">More Effective Agile: A Roadmap for Software Leaders</a>, that just came out in August, the one I'm reviewing here.<br />
<br />
This week, I'm reading his <a href="https://amzn.to/2MP9bmD">Code Complete: A Practical Handbook of Software Construction</a>, 2nd edition, 2004, and <a href="https://amzn.to/31NXYag">Software Requirements</a>, 3rd edition, 2013, by Karl Wiegers and Joy Beatty <i>(or maybe over the next few weeks, since they total some 1500 pages; I note that in the Netflix documentary series "Inside Bill's Brain: Decoding Bill Gates", one of his friends says Gates reads 150 pages an hour; that's a superpower, and I am totally jealous!)</i>.<br />
<br />
These are areas where software engineering practice has continually run into problems.<br />
<br />
<b>The Critical Reading List</b><br />
<br />
Martin's and McConnell's new books are excellent, to the point that I can add them as the other half of this absolutely critical reading list:<br />
<ul>
<li><a href="https://amzn.to/2WhKXVf">The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition</a> (2nd edition), 1995, by Frederick P. Brooks, Jr.</li>
<li><a href="https://amzn.to/2PprEIi">Peopleware: Productive Projects and Teams</a>, 3rd edition, 2013, by Tom DeMarco and Timothy Lister.</li>
<li><a href="https://amzn.to/36vuYaT">Clean Agile: Back to Basics</a>, 2019, by Robert C. Martin.</li>
<li><a href="https://amzn.to/32RY1D5">More Effective Agile: A Roadmap for Software Leaders</a>, 2019, by Steve McConnell.</li>
</ul>
<div>
In fact, I would be so bold as to say that <i>not</i> reading these once you know about them constitutes professional negligence, whether you are an engineer, a manager, or an executive. If you deal with software development in any way, producer or consumer, you <i>must</i> read these.</div>
<div>
<br /></div>
<div>
Brooks' first edition outlined the problems in software engineering in 1975. Twenty years later, his second edition showed that we were still making the same mistakes.<br />
<br />
There are a few items that are extremely dated and quaint. Read those for their historical perspective. But don't for a moment doubt the timely relevance of the rest of the book.<br />
<br />
Brooks is the venerated old man of this. <i>Everybody</i> quotes him, particularly <a href="https://en.wikipedia.org/wiki/Brooks%27s_law">Brooks' Law</a>: Adding human resources to a late software project makes it later.<br />
<br />
Every 12 years after Brooks' first edition, DeMarco and Lister addressed the theme from a different perspective in their editions of <i>Peopleware</i>.</div>
<div>
<br /></div>
<div>
Forty-four years after, we are <i>still</i> making the same mistakes, just cloaked in the Agile name. So McConnell's new book addresses those issues in modern supposedly Agile organizations, with suggestions about what to do about them.<br />
<br />
Meanwhile, Martin's book returns us to the roots of Agile, literally back to the basics to reiterate and re-emphasize them. Because many of them have been lost in what Martin Fowler calls <a href="https://martinfowler.com/articles/agile-aus-2018.html">"the Agile Industrial Complex,"</a> the industry that has grown out of promoting Agile.</div>
<div>
<br /></div>
<div>
The first three books are easy reading. McConnell's is roughly equivalent to two of them put together. It also forms the root of a study tree of additional resources, outlining a very practical and pragmatic approach.<br />
<br />
There are clearly some tensions and disagreements between the authors and the way things have developed. Martin goes so far as to include material with dissenting opinions in his book.<br />
<br />
Don't just read these once. Re-read them at least once a year. Each time, different details will feel more relevant as you make progress.<br />
<br />
<b>Problems</b><br />
<br />
<div>
The problems in the industry that have persisted for decades can be summarized as late projects, over budget, and poor software that doesn't do what it's supposed to do or just plain doesn't work.<br />
<br />
Tied up in this are many details. Poor understanding and management of requirements, woefully underestimated work, poor understanding of hidden complexities, poor testing, poor people management.<br />
<br />
Much of it is the result of applying the Taylor <a href="https://en.wikipedia.org/wiki/Scientific_management">Scientific Management</a> method to software development. Taylorism may work for a predictable production line of well-defined inputs, steps, and outputs, running at a repeatable rate, but it is a terrible model for software management. Software development is not a production line. There are far too many unknowns.<br />
<br />
In general, most problems arise because companies practice the IMH software project management method: Insert Miracle Here. With Agile, they have adopted the IAMH variant: Insert Agile Miracle Here.</div>
<br />
But as Brooks writes, <a href="http://faculty.salisbury.edu/~xswang/Research/Papers/SERelated/no-silver-bullet.pdf">there are no silver bullets</a>. Relying on miracles is not an effective project management technique. This is a source of no end of misery for all involved with software.<br />
<br />
As Sandro Mancuso, author of the Clean Code series book <a href="https://amzn.to/2oLrGzc">The Software Craftsman: Professionalism, Pragmatism, Pride</a> (Yes! Read it!) writes in chapter 7 of <i>Clean Agile</i>, "Craftsmanship", "the original Agile ideas got distorted and simplified, arriving at companies as the promise of <i>a process to deliver software faster.</i>" I.e. miracles.<br />
<br />
<b>A Pet Peeve (Insert Rant Here)</b><br />
<br />
<div>
One of the areas of disagreement between various authors is the open-plan office. The original Agile concept was co-locating team members so that they could communicate immediately, directly, and informally, at higher bandwidth than through emails or heavy formal documents. It was meant to foster collaboration and remove impediments to effective communication.<br />
<br />
<i>Peopleware</i> is extremely critical of the open-plan office, and I couldn't agree more. The prevailing implementation of it is clearly based more on the idea of cutting real-estate and office overhead costs than on encouraging productive communication. The result has all the charm of a cattle concentration feedlot, everyone getting their four square feet to exist in.<br />
<br />
Another distortion of the Agile concepts embraced by management at the cost of actual effective development. That might make the CFO happy, but it's a false economy that should horrify the CTO.<br />
<br />
Those capex savings can incur significant non-recurring engineering costs and create technical problems that will incur further downstream development and support costs. And that just means more opex for facilities where the engineering gets done, because the project takes longer.<br />
<br />
You're paying me all this money to be productive and concentrate on complex problems, then you deliberately destroy my concentration to save on furniture and floorspace? It's like a real-life version of Kurt Vonnegut's short story <a href="https://amzn.to/2oLrGzc">Harrison Bergeron</a>. What does that do to the product design and quality? What customer problems does it create, with attendant opportunity costs?<br />
<br />
I turned down an excellent job offer in 2012 after the on-site interviews because of this. I was bludgeoned by my impression of the office environment: sweatshop. They probably thought of me as a prima donna.<br />
<br />
McConnell also recommends against this, referencing the 2018 article <a href="https://www.inc.com/geoffrey-james/its-official-open-plan-offices-are-now-dumbest-management-fad-of-all-time.html">It's Official: Open-Plan Offices Are Now the Dumbest Management Fad of All Time</a>, which summarized the findings of a Harvard study on the topic. The practice appears to me to be the office-space equivalent of Taylorism.</div>
<div>
<br />
Ok, now that I have all that off my chest, on to the actual reviews.<br />
<br /></div>
<b><i>Clean Agile</i>, Robert C. Martin</b><br />
<b><br /></b>
Martin's premise is that Agile has gotten muddled. He says it has gotten blurred through misinterpretation and usurpation.<br />
<br />
His purpose is to set the record straight, "to be as pragmatic as possible, describing Agile without nonsense and in no uncertain terms."<br />
<br />
He starts out with the history of Agile, how it came about, and provides an overview of what it does. He then goes on to cover the reasons for using it, the business practices, the team practices, the technical practices, and becoming Agile.<br />
<br />
An important concept is the <i>Iron Cross</i> of project management: good, fast, cheap, done: pick any three. He says that in reality, each of these four attributes have coefficients, and good management is about managing those coefficients rather than demanding they all be at %100; that is the kind of management Agile strives to enable, by providing data.<br />
<br />
The next concept is Ron Jeffries' <i>Circle of Life</i>: the diagram decribing the practices of XP (eXtreme Programming). Martin chose XP for this book because he says it is the best defined, the most complete, and the least muddled of the Agile processes. He references Kent Beck's <a href="https://amzn.to/33mtnSR">Extreme Programming Explained: Embrace Change</a> (he prefers the original 2000 edition; my copy is due to arrive week after next).<br />
<br />
The enumeration and description of the various practices surprised me, reinforcing his point that things have gotten muddled. While I was aware of them, I was not aware of their original meanings and intent.<br />
<br />
The most mind-blowing moment was reading about acceptance tests, under the business practices. Acceptance tests have become a real hand-waver, "one of the least understood, least used, and most confused of all the Agile practices."<br />
<br />
But as he describes them, they have the power to be amazing:<br />
<ul>
<li>The business analysts specify the happy paths.</li>
<li>QA writes the tests for those cases early in the sprint, along with the unhappy paths (QA engineer walks into a bar; orders a beer; orders 9999 beers; orders NaN beers; orders a soda for <a href="https://xkcd.com/327/">Little Bobby Tables</a>; etc.). Because you want your QA people to be devious and creative in showing how your code can be abused, so that you can prevent anyone else from doing it. You want Machiavelli running your QA group.</li>
<li>The tests define the targets that developers need to hit.</li>
<li>Developers work on their code, running the tests repeatedly, until the code passes them.</li>
</ul>
<div>
Holy crap! <i>Holy crap!</i> This ties actual business-defined requirements end-to-end through to the running code. It is a fractal-zoom-out-one-level application of Test Driven Development (and we all thought TDD was just for the developer-written unit tests!).</div>
<div>
<br /></div>
<div>
It completely changes the QA model. Then the unit and acceptance tests get incorporated into Continuous Build, under the team practices.</div>
<div>
<br /></div>
<div>
There are other important business practices that I believe are poorly understood, such as splitting and spikes. Splitting means splitting a complex story into smaller stories, as long as you maintain the INVEST guidelines:</div>
<div>
<ul>
<li>Independent</li>
<li>Negotiable</li>
<li>Valuable</li>
<li>Estimable</li>
<li>Small</li>
<li>Testable</li>
</ul>
<div>
Splitting is important when you realize a story is more complex than originally thought, a common problem. Rather than trying to beat it into submission (or be beaten into submission by the attempt), break it apart and expose the complexity in manageable chunks.</div>
</div>
<div>
<br /></div>
<div>
I never knew just what a spike was. It's a meta-story, a story for estimating a story. It's called that because it's a long, thin slice through all the layers of the system. When you don't know how to estimate a story, you create a spike for the sole purpose of figuring that out.</div>
<div>
<br /></div>
<div>
Almost as mind-blowing is his discussion of the technical practices. Mind-blowing because much of this whole area has been all but ignored by most Agile implementations. Reintroducing them is one of the strengths of this book.</div>
<br />
Martin has been talking about this for a while. He gave the talk in this video, <a href="https://youtu.be/hG4LH6P8Syk">Robert C. Martin - The Land that Scrum Forgot</a>, at a 2011 conference (very watchable at 2x speed). The main gist is that Scrum covered the Agile <i>management</i> practices, but left out the Agile <i>technical</i> practices, yet they are fundamental to making the methodology succeed.<br />
<br />
These are the XP practices:<br />
<ul>
<li>Test-Driven Development (TDD), the double-entry bookkeeping of software development.</li>
<li>Refactoring.</li>
<li>Simple Design.</li>
<li>Pair Programming.</li>
</ul>
<div>
Of these, I would say TDD is perhaps the most-practiced. But all of these have been largely relegated to a dismissive labeling as something only the extremos do. Refactoring is seen as something you do separately when things get so bad that you're forced into it. Pair programming in particular is viewed as a non-starter.</div>
<div>
<br /></div>
<div>
I got my Scrum training in a group class taught by <a href="https://www.scruminc.com/about-us/">Jeff Sutherland</a>, so pretty much from the horse's mouth. That was 5 years ago, so my memory is a bit faded, but I don't remember any of these practices being covered. I learned about sprints and stories and points, but not about these.</div>
<div>
<br /></div>
<div>
As Martin describes them, they are the individual daily practices that developers should incorporate into every story as they do them. Every story makes use of them in real-time, not in some kind of separate step.</div>
<div>
<br /></div>
<div>
TDD follows the outline I listed in <a href="https://flinkandblink.blogspot.com/2018/08/review-test-driven-development-for.html">Review: Test Driven Development for Embedded C, James W. Grenning</a>.</div>
<div>
<br /></div>
<div>
Refactoring builds on the TDD cycle, recognizing that writing code that <i>works</i> is a separate dimension from writing code that is <i>clean</i>:</div>
<div>
<ol>
<li>Create a test that fails.</li>
<li>Make the test pass.</li>
<li>Clean up the code.</li>
<li>Return to step 1.</li>
</ol>
</div>
<div>
Simple Design means "writing only the code that is required with a structure that keeps it simplest, smallest, and most expressive." It follows Kent Beck's rules:</div>
<div>
<ol>
<li>Pass all the tests.</li>
<li>Reveal the intent (i.e. readability).</li>
<li>Remove duplication.</li>
<li>Decrease elements.</li>
</ol>
<div>
Pair programming is the one people find most radical and alarming. But as Martin points out, it's not an all-the-time 100% thing. It's an on-demand, as-needed practice that can take a variety of forms as the situation requires.</div>
</div>
<div>
<br /></div>
<div>
Who hasn't asked a coworker to look over some code with them to figure something out? Now expand that concept. It's the power of two-heads-are-better-than-one. Maybe trading the keyboard back and forth, maybe one person driving while the other talks. Sharing information, knowledge, and ideas in both directions, as well as reviewing code in real-time. There's some bang for the buck!</div>
<div>
<br /></div>
<div>
The final chapters cover becoming Agile, including some of the danger areas that get in the way, tools, coaching (pro and con), and Mancuso's chapter on craftsmanship, which reminds us that we do this kind of work because we love it. We are constantly striving to be better at it. I <i>am</i> a software developer. I <i>want</i> to be professional about it. This hearkens back to the roots of Agile.<br />
<br /></div>
<b><i>More Effective Agile</i>, Steve McConnell</b><br />
<br /></div>
<div>
McConnell has a very direct, pragmatic writing style. He is brutally honest about what works and what doesn't, and the practical realities and difficulties that organizations run into.</div>
<div>
<br /></div>
<div>
<div>
His main goal is addressing practical topics that businesses care about, but that are often neglected by Agile purists:<br />
<ul>
<li>Common challenges in Agile implementation.</li>
<li>How to implement Agile in only part of the organization (because virtually every company will have parts that simply don't work that way, or will interact with external entities that don't).</li>
<li>Agile's support for predictability.</li>
<li>Use of Agile on geographically distributed teams</li>
<li>Using Agile in regulated industries.</li>
<li>Using Agile on a variety of different types of software projects.</li>
</ul>
He focuses on techniques that have been proven to work over the past two decades. He generalizes non-Agile approaches as <i>Sequential development</i>, typically in some sort of phased form.</div>
<div>
<br /></div>
<div>
</div>
The book contains 23 chapters, organized into these 4 parts:</div>
<div>
<ul>
<li>INTRODUCTION TO MORE EFFECTIVE AGILE</li>
<li>MORE EFFECTIVE TEAMS</li>
<li>MORE EFFECTIVE WORK</li>
<li>MORE EFFECTIVE ORGANIZATIONS</li>
</ul>
<div>
It includes full bibliography and index.<br />
<br />
Throughout, he uses the key principle of "Inspect and Adapt": inspect your organization for particular attributes, then adapt your process as necessary to improve those attributes.</div>
</div>
<div>
<br /></div>
<div>
Another important concept is that Agile is not one monolithic model that works identically for all organizations. It's not one-size-fits-all, because the full range of software projects covers a variety of situations. So the book covers the various ways organizations can tailor the practices to their needs. Probably to the horror of Agile purists.</div>
<div>
<br /></div>
<div>
Each chapter is organized as follows:</div>
<div>
<ul>
<li>Discussion of key principles and details that support them. This includes problem areas and various options for dealing with them.</li>
<li>Suggested Leadership Actions</li>
<li>Additional Resources</li>
</ul>
<div>
The Suggested Leadership Actions are divided into recommended Inspect and Adapt lists. The Inspect items are specific things to examine in your organization. I suspect they would reveal some rude surprises. The Adapt items cover actions to take based on the issues revealed by inspection.</div>
</div>
<div>
<br /></div>
<div>
The Additional Resources list additional reading if you need to delve further into the topics covered.<br />
<br />
One of the very useful concepts in the book is the "Agile Boundary". This draws the line between the portion of the organization that uses Agile techniques, and the portion that doesn't. Even if the software process is 100% Agile, the rest of the company may not be.<br />
<br />
Misunderstanding the boundary can cause a variety of problems. But understanding it creates opportunities for selecting an appropriate set of practices. This is helpful for ensuring successful Agile implementation across a diverse range of projects.<br />
<br />
A significant topic of discussion is the tension between "pure Agile" and the more Sequential methods that might be appropriate for a given organization at a given point in a project.<br />
<br />
The Agile Boundary defines the interface where the methods meet, and which methods are appropriate on each side of it under given circumstances. Again, Agile is not a single monolithic method that can be applied identically to every single project. As he says, it's not a matter of "go full Agile or go home".<br />
<br />
There's a lot of information to digest here, because it all needs to be taken in the context of your specific environment. The chapters that stand out to me based on my personal experience:<br />
<ul>
<li>More Effective Agile Projects: keeping projects small and sprints short; using velocity-based planning (which means you need accurate velocity measurement), delivering in vertical slices, and managing technical debt; and structuring work to avoid burnout.</li>
<li>More Effective Agile Quality: minimizing the defect creation gap (i.e. finding and removing defects quickly, before they get out); creating and using a <i>definition of done</i> (DoD); maintaining a releasable level of quality at all times; reducing rework, which is typically not well accounted for.</li>
<li>More Effective Agile Testing: using automated tests created by the development team, including unit and acceptance tests, and monitoring code coverage.</li>
<li>More Effective Agile Requirements Creation: stories, product backlog, refining the backlog, creating and using a <i>definition of ready</i> (DoR).</li>
<li>More Effective Agile Requirements Prioritization: having an effective product owner, classifying stories by combined business value and development cost.</li>
<li>More Effective Agile Predictability: strict and loose predictability of cost, schedule, and feature set; dealing with the Agile Boundary.</li>
<li>More Effective Agile Adoptions.</li>
</ul>
<div>
Requirements make an interesting area, because that is often a source of problems. The Agile approach is to elicit just enough requirements up front to be able to size a story, then rely on more detailed elicitation and emergent design when working on the story.</div>
<div>
<br /></div>
<div>
But the problem I've seen with that is one of the classic issues in estimation. Management tends to treat those very rough initial estimates as commitments, not taking into account the fact that further refinement has been deferred. So downstream dependent commitments get made based on them.</div>
<div>
<br /></div>
<div>
The risk comes when further examination of the story reveals that there is more work hidden underneath than originally believed. I've seen this repeatedly. Then the whole chain of dependent commitments gets disrupted, creating chaos as the organization tries to cope.<br />
<br />
For example, consumer-product embedded systems are very sensitive to this. The downstream dependent commitments involve hardware manufacturing and the retail pipeline, where products need to be pre-positioned to prepare for major sales cycles such as holidays.<br />
<br />
The Christmas sales period means consumer products need to be in warehouses by mid-November at the latest. Both the hardware manufacturing facilities (and <i>their</i> supply chains) and the sales channels are Taylor-style systems, relying on bulk delivery and just-in-time techniques. They need predictability. That's your Agile Boundary right there, on two sides of the software project.<br />
<br />
IOT products have fallen into the habit of relying on a day 1 OTA update after the consumer unboxes them, but that's risky. If the massive high-scale OTA of all the fielded devices runs into problems, it creates havoc for consumers, who are not going to be happy. That can have significant opportunity costs if it causes stalled revenue or returns, or some horribly expensive solution to work around a failed OTA, not to mention the reputation effect on future sales.<br />
<br />
What about commercial/industrial embedded systems? Cars, planes, factory equipment, where sales, installation, and operation are dependent on having the software ready. These can have huge ripple effects.<br />
<br />
Online portal rollouts that gate real-world services are also sensitive to it. Martin uses the example of healthcare.gov. People need to have used the system successfully by a certain date in order to access real-world services, with life-safety consequences.<br />
<br />
Both of these highlight the real-world deadlines that make business sense for picking software schedule dates. As software engineers, we can't just whine about arbitrary, unreasonable dates. There's a whole chain of dependencies that needs to be managed.<br />
<br />
Schedule issues need to be surfaced and addressed as soon as possible, just like software bugs. The later in the process a software bug is identified, the more expensive it is to fix, sometimes by orders of magnitude. Dealing with schedule bugs is no different.</div>
<div>
<br /></div>
<div>
In his book on estimation, McConnell talks about the <i>Cone of Uncertainty</i>, the greater uncertainty about estimates early in the project, that narrows to better certainty over time as more information is available. Absolute certainty only comes after the completion. But everybody behaves as if the certainty is much better much earlier.</div>
<div>
<br /></div>
<div>
It's clear from the variety of information in this book that Agile is not simply a template that can be laid down across any organization and be successful. It takes work to adapt it to the realities of each organization. There is no simple recipe for success. No silver bullets.</div>
<div>
<br /></div>
<div>
That's why it's necessary to re-read this periodically, because each time you'll be viewing it in the context of your organization's current realities. That's continuing the Inspect and Adapt concept.<br />
<div>
<b><br class="Apple-interchange-newline" />Update Nov 10, 2019</b><br />
<br />
My copy of Beck's <i>Extreme Programming Explained</i> arrived yesterday, and I've been reading through it. Here we see the benefits of going back to original sources, in this case on open plan offices. In Chapter 13, "Facilities Strategy", he says:<br />
<blockquote class="tr_bq">
<i>The best setup is an open bullpen, with little cubbies around the outside of the space. The team members can keep their personal items in these cubbies, go to them to make phone calls, and spend time at them when they don't want to be interrupted. The rest of the team needs to respect the "virtual" privacy of someone sitting in their cubby. Put the biggest, fastest development machines on tables in the middle of the space (cubbies might or might not contain machines).</i></blockquote>
</div>
So it appears what caught on was the group open bullpen part, and what has been left out was the personal space part (and it's attendant value).<br />
<br />
There's a continuous spectrum on which to interpret Beck's recommendation, with the typical modern open office representing one end (all open space, no private space), and individual offices representing the other (no open space, all private space).<br />
<br />
There's a point on the spectrum where I would shift to liking it, if I had a private place to make my own where I could concentrate in relative quiet, with enough space to bring in a pairing partner.<br />
<br />
Where I find the open office breaks down is the overall noise level from multiple separate conversations. It can be a near-constant distraction when I'm trying to work (hence the rampant proliferation of headphones in open offices).<br />
<br />
Meanwhile, when I need to have a conversation with someone, I want to be able to do it without competing with all those others, and without disturbing those around me.<br />
<br />
What seems to me to have the most practical benefit is optimizing space for two-person interactions, acoustically isolated from other two-person interactions. So individual workspaces with room for two to work together. That allows for individual time as well as the pairing method, from simple rubber-duck debugging to full keyboard and mouse back-and-forth.<br />
<br />
Those are both high-value, high-quality times. That's the real value proposition for the company.<br />
<br />
And in fact, that's precisely the kind of setup Beck says Ward Cunningham told him about.<br />
<br />
Given that most developers now work on dedicated individual machines, through which they might be accessing virtualized cloud computing resources, the argument for a centralized bullpen with machines seems less compelling.<br />
<br />
The open bullpen space seems to be less optimal, but still useful for times when more than two people might be involved.<br />
<br />
This is clearly a philosophical difference from Beck's intent, but I think the costs of open plan offices as he experienced them, tempered by the reality of how they've been adopted, outweigh their benefits.<br />
<br />
Meanwhile, his followup discussion in that chapter is fully in harmony with <i>Peopleware's </i>Part II: "The Office Environment".</div>
</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com3tag:blogger.com,1999:blog-6470064853784686094.post-78017527370898898662019-07-08T04:34:00.000-07:002019-07-08T04:47:50.396-07:00Review: Engineering A Safer World, by Nancy Leveson<i>This is a 6-year-old post cross-posted from <a href="http://www.closegrain.com/">my woodworking blog</a> (written before I had this blog available). It remains as timely and important as ever. I'm reposting it motivated by the discussion of the Boeing 737 MAX, such as at <a href="https://embeddedartistry.com/blog/2019/4/1/what-can-software-organizations-learn-from-the-boeing-737-max-saga">EmbeddedArtistry.com</a> (and mentioned at <a href="https://embedded.fm/episodes/290">Embedded.fm</a>).</i><br />
<br />
As a software engineer I've been a dedicated reader of <a href="http://catless.ncl.ac.uk/risks">RISKS DIGEST</a> for over 20 years. Formally, RISKS is the <i>Forum On Risks To The Public In Computers And Related Systems, ACM Committee on Computers and Public Policy</i>, moderated by <a href="http://www.csl.sri.com/users/neumann/neumann.html">Peter G. Neumann</a> (affectionately known to all as PGN).<br />
<br />
RISKS is an online news and discussion group covering various mishaps and potential mishaps in computer-related systems, everything from data breaches and privacy concerns to catastrophic failures of automated systems that killed people. It's an extremely valuable resource, exposing people to many concerns they might otherwise not know about.<br />
<br />
All back issues are archived and available online. It's fascinating to see the evolution of computer-related risks over time. It's also disheartening to see the same things pop up year after year as sad history repeatedly repeats itself.<br />
<br />
<a href="http://sunnyday.mit.edu/">Nancy Leveson's</a> work on safety engineering has been mentioned in RISKS ever since volume 1, issue 1. She's currently Professor of Aeronautics and Astronautics and Professor of Engineering Systems at MIT. Her 2011 book <i>Engineering A Safer World, Systems Thinking Applied to Safety</i>, was noted in <a href="http://catless.ncl.ac.uk/Risks/26.71.html#subj20" style="color: #1155cc;" target="_blank">RISKS 26.71</a>, but has not yet been reviewed there. I offer this informal review.
<br />
<br />
This book should be required reading for anyone who wishes to avoid having their work show up as a RISKS news item. There's no excuse for not reading it: Leveson and MIT Press have made it available as a free downloadable PDF (555 pages), which is how I read it. The download link is available on the book's webpage at <a href="http://mitpress.mit.edu/books/engineering-safer-world" style="color: #1155cc;" target="_blank">http://mitpress.mit.edu/<wbr></wbr>books/engineering-safer-world</a>.
<br />
<br />
This was my first introduction to formal safety engineering, so yes, I speak with the enthusiasm of the newly evangelized.<br />
<br />
The topic is the application of systems theory to the design of safer systems and the analysis of accidents in order to prevent future accidents (not, notably, to assign blame). Systems theory originated in the 1930's and 1940's to cope with the increasing complexity of systems starting to be built at that time.<br />
<br />
This theory holds that systems are designed, built, and operated in a larger sociotechnical context. Control exists at multiple hierarchical levels, with new properties emerging at higher levels ("emergent properties"). Leveson says safety is an emergent property arising not from the individual components, but from the system as a whole. When analyzing an accident, you must identify and examine each level of control to see where it failed to prevent the accident.
<br />
<br />
So while an operator may have been the person who took the action that caused an accident, you must ask why that action seemed a reasonable one to the operator, why the system allowed the operator to take that action, why the regulatory environment allowed the system to be run that way, etc. Each of these levels may have been an opportunity to prevent the accident. Learning how they failed to do so is an opportunity to prevent future accidents.
<br />
<div>
<br /></div>
<div>
Furthermore, systems and their contexts are dynamic, changing over time. What used to be safe may no longer be. Consider that most systems are in use for decades, with many people coming and going over time to maintain and operate them, while much in the world around them changes. Leveson says most systems migrate to states of higher risk over time. If safety is not actively managed to adapt to this change, accidents become inevitable.</div>
<div>
<br /></div>
<div>
Another important point is the distinction between reliability and safety. Components may operate reliably at various levels, yet still result in an accident, frequently due to the interactions between components and subsystems.</div>
<div>
<br /></div>
<div>
Much of Leveson's view can be summarized in two salient quotes. First is a brief comment on the human factor: "Depending on humans not to make mistakes is an almost certain way to guarantee that accidents will happen."</div>
<div>
<div>
<br /></div>
<div>
The second is more involved: </div>
<br />
<blockquote style="border-bottom-style: none; border-color: initial; border-left-style: none; border-right-style: none; border-top-style: none; border-width: initial; margin-bottom: 0px; margin-left: 40px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">
<div>
"Stopping after identifying inadequate control actions by the lower levels of the safety control structure is common in accident investigation. The result is that the cause is attributed to "operator error," which does not provide enough information to prevent accidents in the future. It also does not overcome the problem of hindsight bias. In hindsight, it is always possible to see that a different behavior would have been safer. But the information necessary to identify that safer behavior is usually only available after the fact. To improve safety, we need to understand the reasons people acted the way they did. Then we can determine if and how to change conditions so that better decisions can be made in the future.</div>
</blockquote>
<br />
<blockquote style="border-bottom-style: none; border-color: initial; border-left-style: none; border-right-style: none; border-top-style: none; border-width: initial; margin-bottom: 0px; margin-left: 40px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">
<div>
"The analyst should start from the assumption that most people have good intentions and do not purposely cause accidents. The goal then is to understand <i>why</i> people did not or could not act differently. People acted the way they did for very good reasons: we need to understand why the behavior of the people involved made sense to them at the time."</div>
</blockquote>
</div>
<div>
<br />
The book is organized into three parts. Part I, "Foundations," covers traditional safety engineering (specifically, why it is inadequate) and introduces systems theory. Part II, "STAMP: An Accident Model Based On Systems Theory," introduces System-Theoretic Accident Model and Processes, covering safety constraints, hierarchical safety control structures, and process models. Part III, "Using STAMP," covers how to apply it, including the STPA (System-Theoretic Process Analysis) approach to hazard analysis and the CAST (Causal Analysis based on STAMP) accident analysis method.</div>
<div>
<br /></div>
<div>
Throughout, Leveson illustrates her points with accidents from various domains. These cover a military helicopter friendly-fire shootdown, chemical and nuclear plant accidents, pharmaceutical issues, the Challenger and Columbia space shuttle losses, air and rail travel accidents, the loss of a satellite, and contamination of a public water supply. They resulted in deaths, injuries with prolonged suffering, destruction, and significant financial losses. There's also one fictional case used for training purposes.</div>
<div>
<br /></div>
<div>
The satellite loss was an example where there was no death, injury, or ground damage, but an $800 million satellite was wasted, along with a $433 million launch vehicle (all due to a single misplaced decimal point in a software configuration file). Financial losses in all cases included secondary costs due to litigation and loss of business. Accidents are expensive in both humanity and money.</div>
<div>
<br /></div>
<div>
Several accidents are examined in great detail to expose the complexity of the event and glean lessons, identifying the levels of control, the system hazards they faced, and the safety constraints they violated. They show that the answer to further prevention is not simply to punish the operator on duty at the time. What's to prevent another accident from occurring under a different operator? What systemic factors exist that increase the likelihood of accidents?</div>
<div>
<br /></div>
<div>
These systems affect us every day. During the time I was reading the book, there was an airline crash at San Francisco, a fiery oil train derailment in Canada, and a major passenger train derailment in Spain. I started reading it while a passenger on an aircraft model mentioned 14 times in the book, and read the remainder while traveling to and from work on the Boston commuter rail.</div>
<div>
<br /></div>
<div>
The book can be read on several levels. At a minimum, the cases studies and analyses are horribly fascinating for the lessons they impart. Fans of <i>The Andromeda Strain</i> will be riveted.</div>
<div>
<br /></div>
<div>
As I read the account of two US Black Hawk helicopters shot down by friendly fire in Iraq, I could visualize a split screen showing the helicopters flying low in the valleys of the no-fly zone to avoid Iraqi air defense radar, the traces going inactive on the AWACS radar scopes, the F-15's picking up unidentified contacts that did not respond to IFF, and the mission controllers back in Turkey, as events ground to their inexorable conclusion. It made my hair stand on end.</div>
<div>
<br /></div>
<div>
All the case studies are equally jaw-dropping, down to the final example of a contaminated water supply in Ontario. Further shades of <i>Andromeda</i>, since that was a biological accident resulting in deaths.<br />
<br />
They're all examples of systems that migrated to very high risk states, where they became accidents waiting to happen. It was just a matter of which particular event out of the many possible triggered the accident.</div>
<div>
<br /></div>
<div>
Part of what's so shocking about these cases is the enormously elaborate multilayered safety systems that were in place. The military goes to great lengths in its air operations control to avoid friendly fire incidents, the satellite software development process had numerous checkpoints, NASA had a significant safety infrastructure.<br />
<br />
Yet it seems that this very elaborateness contributed to a false sense of safety, with uncoordinated control leaving gaps in coverage. In some cases this led to complacency that resulted in scaling back safety programs.</div>
<div>
<br /></div>
<div>
The other shocking cases were at the opposite end of the spectrum, where plants were operated more fast and loose.</div>
<div>
<br /></div>
<div>
The one bright spot in the case studies is the description of the US Navy's SUBSAFE program, instituted after the loss of the <i>USS Thresher</i> in 1963. It flooded during deep dive testing; despite emergency recovery attempts by the crew, they were unable to surface. Just pause and think about that for a moment.</div>
<div>
<br /></div>
<div>
SUBSAFE is an example of a tightly focused and rigorously executed safety program. The result is that no submarine has been lost in 50 years, with the exception of the <i>USS Scorpion</i> in 1968, where the program requirements were waived. The result of that tragic lesson was the requirements were never again waived.</div>
<div>
<br /></div>
<div>
The book can be read at an academic level, as a study of the application of systems theory to the creation of safer systems and analysis of accidents. It can be read at an engineering level, as a guide on how to apply the methodology in the development and operation of such systems. It's not a cookbook, but it points you in the right direction. It includes an extensive bibliography for follow-up.</div>
<div>
<br /></div>
<div>
Even those who work on systems that don't present life safety or property damage risks can benefit, because any system behaving poorly can make people's lives miserable. They frequently pose significant business risks, affecting the life and death of a company.</div>
<div>
<br /></div>
This book paired with PGNs book <a href="http://www.amazon.com/gp/product/020155805X/ref=as_li_qf_sp_asin_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=020155805X&linkCode=as2&tag=closegrain-20">Computer-Related Risks</a><img alt="" border="0" src="http://ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=as2&o=1&a=020155805X" height="1" style="border: none !important; margin: 0px !important;" width="1" /> would make an excellent junior or senior level college survey course for all engineering fields, along the lines of "with great power comes great responsibility". While some might feel it's a text more suited to graduate level practicum, I think it's worth conveying at the undergraduate level for broader distribution.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-38921956835339191122018-11-23T12:36:00.000-08:002018-11-24T09:21:02.676-08:00Review: Web-Based Course Test-Driven Development For Embedded C/C++, James W. Grenning<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJAv5T4fZoQW-tvQvfnSY9wYoeIf3y0zqJB9vRQE5-W-8ha5NKoVXHydTrH1TsdVqZZ0lH4qKbMyHkd4BBHprdvdHUzMQ_4vUXeIg23w1Rav3qmoaBU4q-oOhDlOIbQ6UlsBQ0blKorcJp/s1600/GrenningCourseCertificate.PNG" imageanchor="1"><img border="0" data-original-height="612" data-original-width="793" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJAv5T4fZoQW-tvQvfnSY9wYoeIf3y0zqJB9vRQE5-W-8ha5NKoVXHydTrH1TsdVqZZ0lH4qKbMyHkd4BBHprdvdHUzMQ_4vUXeIg23w1Rav3qmoaBU4q-oOhDlOIbQ6UlsBQ0blKorcJp/s400/GrenningCourseCertificate.PNG" width="500" /></a>
<br />
<i>Full disclosure: I was given a seat in this course by James Grenning.</i><br />
<i></i><br />
I took <a href="https://wingman-sw.com/">James Grenning's</a> 3-day web-based course <a href="https://wingman-sw.com/training/remote-delivered-tdd">Test-Driven Development for Embedded C/C++</a> September 4-6, 2018. It was organized as a live online delivery, 5 hours each day. The schedule worked out perfectly for me in Boston, starting at 9AM each morning, but he had attendees from as far west as California and as far east as Germany, spanning 9 time zones.<br />
<br />
The participants ranged from junior level embedded developers to those with more than 20 years of experience. One worked in a fully MISRA-compliant environment. This was the first introduction to TDD for some of them.<br />
<br />
The course was organized as blocks consisting of presentation and discussion, coding demo, then live coding exercises. It used <a href="http://cpputest.github.io/">CppUTest</a> as the TDD framework.<br />
<br />
The short answer: this is an outstanding course. It will change the way you work. I highly recommend it, well worth the investment in time and money. The remote delivery method worked great.<br />
<br />
I had previously read Grenning's book, <a href="https://amzn.to/2n03PqX">Test Driven Development for Embedded C</a>, which I <a href="http://flinkandblink.blogspot.com/2018/08/review-test-driven-development-for.html">reviewed</a> in August. I covered a lot of his technical information on TDD in the review, so I'll only touch on that briefly here. He covers the same material in the course presentation portions.<br />
<br />
The course naturally has a lot of overlap with the book, so each can serve as a standalone resource. But I found the combination of the two to be extremely valuable. They complement each other well because the book provides room to delve more deeply into background information, while the course provides guided practice time with an expert.<br />
<br />
Reading the book first meant I was fully grounded in the motivations and technical concepts of TDD, so I was ready for them when he covered them in the course. I was also already convinced of its value. What the live course brings to that part is the opportunity to ask questions and discuss things.<br />
<br />
You can certainly take the course without first reading the book, which was the case for several of the participants.<br />
<br />
<br />
<b>Presentations</b><br />
<div>
<b><br /></b></div>
<div>
For the presentation portions, Grenning covered the issues with development using non-TDD practices, what he calls "debug-later programming" (DLP). This consists of a long phase of debug fire-fighting at the end of development, that often leaves bugs behind.</div>
<div>
<br /></div>
<div>
He introduced the TDD microcycle, comparing <a href="http://blog.wingman-sw.com/archives/16">the physics of DLP to the physics of TDD</a>. By physics he means the time domain, the time taken from injection of a bug (repeat after me: "I are a ingenuer, I make misteaks, I write bugs") to its removal. This is one of the most compelling arguments for adopting TDD. TDD significantly compresses that time frame.</div>
<div>
<br /></div>
<div>
He covered how to apply the process to embedded code and some of the design considerations. He also talked about the common concerns people have about TDD.</div>
<div>
<br /></div>
<div>
One quote from <a href="https://en.wikipedia.org/wiki/Kent_Beck">Kent Beck</a> that I really liked:</div>
<blockquote class="tr_bq">
<i>TDD is a great excuse to think about the problem before you think about the solution.</i></blockquote>
He covered the concept of test fakes and the use of "spies" to capture information. He covered mocks as well, including mocking the actual hardware so that you can run your tests off-target.<br />
<br />
He covered refactoring to keep tests easy to follow and maintain. He also covered refactoring of "legacy code" (i.e. any production code that was not built using TDD), including "code smells" and "code rot", using TDD to provide a safety harness. This included a great quote from <a href="https://en.wikipedia.org/wiki/Donald_Knuth">Donald Knuth</a> (bold emphasis mine):<br />
<blockquote class="tr_bq">
<i>Let us change our traditional attitude to the construction of programs. Instead of imagining that our main task is to instruct a computer what to do, <b>let us concentrate rather on explaining to human beings what we want a computer to do</b>.</i></blockquote>
<div>
<br /></div>
<b>Coding Demos</b><br />
<b></b><br />
Grenning performed two primary live coding demos. First, he used TDD to build a circular buffer, a common data structure in embedded systems. He used this to demonstrate the stepwise process of applying the TDD microcycle.<br />
<br />
Second, he performed a refactoring demo on a set of tests. He used this to show how to apply the refactoring steps to simplify the tests and make the test suite more readable and maintainable.<br />
<br />
This was just as valuable as the TDD microcycle, because a clean test suite means it will live a long and useful life. Failing to refactor and keep it clean risks making it a one-off throwaway after getting its initial value.<br />
<br />
<br />
<b>Coding Exercises</b><br />
<b><br /></b>
Grenning uses <a href="http://www.cyber-dojo.org/">Cyber-Dojo</a> to conduct exercises (as well as his demos). This is a cloud-based, Linux VM, ready-to-use, code-build-run environment that allows each student to work individually, but he can monitor everyone's code as they work. This turned out to be one of the most valuable aspects of the course.<br />
<br />
I should also mention that I had read <a href="http://langrsoft.com/">Jeff Langr's</a> book <a href="https://amzn.to/2xJmpZ9">Modern C++ Programming with Test-Driven Development: Code Better, Sleep Better</a> in between reading Grenning's book and taking this course. Langr puts a lot of emphasis on short, easily-readable tests, and that's something that also comes out in Grenning's class.<br />
<br />
What was so valuable about doing these exercises in Cyber-Dojo is that Grenning was able to stop someone who was heading off in the wrong direction and quickly bring them back on track, or help them if they weren't sure what to do next. That fast feedback cycle is very much in tune with TDD itself. It works just as well as a teaching method.<br />
<br />
So if someone started writing code without a test, or wrote too much code for what the test covered, or had too much duplication in tests, or had too much setup that could be factored out, he let them know and guided them back. In some cases he interrupted the exercise to go through someone's code on the screen with everybody watching, not to put them on the spot, but to cover the issues that we all run into.<br />
<br />
That's critical because learning to truly work effectively in TDD style requires a reorientation of your thinking. We all have the coding habits of years that need to be overcome.<br />
<br />
That doesn't happen automatically just because you read a book and have decided to follow it. It takes effort; half the effort is just noticing that you're straying from the intended path. That's the value of having a live instructor who can watch over your shoulder. It's like being an apprentice under the watchful eye of a master craftsman.<br />
<br />
For me, this was ultimately the greatest value in the class. Having Grenning provide real-time guidance had an immediate effect on my coding, for both the test code and the production code. Whether it was talking about my mistakes or someone else's, I was able to immediately improve my work.<br />
<br />
That made a huge difference between the test code I wrote before the class and the test code I wrote by the end of the class.<br />
<br />
The coding exercises were building our own circular buffer, building a light controller spy, using TDD with the spy to implement a light scheduler, and implementing a flash chip driver. Note that these exercises are also covered in his book.<br />
<br />
I also found that Cyber-Dojo made for an interesting example of pair programming, something I've never done before. Grenning provided initial files to work on, like a pair partner guiding you in the next step, then provided active feedback, like a partner asking questions and making suggestions: "Are you missing something there? What if you tried this? Wait, before you do that...".<br />
<br />
<br />
<b>The Big Lesson</b><br />
<b><br /></b>
The big lesson for me from this course was that it finally drove home that TDD is <b>ALL ABOUT DEVELOPMENT</b>! Sometimes I have to be clubbed over the head for something to really sink in, and that's what happened here.<br />
<br />
We get so focused on the word "test" in TDD that we jump to the conclusion that it's just a test methodology. We emphasize test, as in <b>TEST</b>-Driven Development.<br />
<br />
But really, the emphasis should be reversed, it's Test-Driven <b>DEVELOPMENT</b>. That means you apply design concepts and address the requirements of the product as you engage in a very active development thought process that is driven forward by tests.<br />
<br />
Did you ever write some throwaway test code just so you could see how something worked, or to explore some design ideas? Hmmm, well TDD formalizes that.<br />
<br />
The fact that you do end up with useful unit tests is almost a side effect of the process. An extremely valuable side effect, but a side effect nonetheless.<br />
<br />
The real output of the process is <b>working production code</b>. That's what really matters. That's the real goal.<br />
<br />
At some point on the last day of the course, I recognized the change in emphasis deep in my being. Maybe the difference is subtle, but it is critical.<br />
<br />
That recognition first started to dawn after I read the book and applied it at work. I was amazed at the cleanliness of the resulting code. It was <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a> and <a href="https://medium.com/mutual-of-omaha-digital-experience-and-design-team/damp-programming-reviving-readability-d84647cc5b2e">DAMP</a> and <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a>, with no further refinement or debugging required.<br />
<br />
Yes, I had a unit test suite. But look at the production code! It was breathtaking, right out of the chute. <i>That</i> was motivating.<br />
<br />
It was in that receptive frame of mind that I did the coding exercises in the course. That was when the club hit. It was one of those moments of realization where you divide time into what came before, and what came after, the physical moment of grok, providing a whole new lens through which to perceive the work.<br />
<br />
Savor that consideration for a moment.<br />
<br />
People have been saying for years that TDD is about <i>development</i>, but we tend to focus on the test. Grenning emphasizes <i>development</i> when he talks about developing while "test-driving", meaning he is doing his development driven by tests. I guess it just takes time for the real implications to sink in.<br />
<br />
One of Grenning's slides quotes <a href="https://en.wikipedia.org/wiki/Edsger_W._Dijkstra">Edsger Dijkstra</a>:<br />
<blockquote class="tr_bq">
<i>Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result, the programming process will become cheaper. If you want more effective programmers, you will discover that <b>they should not waste their time debugging, they should not introduce the bugs to start with</b>.</i></blockquote>
While we all aspire to be like Dijkstra, this seems like a pipe dream. <i>Until you realize that TDD does exactly that.</i> It provides the <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm">shortest path</a> to working software. I think he would have liked that.<br />
<br />
Now that I've relegated the test aspect of this to second-class citizenship, let me bring it back to prominence.<br />
<br />
The testing aspect approaches Dijkstra's ideal, because it finds bugs immediately as part of the <i>code, build, test</i> cycle. So the bugs are squashed before they've had time to scatter and hide in the dark corners. That reduces the dreaded unbounded post-development debug cycle to near zero.<br />
<br />
If you don't let bugs get into the code, you won't have to spend time later removing them. Yeah, what Dijkstra said.<br />
<br />
This doesn't guarantee bug-free code. There might still be bugs that occur during the integration of parts that are working (for example, one module uses feet, while another uses inches), or the code may not have addressed the requirements properly (the customer wanted a moving average of the last 5 data points, while the code uses the average of all data points), but as a functional unit, each module is internally consistent and working according to its API.<br />
<br />
The resulting unit test suite is an extremely valuable resource, just as valuable as the production code. What makes it so valuable? Two things: safety harness, and example usage.<br />
<br />
It provides a safety harness to allow you to do additional work on the code, then run the suite as a regression test to prove you haven't broken anything. Or to detect breakage so you can fix it immediately.<br />
<br />
Using and extending the suite liberates you to make changes to the code safely. Need to add some functionality? Fix one of those integration or requirements bugs? Refactor for better performance or maintainability? Clean up some tech debt? Have at it.<br />
<br />
You can instantly prove to yourself that you haven't screwed anything up, or show that you have, so that you can fix it before it ever gets committed to the codebase. No one will ever see that dirty laundry.<br />
<br />
It provides example usage, showing how to use the API: how to call the various functions, in what order, how to setup and drive various behavioral scenarios, how to exercise the interfaces for different functional behaviors, how different parameters affect things, how to interpret return values.<br />
<br />
This is real, live code, showing how to use the production code as a client. You can even get creative and add exploratory tests that push the production code in odd directions to see what happens. Grenning calls these <i>characterization tests</i> and <i>learning tests</i>.<br />
<br />
The test suite is actually something quite magical: self-updating documentation! Since you need to invest the time to maintain the tests in order to get the development done, you are also automatically updating the example usage documentation <i>for free</i>.<br />
<br />
You might argue that tools like Doxygen offer similar self-updating capability, but they still require updating textual explanations along with the code. They are subject to the same staleness that can happen with any comments, where the comments (or Doxygen annotations) aren't kept up to date with code changes (see Tim Ottinger's <a href="http://agileinaflash.blogspot.com/2009/04/rules-for-commenting.html">Rules for Commenting</a> for advice to help avoid stale comments).<br />
<br />
But if you want to really know how to use the production code, go read the tests! If you've truly followed the TDD process as Grenning shows you in this course, they will tell you how to produce every bit of behavior that it is capable of, because every bit of behavior implemented will have been driven by the tests.<br />
<br />
That's the full-circle, closed-loop feedback power of test-driven DEVELOPMENT.<br />
<br />
Doxygen still has its place. I think of the Doxygen docs as API <i>reference</i>, while the test suite is API <i>tutorial</i>, showing actual usage.<br />
<i></i><br />
<br />
<b>Another Lesson</b><br />
<br />
I've already alluded to the other interesting lesson that I drew from this course:<i> it takes practice!</i> We're not used to working like this, so it takes practice and self-awareness to learn how to do it.<br />
<br />
That was particularly driven home by the coding exercises. Even though I had just read his book and followed through the exact same exercises, and read Langr's book, and applied the knowledge at work, I still had trouble getting rolling on the first couple of exercises. It was a matter of instilling the new habits.<br />
<br />
It took a few times having Grenning redirect me (or listen to the advice he gave someone else). By the final exercise, after the benefit of his live feedback, I was able to catch myself in time and start applying the habits on my own.<br />
<br />
It's still going to take some time. I'll know I've gotten there when I start thinking of the tests automatically as the first step of coding.<br />
<br />
<br />
<b>Third Time's A Charm</b><br />
<div>
<br /></div>
<div>
At one point in the discussion I mentioned that Grenning's book and this course represented my third attempt at using TDD, and one of the participant said he would be interested in hearing about my previous attempts.<br />
<br />
My first attempt was in 2007, when I was introduced to TDD by a coworker. I read Kent Beck's <a href="https://amzn.to/2S4msHE">Test Driven Development: By Example</a> and used it to develop the playback control module for a large video-on-demand server intended for use in cable provider head ends.<br />
<br />
This was both a great success and a classic failure. It was a great success in that it accelerated my work on the module, avoiding many bugs and shortening the debug cycle. In that respect it lived up to the promise of TDD completely.<br />
<br />
It was a classic failure in that I made the tests far too brittle. I put too much internal knowledge of the module in them, with many internal details that were useful when I was first developing the module, but that became a severe impediment to ongoing maintenance.<br />
<br />
The classic symptom of this problem was that a minor change in implementation would cause a cascade of test failures. The production code was fine, but some internal detail such as a counter value that was being checked by the tests had changed. Otherwise the test code itself was also fine. But I had overburdened it with details that should have been hidden by encapsulation.<br />
<br />
The result was that ultimately I had to abandon the test suite. It had provided good initial value, but failed to deliver on-going value because it became a severe maintenance burden.<br />
<br />
This is exactly the type of situation that Grenning's course seeks to prevent. During coding exercises, he watches out for cases of inappropriate information exposure. Thus another benefit of this is improved encapsulation and information hiding.<br />
<br />
My second attempt was in 2013, when I wanted to refactor some of the code in an IP acceleration server as part of improvements to one of its features. I had read Michael Feathers' <a href="https://amzn.to/2DUC0ei">Working Effectively with Legacy Code</a>, and found that many of the things he covered applied to the codebase I was working on.<br />
<br />
This was a revenue-generating service product, so I needed to be sure I didn't break it.<br />
<br />
The main strategy the book covers is to use TDD to provide that safety harness I mentioned above, in order to verify that the legacy code behaves the same after modification as it did before.<br />
<br />
I began building a set of test fakes that could be used with Google Test. One issue was that the code relied heavily on the singleton pattern, so there always had to be some implementation of each class that would satisfy the linker. And of course there were chains of such dependencies interlocked in a web.<br />
<br />
My first task was to replace that bit by bit with dependency injection. I focused just on the parts necessary to allow me to test the area I was modifying. Part of Feathers' strategy is to tackle just enough of the system at a time to be able to make progress, rather than a wholesale break-everything-down-and-rebuild approach.<br />
<br />
I had enough success with this that once I finished my primary work on the feature changes, I embarked on a background project to put the entire codebase into 100% dependency injection. That would allow me to build unit tests for any arbitrary component, in combination with any set of faked dependencies, with the longer-term goal of building out near-100% unit test coverage incrementally.<br />
<br />
However, not too long after starting this, I ended up changing jobs. So once again I got the short-term benefit from TDD, but didn't reap the long-term benefit. It was a useful exercise to go through, though, providing good experience on how to migrate such a codebase to TDD.<br />
<br />
This is another area that Grenning's course covers.<br />
<br /></div>
<br />
<b>Related Links</b><br />
<b><br /></b>For the perspective of another class participant, see <a href="http://embeddedartistry.com/">Phillip Johnston's</a> post <a href="https://embeddedartistry.com/blog/2018/9/14/what-i-learned-at-james-grennings-remote-tdd-course">What I Learned from James Grenning's Remote TDD Course</a>.<br />
<br />
There are things about the TDD process that make people suspicious. Is it just hacking? In <a href="http://www.ganssle.com/articles/aninterviewwithjamesgrenning.htm">this interview with Grenning</a>, embedded systems expert <a href="http://www.ganssle.com/">Jack Ganssle</a> raises some of those concerns. Grenning explains how the process works to reach the goal of well-designed, working production code that meets customer requirements.<br />
<br />
<a href="https://www.embedded.fm/">Elecia and Christopher White</a> have a great <a href="https://www.embedded.fm/episodes/109-repeat">interview podcast with Grenning</a>. Best joke: how many Scrum masters does it take to manage one developer? Also good Shakespeare and Bradbury quotes that are much ado about programming.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-45800284909599769132018-11-16T04:57:00.000-08:002018-11-16T05:11:06.992-08:00Accuracy Vs. PrecisionThis is nothing new, but it's something that needs to be constantly hammered home. It's an important point that can make a critical difference in the behavior of embedded systems interacting with the sloppiness of physics in the real world.<br />
<div>
<br /></div>
<div>
I was reminded of the topic by <a href="https://www.embedded.fm/">Elecia White's</a> excellent video <a href="https://youtu.be/Hq42Vv9VXtM">Intro to Inertial Sensors: From Taps to Gestures to Location</a>. The inertial sensors that are now common in smartphones and embedded systems are <a href="https://en.wikipedia.org/wiki/Accelerometer">accelerometers</a>, <a href="https://en.wikipedia.org/wiki/Gyroscope">gyroscopes</a>, and <a href="https://en.wikipedia.org/wiki/Magnetometer">magnetometers</a>, possibly integrated into a single <a href="https://en.wikipedia.org/wiki/Inertial_measurement_unit">Inertial Measurement Unit</a> (IMU).</div>
<div>
<br /></div>
<div>
These amazing devices are <a href="https://en.wikipedia.org/wiki/Microelectromechanical_systems">MicroElectroMechanical Systems</a> (MEMS), explained in <a href="https://howtomechatronics.com/">Dejan Nedelkovski's</a> equally excellent video <a href="https://youtu.be/eqZgxR6eRjo">How MEMS Accelerometer Gyroscope Magnetometer Work & Arduino Tutorial</a>.</div>
<div>
<br /></div>
<div>
But working in the digital world with sensor data converted from the analog world poses interesting problems. Some of these are addressed in <a href="http://jackcrenshaw.com/">Jack W. Crenshaw's</a> amazing book <a href="https://amzn.to/2RZpw84">Math Toolkit for Real-Time Programming</a>. There is always error in the system to some degree, so you have to be prepared to handle it.</div>
<div>
<br /></div>
<div>
Accuracy and precision are two of those problems, and have been since the dawn of measurement. It's important to understand the distinction between them. They are often confused in informal usage.<br />
<br />
A common analogy for understanding them is taken from riflery, showing a shooting target. White includes a version of it in her video. As I learned while earning the Boy Scout riflery merit badge at <a href="https://resicafalls.org/">Resica Falls</a> summer camp lo these many years ago, you want your shots to be tightly grouped together (precision), and you want that group to be on-target, centered around the bull's-eye (accuracy).<br />
<br />
The following image is taken from the NOAA article <a href="https://celebrating200years.noaa.gov/magazine/tct/tct_side1.html">Accuracy Versus Precision</a>, which does a nice job of explaining the difference. I'll briefly restate it here should NOAA scientific information mysteriously disappear from the Web.<br />
<br />
<b>Accuracy</b> is how close a measurement is to the true value, how close it is to the bull's-eye. <b>Precision</b> is how closely repeated measurements come to duplicating measured values, how tightly they are grouped.</div>
<div>
<br /></div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbD5k8I_2xVjl4Py2SyFvfcThD0Hc0SkFEAEogUN3VewYQf9jkRJB5ok-QMCWczlCBME0dxn1HlHeGi3prFat4OnbGUWLRZVCheQWzBxsFCxqBqMeRqXuSI2vObktOeJZBaTOKt5WwcSqP/s1600/accuracy_vs_precision_556.jpg" imageanchor="1"><img border="0" data-original-height="527" data-original-width="556" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbD5k8I_2xVjl4Py2SyFvfcThD0Hc0SkFEAEogUN3VewYQf9jkRJB5ok-QMCWczlCBME0dxn1HlHeGi3prFat4OnbGUWLRZVCheQWzBxsFCxqBqMeRqXuSI2vObktOeJZBaTOKt5WwcSqP/s1600/accuracy_vs_precision_556.jpg" /></a>
<br />
<b><br /></b>
<b>Not Accurate Not Precise:</b> these are not close to the bull's-eye, so the measurements are not close to the true value, and they are not tightly grouped, so repeated measurements have a lot of difference.<br />
<br />
<b>Accurate Not Precise:</b> these are close to the bull's-eye, so the measurements are centered around the true value, but they are not tightly grouped, so repeated measurements range all over the place.<br />
<b><br /></b>
<b>Not Accurate Precise:</b> these are not close to the bull's-eye, so the measurements are not close to the true value, but they are tightly grouped, so repeated measurements are close to each other. From a riflery perspective, this is good, because it means you have control, you just need to adjust your sight to compensate.<br />
<b><br /></b>
<b>Accurate Precise:</b> these are both close to the bull's-eye and tightly grouped. The measurements are on-target, close to the true value, and repeated measurements give close to the same result.<br />
<br />
In an embedded system, you need to characterize and calibrate things. Characterization means understanding how much variation a sensor has in its measurements, how precise it is (which, as White explains, can vary with temperature and barometric pressure, plus humidity, external vibration, external electrical and magnetic fields, external sources of <a href="https://en.wikipedia.org/wiki/Electromagnetic_interference">Radio Frequency Interference</a> (RFI), and other factors; man, the real world is a sloppy place!).<br />
<br />
Calibration determines how far off the measurements are from the true value and adjusts the values to compensate for the difference.<br />
<br />
Meanwhile, the calculations that use the values must be able to handle the accuracy and precision appropriately, along with odd cases such as a true zero value being measured as a small negative value (because the measurement is centered around zero, but may range between small negative and positive limits). Treating values as if they are more accurate or precise than they really are is downright dangerous.<br />
<br />
That can lead the embedded system to crash or take inappropriate actions. If it happens to be controlling the flight of an airplane or the operation of a chemical plant, people can be killed and tremendous damage can result. If it happens to be controlling a consumer device, the consequences may be less dire, but can be equally damaging to the company.<br />
<br />
There is always error and noise in the system. You have to understand it and how to manage it.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-14814233568340678282018-09-22T06:35:00.000-07:002020-01-31T16:06:05.598-08:00So You Want To Be An Embedded Systems Developer<i>Then listen now to what I say.</i><br />
<i>Just get an electric guitar</i><br />
<i>and take some time and learn how to play.</i><br />
<br />
Oh, wait, that's <a href="https://en.wikipedia.org/wiki/So_You_Want_to_Be_a_Rock_%27n%27_Roll_Star">a song by the Byrds</a>. But the strategy is the same. Get some information and tools and learn how to use them. No need to sell your soul to the company.<br />
<br />
The items I've listed below are sufficient to get you started on a career as an embedded systems developer. There are of course many additional resources out there. But these will arm you with enough knowledge to begin.<br />
<br />
I own or have watched every resource and piece of hardware listed on this page. I've either gone through them entirely, or am in the process of doing so. I can vouch for their usefulness in getting you up to speed. It's a firehose of learning.<br />
<br />
My personal learning method is to bounce around between multiple books and videos in progress, while spending time hands-on with the hardware. This is similar to a college student juggling multiple classes with labs (without tests, term papers, or due dates!).<br />
<br />
Your method may be different. Feel free to approach things in a different order. I offer this in the spirit of <a href="https://flinkandblink.blogspot.com/2018/04/sodoto-see-one-do-one-teach-one.html">sodoto</a>.<br />
<br />
<b>What's An Embedded System?</b><br />
<br />
It's a computer that's embedded inside another product, like a car, a microwave, a robot, an aircraft, or a giant industrial machine in a factory; or an IoT device like an Amazon Echo, a Sonos speaker, or a SimpliSafe home security system. You think of the thing as the end product, not as a computer. The computer happens to be one of the things inside that makes it work.<br />
<br />
The fascinating thing about embedded systems is that you get to have your hands in the guts of things. The code you write makes a physical object interact with the real world. It's a direct control feedback loop. Working on them is incredibly seductive.<br />
<br />
Embedded systems are a multi-disciplinary endeavor. At a minimum they require a mix of electronics and software knowledge. Depending on the particular application (the end product you're building), they may also require some combination of mechanical, materials science, physics, chemical, biological, medical, or advanced mathematical knowledge.<br />
<br />
<b>Hobbyist vs. Professional Hardware</b><br />
<br />
There's a wide range of hardware available to learn on, at very reasonable prices. Most of the microcontrollers and boards were originally aimed at the professional community, but advances in technology and falling prices have made them accessible to the hobbyist and educational communities.<br />
<br />
Meanwhile, those same advances have enabled hardware aimed directly at the hobbyist and educational communities. Some of that hardware has advanced to the point that it is used in the professional community. So the lines have been blurred.<br />
<br class="Apple-interchange-newline" />
All of the boards covered here have a variety of hardware interfaces and connectors that allow you to connect up other hardware devices. These are the various sensors, indicators, and actuators that allow an embedded system to interact with the real world.<br />
<br />
Two hobbyist/educational platforms are <a href="https://www.arduino.cc/">Arduino</a> and <a href="https://www.raspberrypi.org/">Raspberry Pi</a>. For a beginner, these offer a great way to start out. There's an enormous amount of information available on using them from the hobbyist, educational, and maker communities.<br />
<br />
I've listed a few books on them below in the Primary Resources, and there are a great many more, as well as free videos and websites. These books tend to be written at a more beginner level than books aimed at professionals.<br />
<br />
Arduino is a bare-metal platform, meaning it doesn't run an operating system. An IDE (Integrated Development Environment) is available for free, for writing and running programs on it. You program it with the C and C++ programming languages.<br />
<br />
Many of the low-level details are taken care of for you. That's both the strength and the weakness of Arduino.<br />
<br />
It's a strength because it offers a quick streamlined path to getting something running. That makes it a great platform for exploring new concepts and new hardware.<br />
<br />
It's a weakness because it isolates you too much from the critical low-level details that you need to understand in order to progress beyond the level of beginner.<br />
<br />
Those low-level details are the difference between success in the real world and dangerous mediocrity. Dangerous as in you can actually get people killed, so if you want to do this professionally, you need to understand the responsibility you're taking on.<br />
<br />
My attitude is to take advantage of that streamlined path whenever needed, and use it to boost yourself into the more demanding work. There are always going to be new pieces of hardware to hook up to an Arduino. I'll always start out at the beginner level learning about them.<br />
<br />
In that context, Arduino makes a great prototyping and experimentation platform, <i>without</i> having to worry so much about the low-level details. Then, every bit of knowledge I pick up that way can be carried over to more complex platforms. Meanwhile, Arduino is a perfectly capable platform in its own right.<br />
<br />
Raspberry Pi is a Linux platform, meaning it is a single-board computer running the Linux operating system. In some ways it is similar to Arduino, in that many low-level details are taken care of for you.<br />
<br />
But it is more capable due to more hardware interfaces and the Linux environment. It can operate as a full desktop computer in the palm of your hand. You program it with the Python, C, and C++ programming languages, as well as others. The Linux capability opens up lots of possibilities.<br />
<br />
Many of the same arguments for and against Arduino apply to Raspberry Pi. It also offers a great way to learn Linux and its application to embedded systems. It can be used at the beginner level, but also offers greater range to go beyond that.<br />
<br />
Professional hardware, aimed at commercial and industrial use, offers the classic embedded systems development experience. This is where you need to be able to dig down to the low levels. These platforms run both bare-metal and with operating systems.<br />
<br />
The operating systems tend to be specialized, especially when the application requires true <i>hard real-time</i> behavior, but also include embedded Linux.<br />
<br />
Hard real-time means the system must respond to real-world stimulus on short, fixed deadlines, reliably, every time, or the system fails. For instance, an aircraft flight control system that must respond to a sensor input within 100ms, or the plane crashes. Or a chemical plant process control system that must respond to a sensor within 100ms, or the plant blows up and spews a cloud of toxic chemicals over the neighboring city. Or a rocket nozzle control system that must respond to guidance computer input within 50ms or it goes off course and has to be destroyed, obliterating $800 million worth of satellite and launch vehicle.<br />
<br />
Those are what system failure can mean, showing the responsibilities. There are hard real-time systems with less severe consequences of failure, as well as <i>soft real-time</i> systems with looser deadlines and allowable failure cases (such as a smart speaker losing its input stream after 200ms and failing to play music), but it's important to keep in mind what can be at stake.<br />
<br />
If your goal is to work professionally as an embedded systems developer, you need to be able to work with the professional hardware. But don't hesitate to use the hobbyist hardware to give you a leg up learning new things. The broad range of experience from working with all of them will give you great versatility and adaptability.<br />
<br />
<b>The Primary Resources</b><br />
<br />
The items listed below are all excellent resources that provide the minimum required knowledge for a beginner, progressing up to more advanced levels. If you already have some knowledge and experience, they'll fill in the gaps.<br />
<br />
These are well-written, very practical guides. There's some overlap and duplication among them, but each author has a different perspective and presentation, helping to build a more complete picture.<br />
<br />
They also have links and recommendations for further study. Once you've gone through them, you'll have the background knowledge to tackle more advanced resources.<br />
<br />
The most important thing you can do is to <i>practice the things covered</i>. This material requires hands-on work to really get it down, understand it, and be able to put it to use, especially if you're using it to get a job.<br />
<br />
Whether you practice as you read along or read through a whole book first, invest the time and effort to <i>actually do</i> what it says. That's how you build the skills and experience that will help you in the real world.<br />
<br />
Expect to spend a few days to a few weeks on each of these resources, plus a few months additional. While they're mostly introductory, some assume more background knowledge than others, such as information on binary and hexadecimal numbers. You can find additional information on these topics online by searching on some of the keywords.<br />
<br />
Some of the material can be very dense at first, so don't be afraid to go through it more than once. Also, coming back to something after having gone through other things helps break through difficulties.<br />
<br />
Looking at this list, it may seem like a lot. Indeed, it is an investment in time and money, some items more than others. But if you think of each one as roughly equivalent to half a semester of a college course once you put in the time to practice the material, this adds up to about two years worth of focused college education.<br />
<br />
That's on par with an Associate degree, or half of a Bachelor's degree. And it will leave you with practical skills that you can put to use on a real job.<br />
<br />
These are in a roughly recommended order, but you can go through the software and electronics materials in parallel. You might also find it useful to jump around between different sections of different books based on your knowledge level at the time. Note that inexpensive hardware is listed in the next part of this post, including some of the boards these use.<br />
<br />
If you find some of the material too difficult, never fear, back off to the beginner resources. If you find some too simple, never fear, it gets deep. Eventually, it all starts to coalesce, like a star forming deep in space, until it ignites and burns brightly in your mind.<br />
<br />
The resources:<br />
<blockquote class="tr_bq">
<a href="https://youtu.be/nL34zDTPkcs">You can learn Arduino in 15 minutes.</a>: This is a nice short video that talks about the basics of Arduino microcontroller systems. It helps to start breaking down the terminology and show some of the things involved. That makes it a good introduction to more involved topics. You can also dive down the rabbit hole of endless videos on Arduino, microcontrollers, and electronics from here. This guy's channel alone offers lots of good information.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2z7oJu5">Hacking Electronics: Learning Electronics with Arduino and Raspberry Pi</a>, 2nd Edition, 2017, by Simon Monk. This is a great beginner-level hands-on book that covers just enough of a wide range of hardware and software topics to allow you to get things up and running, starting from zero knowledge.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/31HfOfc">Programming the Raspberry Pi: Getting Started with Python</a>, 2nd Edition, 2016, by Simon Monk. This is a nice practical guide to Python on the Raspberry Pi, with much more detail on programming than his Hacking Electronics above. Meanwhile it has less beginner information on hardware. So the two books complement each other nicely.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2z3GD0R">Programming Arduino: Getting Started with Sketches</a>, 2nd Edition, 2016, by Simon Monk. Similar to his book on Python, but for C on Arduino, also a nice complement to his Hacking Electronics.</blockquote>
<blockquote class="tr_bq">
<a href="https://www.embedded.fm/blog/ese101">Embedded Software Engineering 101</a>: This is a fantastic blog series by Christopher Svec, Senior Principal Software Engineer at iRobot. What I really like about it is that he goes through things at very fine beginner steps, including a spectacular introduction to microcontroller assembly language.</blockquote>
<blockquote class="tr_bq">
<a href="https://www.youtube.com/channel/UCMGXFEew8I6gzjg3tWen4Gw">Modern Embedded Systems Programming</a>: This is a breathtakingly spectacular series of short videos by Miro Samek that take you from the ground up programming embedded systems. They're fast paced, covering lots of material at once, including the C programming language, but he does a great job of breaking things down. He uses an inexpensive microcontroller evaluation kit (see hardware below) and the free size-limited evaluation version of the IAR development software suite. He also has a page of <a href="http://www.state-machine.com/quickstart/">additional resource notes</a>. What I really like about this is that in addition to covering a comprehensive set of information with many subtle details, he shows exactly how the C code translates to data and assembly instructions in microcontroller memory and registers. In contrast to Arduino, this is <i>all</i> the low-level details. You will <i>know</i> how things work under the hood after this course (currently 27 videos). Along the way you'll pick up all kinds of practical design, coding, and debugging skills that would normally take years to acquire. Did I mention this course is freakin' awesome?</blockquote>
<blockquote class="tr_bq">
<a href="http://robogrok.com/">RoboGrok</a>: This is an amazing complete online 2-semester college robotics video course by Angela Sodemann at Arizona State University, available to the public. Start with the <a href="http://robogrok.com/1-1-1_Preliminaries.php">preliminaries page</a>. In addition to some of the basics of embedded systems, it covers kinematics and machine vision, doing hands-on motor and sensor control through a PSoC (Programmable System on a Chip) board. She sells a parts kit, listed below. This is a great example of applied embedded systems.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2DkieZS">C Programming Language</a>, 2nd Edition, 1988, by Brian W. Kernighan and Dennis M. Ritchie: C is the primary language used for embedded systems software, though C++ is starting to become common. This is the seminal book on C, extremely well-written, that influenced a generation of programming style and other programming books. The resources listed above all include some basics of C, and this will complete the coverage.</blockquote>
<blockquote class="tr_bq">
<a href="https://barrgroup.com/Embedded-Systems/Books/Embedded-C-Coding-Standard">Embedded C Coding Standard</a>, 2018 (BARR-C:2018), by Michael Barr: This will put you on the right track to writing clean, readable, maintainable code with fewer bugs. It's a free downloadable PDF, which you can also order as an inexpensive paperback. Coding standards are an important part of being a disciplined developer. When you see ugly, hard to read code, you'll appreciate this.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2Yg0uIE">Programming Embedded Systems: in C and C++</a>, 1999, by Michael Barr: Even though this is now 20 years old, it's a great technical introduction and remains very relevant. Similar in many respects to Samek's video series, it takes a beginner through the process of familiarizing yourself with the processor and its peripherals, and introduces embedded operating system concepts. There is a later edition available, but this one is available used at reasonable prices. </blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2KDuV3U">Programming Arduino Next Steps: Going Further with Sketches</a>, 2nd Edition, 2019, by Simon Monk. This goes deeper into Arduino, covering more advanced programming and interfacing topics. It also includes information on the wide array of third-party non-Arduino boards that you can program with the IDE. This starts to get past the argument that Arduino is just for beginners doing little toy projects.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2MUVgbh">Making Embedded Systems: Design Patterns for Great Software</a>, 2011, by Elecia White. This is an excellent book on the software for small embedded systems that don't use operating systems (known as <i>bare-metal, hard-loop</i>, or <i>superloop</i> systems), introducing a broad range of topics essential to all types of embedded systems. And yes, the topic of design patterns is applicable to embedded systems in C. It's not just for non-embedded systems in object-oriented languages. The details of implementation are just different. </blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2Kz0vj0">Exploring Raspberry Pi: Interfacing to the Real World with Embedded Linux</a>, 2016, by Derek Molloy. This goes into significantly more depth on the Raspberry Pi and embedded Linux. It's quite extensive, so is best approached by dividing it into beginner, intermediate, and advanced topics, based on your knowledge level at the moment. Spread out your reading accordingly. It has great information on hardware as well as software, including many details of the Linux environment. Two particularly fascinating areas are using other microcontrollers such as Arduino as slave real-time controllers, and creating Linux Kernel Modules (LKMs).</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2DvX5fI">Make: Electronics: Learning Through Discovery</a>, 2nd Edition, 2015, by Charles Platt. This is hands down the best book on introductory electronics I've ever seen. Platt focuses primarily on other components rather than microcontrollers, covering what all those other random parts on a board do. See <a href="http://flinkandblink.blogspot.com/2018/05/review-make-electronics-and-makemore.html">Review: Make: Electronics and Make:More Electronics</a> for more information on this and the next book, and <a href="http://flinkandblink.blogspot.com/2018/09/learning-about-electronics-and.html">Learning About Electronics And Microcontrollers</a> for additional resources.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2QR6roQ">Make: More Electronics: Journey Deep Into the World of Logic Chips, Amplifiers, Sensors, and Randomicity</a>, 2014, by Charles Platt. More components that appear in embedded systems.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/30rF4JK">Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems</a>, 2006, David J. Agans. By now you've found many ways to get into trouble with code and hardware. This is a fantastic book for learning how to get out of trouble. It's a simple read that outlines a set of very practical rules that are universally applicable to many situations, then elaborates on them with real-life examples. </blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2MVhb22">Real-Time Concepts for Embedded Systems</a>, 2003, by Qing Li and Caroline Yao. This is an introduction to the general concurrency control mechanisms in embedded operating systems (and larger-scale systems).</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2YgO7vI">Reusable Firmware Development: A Practical Approach to APIs, HALs, and Drivers</a>, 2017, by Jacob Beningo. This covers how to write well-structured low-level device driver code in a way that you can use on multiple projects. Embedded systems are notorious for having non-reusable low-level code, written in a way that's very specific to a specific hardware design, which can often ripple up to higher levels. That means you have to rewrite everything every time for every project. Good Hardware Abstraction Layers (HALs) and Application Programming Interfaces (APIs) provide a disciplined, coherent approach that allows you to reuse code across projects, saving you enormous amounts of time in development and testing. This also helps you become a better designer, because it encourages you to think in a modular way, starting to think in terms of broader architecture in a strategic manner, not just how to deal with the immediate problem at hand in a tactical manner.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2yAP9V8">Embedded Systems Architecture</a>, 2018, by Daniele Lacamera. This is a very up-to-date book that uses the popular ARM Cortex-M microcontroller family as its reference platform. That makes it a great complement to Samek's video series, since the TI TIVA C that he uses is an ARM Cortex-M processor. This also goes into more detail on areas such as the toolchain (including debugging with OpenOCD), bootloading, and memory management. It briefly uses the ST STM32F746 Discovery board as an example.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2yAP9V8">Embedded Systems Fundamentals with Arm Cortex-M based Microcontrollers: A Practical Approach</a>, 2017, by Alexander G. Dean. As the name indicates, this is another detailed book on ARM Cortex-M, intended as a college-level textbook. Among other good practical details, it includes a nice chapter on analog interfacing. It uses the inexpensive NXP FRDM-KL25Z development board for hands-on examples.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2Z6ZJOv">TI Tiva ARM Programming For Embedded Systems: Programming ARM Cortex-M4 TM4C123G with C</a>, 2016, by Muhammad Ali Mazidi, Shujen Chen, Sarmad Naimi, and Sepehr Naimi. This is a detailed book that uses the exact same Tiva C board as Samek's video series. </blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2xs9HPi">Designing Embedded Hardware: Create New Computers and Devices</a>, 2nd Edition, 2005, by John Catsoulis. This covers the hardware side of things, an excellent complement to White's book. It provides the microcontroller information to complement Platt's books.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2I4jUpb">Test Driven Development for Embedded C</a>, 2011, by James Grenning. This is a spectacular book on designing and writing high quality code for embedded systems. See <a href="http://flinkandblink.blogspot.com/2018/08/review-test-driven-development-for.html">Review: Test Driven Development for Embedded C, James W. Grenning</a> for full details. Just as White's book applies concepts from the OO world to embedded systems, Grenning applies Robert C. Martin's "Clean Code" concepts that are typically associated with OO to embedded systems. We'll all be better off for it.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2xJmpZ9">Modern C++ Programming with Test-Driven Development: Code Better, Sleep Better</a>, 2013, by Jeff Langr. This is an equally spectacular book on software development. It reinforces and goes into additional detail on the topics covered in Grenning's book, so the two complement each other well. Even if you don't know C++, it's generally easy enough to follow and the material still applies.</blockquote>
<blockquote class="tr_bq">
<a href="https://youtu.be/BtXyvVy67Qs">Taming Embedded C (part 1)</a>, 2016, by Joe Drzewiecki. This YouTube video is part of the <a href="https://www.youtube.com/playlist?list=PL9B4edd-p2ah3Uj_HhARAW6jOe5IbvvVN">Microchip MASTERs conference</a> series. It covers some of the things that can be risky in embedded code and some methods for avoiding them. This gets into the characteristics that make embedded systems more challenging. I like to watch videos like this at 2X speed initially. Then I go back through sections at normal speed if I need to watch them more carefully.</blockquote>
<div>
<blockquote class="tr_bq">
<a href="https://youtu.be/RnOzACq78dk">Interrupt and Task Scheduling - No RTOS Required</a>, 2016, by Chris Tucker. Another MASTERs video, this covers a critical set of topics for working in embedded systems.</blockquote>
<b>Some Advanced Resources</b><br />
<b><br /></b>
Ready to dig in further and deeper?<br />
<blockquote class="tr_bq">
<a href="https://amzn.to/2ZbsKMI">MC/OS the Real-Time Kernel</a>, 1992, by Jean Labrosse. Labrosse decided to write his own real-time operating system when he had trouble getting support for a commercial one he was using. The rest is history. You can hear some of that history in this podcast interview with him, <a href="https://embedded.fm/episodes/175">"How Hard Could It Be?"</a>. This not only explains how things work under the hood, it gives you the source code.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/33B4SSj">MC/OS III, The Real-Time Kernel for the Texas Instruments Stellaris MCUs</a>, 2010, by Jean Labrosse. This covers the 3rd generation of MC/OS, as well as details on the Stellaris microcontroller covered in Samek's video series. You can also download a free <a href="https://www.micrium.com/books/ucosiii/">PDF version of this</a>, as well as companion software. The MC/OS II and other books are also available there. The value in getting multiple versions is to see how the software evolved over time.</blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/33ANm0r">Software Engineering for Embedded Systems: Methods, Practical Techniques, and Applications</a>, 2nd edition, 2019, edited by Robert Oshana and Mark Kraeling. This is a broad survey of topics by various authors (Labrosse wrote the chapter on real-time operating systems).</blockquote>
<div>
<b>Some Hardware</b></div>
<div>
<b><br /></b></div>
The items listed below include some of the inexpensive boards and evaluation kits used in the resources above. There are a bazillion microcontroller boards out there that are useful for learning how to work on embedded systems. It's worth getting some from different vendors so you can learn their different microcontrollers, different capabilities, and different toolchains.<br />
<br />
That also helps you appreciate the importance of abstracting low-level hardware differences in writing your code. Each vendor provides a range of support tools as part of the package.<br />
<br />
Note that large vendor websites can be a pain, because they want you to create an account with profile, asking questions like your company name (call it "Independent"), what your application is, how many zillion parts you expect to order, when you expect to ship your product, etc. They're setup for industrial use, not hobbyist/individual use. They also may work through distributors like Mouser or Digi-Key for shipping and orders. Just roll with it!<br />
<br />
The hardware:<br />
<blockquote class="tr_bq">
<a href="https://www.sparkfun.com/products/11021">Arduino Uno - R3</a>, $22.95. This is the board used in the Arduino video listed above. There's also a wide array of "shields" available, external devices that connect directly to the board. Exploring these is one of the great educational values that Arduino offers. Remember that because Arduino takes care of many of the details for you, you can be up and learning about new devices faster. Then you can take that knowledge and apply it to other boards. You can also download the Arduino IDE there.</blockquote>
<blockquote class="tr_bq">
<a href="https://www.adafruit.com/product/3775">Raspberry Pi 3 - Model B+</a>, $35. This is an updated version of the boards used in Simon Monk's books above. You will also need the <a href="https://www.adafruit.com/product/1995">5V 2.5A Switching Power Supply with 20AWG MicroUSB Cable</a>, $7.50, and the <a href="https://www.adafruit.com/product/3259">8GB Card With full PIXEL desktop NOOBS - v2.8</a>. You may also want the <a href="https://www.adafruit.com/product/2775">Mini HDMI to HDMI Cable - 5 Feet</a>, $5.95, and the <a href="https://www.adafruit.com/product/2992">Ethernet Hub and USB Hub w/ Micro USB OTG Connector</a>, $14.95. These are sufficient to connect it to a monitor, keyboard, and mouse, and use it as a desktop Linux computer.</blockquote>
</div>
<blockquote class="tr_bq">
<a href="http://www.ti.com/tool/MSP-EXP430F5529LP">Texas Instruments MSP430F5529 USB LaunchPad Evaluation Kit</a>, $12.99 (16-bit microcontroller). An evaluation kit is a complete ready-to-use microcontroller board for general experimentation. Christopher Svec uses this kit in his blog series above, where he also covers using the free downloadable development software. If you buy directly from the TI site, register as an "Independent Designer".</blockquote>
<blockquote class="tr_bq">
Texas Instruments Stellaris LaunchPad Evaluation Kit, was $12.99 (32-bit microcontroller). This is the kit Miro Samek started out with in <a href="https://youtu.be/3V9eqvkMzHA">lesson 0</a> of his video series above. However, as he points out at the start of <a href="https://youtu.be/jmzvued3w3Y">lesson 10</a>, TI no longer sells it, and has replaced it with the Tiva C LaunchPad, which is an acceptable replacement (see next item below). </blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>You might be able to find the Stellaris offered by third-party suppliers. But you have to be careful that you're actually going to get that, and not the Tiva C kit, even though they list it as Stellaris. I now have two Tiva C boards because of that, one that I order directly from TI, and one that was shipped when I specifically ordered a Stellaris from another vendor.</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>Fortunately, that doesn't matter for this course, but it highlights one of the problems you run into with embedded systems, that vendors change their product lines and substitute products (sometimes it's just rebranding existing products with new names, which appears to be what TI did here). That can be confusing and annoying at the least, and panic-inducing at the worst, if something you did in your project absolutely depends on a hardware feature of the original product that's not available on the replacement.</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>One of the design lessons you should learn is to future-proof your projects and try to isolate hardware-specific features so that you can adapt to the newer product when necessary.</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<a href="https://www.ti.com/store/ti/en/p/product/?p=EK-TM4C123GXL">Texas Instruments Tiva C TM4C123G LaunchPad Evaluation Kit</a>, $12.99 (32-bit microcontroller). This is TI's replacement for the Stellaris LaunchPad, that you can use with Miro Samek's video series. Samek addresses the replacement issue at the beginning of <a href="https://youtu.be/jmzvued3w3Y">lesson 10</a>. The good news is that he says the Tiva C is equivalent to the Stellaris (apparently all TI did was rename the product), so it's usable for the course. You'll notice that some parts of the <i>toolchain</i> (the software you use to develop the software for the board, in this case the IAR EWARM size-limited evaluation version) still refer to it as TI Stellaris. </blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>The specific TI device on the board is the TM4C123GH6PM, so when you set the EWARM Project->Options->General Options->Device, you can select TexasInstruments->TM4C->TexasInstruments TM4C123GH6PM, not theLM4F120H5QR that's on the Stellaris board. However, Samek shows that you can continue to use the toolchain configured for Stellaris.</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>That's one of those details that can be maddening when vendors swap parts around on you. Getting it wrong can produce subtle problems, because some things may work fine (you selected a device variant very similar to the one you need), but others won't. Welcome to the world of embedded development! Small details matter. The alphabet soup and sea of numbers in the product names can also drive you batty and be a source of mistakes. PAY CLOSE ATTENTION!</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>A related detail: the file lm4f120h5qr.h that Samek supplies in his projects for working with the Stellaris board's processor also works with the Tiva C board's </i><i>processor. However, there is also a TM4C123GH6PM.h file for the Tiva processor. Both files are in the directory C:\Program Files (x86)\IAR Systems\Embedded Workbench 8.2\arm\inc\TexasInstruments (or whichever version of EWARM you have).</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<i>You can copy them to your project directory, or have the compiler use that directory as an additional include directory by selecting Project->Options->C/C++ Compiler and clicking the ... button next to the "Additional include directories:" box.</i></blockquote>
</blockquote>
<blockquote class="tr_bq">
<a href="https://www.st.com/en/evaluation-tools/32f746gdiscovery.html">STMicroelectronics STM32F746 Discovery Board</a>, $54 (ARM Cortex-M7 microcontroller). This is used briefly in Daniele Lacamera's book above. It's relatively expensive compared to the other evaluation kits here, but includes a 4.3" LCD capacitive touch screen and other hardware elements, making it a much more capable platform, and still an outstanding value.</blockquote>
<blockquote class="tr_bq">
<a href="https://www.nxp.com/design/development-boards/freedom-development-boards/mcu-boards/freedom-development-platform-for-kinetis-kl14-kl15-kl24-kl25-mcus:FRDM-KL25Z">NXP Semiconductor FRDM-KL25Z Freedom Development Board</a>, $15 (ARM Cortex-M0+ microcontroller). This is the board that Alexander Dean uses in his book above.</blockquote>
<blockquote class="tr_bq">
<a href="https://store.digilentinc.com/uc32-arduino-programmable-pic32-microcontroller-board/">uC32: Arduino-programmable PIC32 Microcontroller Board</a>, $34 (Microchip PIC32 32-bit processor). This isn't covered specifically by any of the resources above, but the PIC32 microcontroller is a popular family that offers a different hardware environment. This is programmable using the Arduino IDE, and can also be programmed using Microchip's MPLAB IDE.</blockquote>
<blockquote class="tr_bq">
<a href="https://www.adafruit.com/product/2975">Adafruit Parts Pal</a>, $19.95. This is a small general parts kit for working with the various boards above. It includes LEDs, switches, resistors, capacitors, simple sensors, a small breadboard, and jumper wires for interconnecting things, plus a few other interesting items.</blockquote>
<blockquote class="tr_bq">
<a href="http://robogrok.com/Parts.php">RoboGrok parts kit</a>, $395. This is the parts kit for Angela Sodemann's course above. While you can gather the parts yourself for less, she saves you all the work of doing that, and buying her kit is a nice way of compensating her. </blockquote>
<blockquote class="tr_bq">
<a href="https://amzn.to/2MV2axd">Extech EX330 Autoranging Mini Multimeter</a>, $58. There are also a bazillion multimeters out there. This is one reasonable mid-range model. The multimeter is a vital tool for checking things on boards.</blockquote>
<blockquote class="tr_bq">
One of the following logic analyzers. A logic analyzer is an incredibly valuable tool that allows you to see complex signals in action on a board. They used to cost thousands of dollars and need a cart to roll them around. These are miraculously miniaturized versions that fit in your pocket, at a price that makes them a practical, must-have personal tool. They plug into your USB port and are controlled via free downloadable software you run on your computer:</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<a href="http://saleae-usd.myshopify.com/?rfsn=1807544.21a5e9">Saleae Logic 8 Logic Analyzer, 8 D/A Inputs, 100 MS/s</a>, $199 with awesome "enthusiast/student" discount of $200, using a discount code that <a href="https://blog.saleae.com/saleae-discounts/">you can request</a> and apply to your cart when checking out, thanks guys! This is also covered briefly in Svec's blog series. You can play with the software in simulation mode if you don't have an analyzer yet.</blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<a href="https://store.digilentinc.com/analog-discovery-2-pro-bundle/">Digilent Analog Discovery 2, 100MS/s USB Oscilloscope, Logic Analyzer and Variable Power Supply, Pro Bundle</a>, $299. As amazing as the Saleae is, this one adds oscilloscope, power supply, and signal generator functions, combining a number of pieces of equipment into one tiny package. They also have academic discounts for those who qualify (36% discount on the base unit).</blockquote>
</blockquote>
For a full shopping list to equip a personal electronics lab, see the <b>Shopping List</b> heading at <a href="http://flinkandblink.blogspot.com/2018/04/limor-fried-is-my-new-hero.html">Limor Fried Is My New Hero</a>. That page also has many links to resources on how to use the tools.<br />
<br />
<b>Glossaries</b><br />
<br />
It can be a bit maddening as you learn the vocabulary, with lots of terms, jargon, and acronyms being thrown around as if you completely understood them. As you get through the resources, the accumulation of knowledge starts to clarify things. Sometimes you'll need to go back and reread something once you get a little more information.<br />
<ul>
<li>Barr Group <a href="https://barrgroup.com/Embedded-Systems/Glossary">Embedded Systems Glossary</a>.</li>
<li>Maxim Integrated <a href="https://www.maximintegrated.com/en/glossary/index.mvp">Glossary Of Electrical Engineering Terms</a>.</li>
<li>Byte Craft <a href="http://www.bytecraft.com/Glossary_of_Embedded_Systems_Terminology">Glossary Of Embedded Systems Terminology</a>.</li>
<li>Embedded Artistry <a href="https://embeddedartistry.com/glossary/">Glossary</a>.</li>
</ul>
<b>Other Links</b><br />
<b><br /></b>
These sites have articles and links useful for beginners through advanced developers.<br />
<ul>
<li><a href="http://embedded.fm/">Embedded.fm</a>: podcast and blogs by Elecia White, Christopher White, Andrei Chichak, and Chris Svec.</li>
<li>Embedded Artistry <a href="https://embeddedartistry.com/beginners/">Resources For Beginners</a>.</li>
<li><a href="http://www.state-machine.com/">Quantum Leaps</a>: Miro Samek's website, including his free downloadable book <a href="http://www.state-machine.com/psicc2/">Practical UML Statecharts in C/C++, 2nd Edition: Event-Driven Programming for Embedded Systems</a>, which is just as spectacular as his video series.</li>
<li><a href="http://www.ganssle.com/">The Ganssle Group</a>: Jack Ganssle is a legendary pioneer in the embedded systems field.</li>
<li><a href="https://barrgroup.com/">Barr Group</a>: Barr is another expert in the field.</li>
<li><a href="https://betterembsw.blogspot.com/2017/11/embedded-software-course-notes-on-line.html">Better Embedded System SW</a>: Prof. Philip Koopman, Carnegie-Mellon U., course notes and blog posts. His book <a href="http://www.koopman.us/">Better Embedded System Software</a> is another good resource, you can get it half-off at this site.</li>
<li><a href="https://wingman-sw.com/">Wingman Software</a>: James Grenning's website.</li>
<li><a href="http://langrsoft.com/">Langr Software Solutions</a>: Jeff Langr's website.</li>
</ul>
<div>
<b>Final Thought</b></div>
<div>
<br /></div>
<div>
Our society is becoming more and more dependent on embedded systems and the various backend and support systems they interact with. It's our responsibility as developers to <a href="http://flinkandblink.blogspot.com/2018/03/we-need-to-build-security-in.html">build security in</a> to make sure that we're not creating a house of cards ready to collapse at any moment. Because people's lives can depend on it.<br />
<br />
If you think I'm overstating that, see <a href="https://amzn.to/2xIANB9">Bruce Schneier's new book</a>. We are the ones on the front lines.</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com2tag:blogger.com,1999:blog-6470064853784686094.post-63133531230730859432018-09-22T02:26:00.000-07:002018-09-23T11:10:38.801-07:00Learning About Electronics And Microcontrollers<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWHaFi3EznguLFpVWu_nT-iugxCiWA54Br5GrDI4yFn4UxlwJLwW_cnDTaBpkZYOGAd95ho27PuVB50-UpnX7QxbJysNr-zCz2RSp2PWZoFUZfygOWNDXvvIb64hKdDBB405zahXyPoLVW/s1600/DSCN0217.png" imageanchor="1"><img border="0" data-original-height="945" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWHaFi3EznguLFpVWu_nT-iugxCiWA54Br5GrDI4yFn4UxlwJLwW_cnDTaBpkZYOGAd95ho27PuVB50-UpnX7QxbJysNr-zCz2RSp2PWZoFUZfygOWNDXvvIb64hKdDBB405zahXyPoLVW/s400/DSCN0217.png" width="560" /></a><br />
<i><span style="color: #38761d;">Tutorial books: to read beginning to end. "Make: Electronics" is the best introduction to electronics I've ever seen in any form, a must read!</span></i><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-xp5n5xms-7eZfIQ8tcVNKZ2EKYAH34Djcwpi1iSCAzpPnk5gip09jVvHOIHq2I-4gqjhfeeisQBbrTtritIFwPCduRNdE3IyHPmby5NGqX9Pn0K_PbSM1n0vbBuO-2XRbvM6BaUbE-l8/s1600/DSCN0219.png" imageanchor="1"><img border="0" data-original-height="960" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-xp5n5xms-7eZfIQ8tcVNKZ2EKYAH34Djcwpi1iSCAzpPnk5gip09jVvHOIHq2I-4gqjhfeeisQBbrTtritIFwPCduRNdE3IyHPmby5NGqX9Pn0K_PbSM1n0vbBuO-2XRbvM6BaUbE-l8/s400/DSCN0219.png" width="560" /></a><br />
<span style="color: #38761d;"><i>Reference books: to jump around and get more details as necessary.</i></span><br />
<br />
If you're new to electronics and microcontrollers and want to learn about them, here's my recommended reading list, in order (Amazon links)<br />
<ul>
<li>Tutorial books:</li>
<ul>
<li><a href="https://amzn.to/2k91vwa">Make: Electronics: Learning Through Discovery, 2nd ed, by Charles Platt</a></li>
<li><a href="https://amzn.to/2k966Pc">Make: More Electronics: Journey Deep Into the World of Logic Chips, Amplifiers, Sensors, and Randomicity, by Charles Platt</a></li>
<li><a href="https://amzn.to/2GAem3w">Hacking Electronics: Learning Electronics with Arduino and Raspberry Pi, 2nd ed., by Simon Monk</a></li>
</ul>
<li>Reference books:</li>
<ul>
<li><a href="https://amzn.to/2Lg1dA0">Encyclopedia of Electronic Components Volume 1: Resistors, Capacitors, Inductors, Switches, Encoders, Relays, Transistors</a></li>
<li><a href="https://amzn.to/2rZe0ho">Encyclopedia of Electronic Components Volume 2: LEDs, LCDs, Audio, Thyristors, Digital Logic, and Amplification</a></li>
<li><a href="https://amzn.to/2IBd89O">Encyclopedia of Electronic Components Volume 3: Sensors for Location, Presence, Proximity, Orientation, Oscillation, Force, Load, Human Input, Liquid Light, Heat, Sound, and Electricity</a></li>
<li><a href="https://amzn.to/2IWNuzV">Practical Electronics for Inventors, 4th ed., by Paul Scherz and Simon Monk</a></li>
</ul>
</ul>
<div>
<div>
There's been an evolutionary leap forward in the affordability and ease of use of microcontrollers as a result of inexpensive open-source hardware and free open-source software.<br />
<br />
These have removed many of the traditional roadblocks that made learning to use microcontrollers daunting. Working with them is now within reach of everyone from elementary school children to adults.<br />
<br />
Microcontrollers are then excellent platforms for <a href="http://flinkandblink.blogspot.com/p/learntocode.html">learning to code</a>, because they allow you to interact directly with the physical world. Making the hardware do what your code told it to do is very satisfying.<br />
<br />
<b>Tutorial Books</b><br />
<br /></div>
<div>
</div>
<b><i>First Two Books</i></b><br />
<br />
These cover basic electronics, and make an outstanding starting point. You can read <a href="http://flinkandblink.blogspot.com/2018/05/review-make-electronics-and-makemore.html">my review of them</a> to see why I like them so much (as well as information on components kits for the experiments in the first book).<br />
<br />
They don't focus on microcontrollers. Instead, they focus on the other parts that surround microcontrollers, as well as projects that don't need microcontrollers.<br />
<br />
If you're impatient to get on to the microcontroller information, save <i>Make: More Electronics</i> until later. <b>But you should definitely start with <i>Make: Electronics</i> no matter what.</b><br />
<br /></div>
<div>
<b><i>Third Book</i></b><br />
<br /></div>
<div>
This briefly covers some of the same basics as the first two, then covers some other useful basics. The first two and this one complement each other very well.<br />
<br />
It then gets into working with Arduino and Raspberry Pi microcontrollers. It includes a number of simple projects for working with external modules and sensors.<br />
<br class="Apple-interchange-newline" />
Arduino is programmed in C on "bare metal", i.e. without an operating system, and Raspberry Pi is programmed in Python on embedded Linux, so the two illustrate the variety of programming and runtime environments for microcontrollers.<br />
<div>
<br /></div>
I found this to be a nice gentle introduction to the practicalities of working with microcontrollers and the huge array of third-party modules available. While it only skims the surface of a vast topic, it makes an excellent jumping off point for learning about embedded systems.<br />
<br />
<b>Reference Books</b><br />
<br /></div>
<div>
<b><i>First Three Books</i></b><br />
<b><br /></b>These gather in one place information from a wide array of resources on how to use a wide array of electronic components. They focus on practical concerns rather than theory, and are illustrated with the same excellent color diagrams and photos as <i>Make: Electronics</i>.<br />
<br />
For each component, they contain the following sections: What It Does, How It Works, Variants, Values, How To Use It, and What Can Go Wrong.<br />
<b><br /></b>
<b><i>Fourth Book</i></b><br />
<br /></div>
<div>
This is like multiple smaller books bound into one. It starts with an extensive chapter on theory and related math. The authors point out that much of the math throughout the book is simply to prove the theory, so if you're not interested in that level of detail, you can skip over it.<br />
<br />
The remainder of the book covers a broad range of devices, providing both theory and practical material. It has a chapter on microcontrollers that makes a good follow-up to <i>Hacking Electronics</i>.</div>
<div>
<br />
<b>Using The Books</b><br />
<br /></div>
<div>
The tutorial books are meant to be read from beginning to end as you tinker with their projects. They are easy reading with hands-on experiments, where each chapter builds on previous material.<br />
<br />
The reference books are meant to read here and there, jumping around as you need more details on a specific topic.<br />
<br /></div>
<div>
Even though some of the topics are duplicated between all the books, each author and each book has a different perspective. Each has a different emphasis and presentation.</div>
<div>
<br /></div>
<div>
They complement each other to give a more complete picture because one author may delve deeper into details that another glosses over. You may prefer one author's explanation over another's. No single resource is ever able to give the whole story, so it helps to have multiple perspectives.</div>
<div>
<br /></div>
<div>
These books will give you a good foundation so that you'll be able to understand other books and resources.<br />
<br />
If you're interested in doing embedded systems software development, see <a href="http://flinkandblink.blogspot.com/2018/09/so-you-want-to-be-embedded-systems.html">So You Want To Be An Embedded Systems Developer</a>.<br />
<br />
<b>Electronics Suppliers</b><br />
<br />
There are two outstanding suppliers of discrete electronics, microcontrollers, tools, modules, sensors, and breakout boards that cater to the small-scale needs of hobbyists, students, and experimenters:<br />
<ul>
<li><a href="https://www.adafruit.com/">Adafruit</a></li>
<li><a href="https://www.sparkfun.com/">SparkFun</a></li>
</ul>
You can read my paean to Adafruit at <a href="http://flinkandblink.blogspot.com/2018/04/limor-fried-is-my-new-hero.html">Limor Fried Is My New Hero</a>,which includes the shopping list for setting up my small-scale electronics lab. For a simple example of using this equipment, see <a href="http://flinkandblink.blogspot.com/2018/04/first-use-of-new-tools.html">First Use Of New Tools</a>.<br />
<br />
There are several suppliers for industrial scale, but who also supply at small scale (do you need 10 pieces, or 10 million?):<br />
<ul>
<li><a href="https://www.digikey.com/">Digi-Key Electronics</a></li>
<li><a href="https://www.mouser.com/">Mouser Electronics</a></li>
<li><a href="http://www.newark.com/">Newark element14</a> (for the curious, element 14 is Silicon (chemical symbol Si), a major element in electronics)</li>
<li><a href="https://www.mcmaster.com/">McMaster-Carr</a>: not for electronics, but for all other mechanical parts, supplies, raw materials, and tools.</li>
</ul>
<div>
<i><b>Supplier Learning Resources</b></i></div>
<br />
All these suppliers have extensive online learning resources. However, the industrial suppliers don't have much for the absolute beginner; they're good once you've built up some background knowledge.<br />
<br />
<a href="https://learn.adafruit.com/">Adafruit Learn</a> and <a href="https://learn.sparkfun.com/">SparkFun Learn</a> resources are in both written and video form. You can scan through videos for a quick overview pass by setting the speed in the YouTube window settings (the gear icon) to 2x, then come back and watch at normal speed for a second pass.<br />
<br />
There's a lot of duplication between them (and between these resources and the books above), but it's useful to see how different people approach the same topics. Just like reading books by different authors, they provide additional perspectives to help fill in the gaps.<br />
<br />
Both sites can be a bit overwhelming to dig through, so I've selected a number of beginner resources below, organized by supplier and then type of resource. Many of them have links to additional information.<br />
<br />
These Adafruit videos by <a href="https://learn.adafruit.com/category/collins-lab">Collin Cunningham</a> cover basic electronics lab skills:<br />
<ul>
<li><a href="https://learn.adafruit.com/collins-lab-soldering">Soldering and Desoldering</a>: how to solder components together properly, and how to pull them apart for salvage and rework.</li>
<li><a href="https://youtu.be/QzoPxvIM2qE">Surface Mount Soldering</a>: how to solder surface-mount components.</li>
<li><a href="https://learn.adafruit.com/collins-lab-multimeters">Multimeters</a>: how to use a meter for basic measurements.</li>
<li><a href="https://youtu.be/ThrK2spjrLs">Oscilloscopes</a>: how to use an oscilloscope for advanced measurements and waveforms.</li>
<li><a href="https://youtu.be/J-1phA_vKDg">Hand Tools</a>: the basic hand tools used for assembling and disassembling electronics.</li>
<li><a href="https://youtu.be/9cps7Q_IrX0">Schematics</a>: how to read schematics (no, they're not Greek!).</li>
<li><a href="https://youtu.be/w0c3t0fJhXU">Breadboards and Perfboards</a>: how to combine the parts on a schematic into a functioning circuit.</li>
<li><a href="https://youtu.be/-mHLvtGjum4">Ohm's Law</a>: understanding the relationship between voltage, current, and resistance.</li>
</ul>
He also has these videos on the basics of various components:<br />
<ul>
<li><a href="https://youtu.be/8ftkn7nZSDo">Batteries</a>: the basics of using batteries to supply DC power to projects.</li>
<li><a href="https://youtu.be/RRW4pBCzXBA">Solar Cells</a>: using solar cells to keep the batteries charged.</li>
<li><a href="https://youtu.be/qPt0X5s-1JU">Power Supplies</a>: using an AC power supply to supply DC power to projects.</li>
<li><a href="https://youtu.be/Lf7JJAAZxEU">Pulse Width Modulation</a>: using a PWM converter to change DC input voltage to lower effective DC voltage, or as a simple digital-to-analog converter (DAC).</li>
<li><a href="https://youtu.be/S2AHimvbovI">Switches</a>: understanding the different types of switches for manually controlling projects. </li>
<li><a href="https://youtu.be/-td7YT-Pums">The Transistor</a></li>
<li><a href="https://youtu.be/ZYH9dGl4gUE">The Capacitor</a></li>
<li><a href="https://youtu.be/AqzYsuTRVRc">The Diode</a></li>
<li><a href="https://youtu.be/STDlCdZnIsw">The Inductor</a></li>
<li><a href="https://youtu.be/VPVoY1QROMg">The Resistor</a></li>
<li><a href="https://youtu.be/P3PDLsJQcGI">The LED</a></li>
<li><a href="https://youtu.be/uSRIc-sEgPw">The Integrated Circuit (IC)</a></li>
<li><a href="https://youtu.be/pnf8ojsK6S4">The Arduino</a></li>
</ul>
These are Adafruit written guides by various contributors:<br />
<ul>
<li><a href="https://learn.adafruit.com/adafruit-guide-excellent-soldering">Adafruit Guide To Excellent Soldering</a></li>
<li><a href="https://learn.adafruit.com/multimeters">Multimeters</a></li>
<li><a href="https://learn.adafruit.com/wires-and-connections">Wires and Connections</a></li>
<li><a href="https://learn.adafruit.com/all-about-batteries">All About Batteries</a></li>
<li><a href="https://learn.adafruit.com/power-supplies">Power Supplies</a></li>
<li><a href="https://learn.adafruit.com/all-about-leds">All About LEDs</a></li>
</ul>
These SparkFun videos by <a href="http://shawnhymel.com/">Shawn Hymel</a> cover basic electronics lab skills:<br />
<ul>
<li><a href="https://youtu.be/SLkPtmnglOI">How to Use a Multimeter</a>: how to use a digital multimeter (DMM) to make basic measurements.</li>
<li><a href="https://youtu.be/uraPWaeAgYA">How to Use a Power Supply</a>: how to use a bench power supply unit (PSU) to power a project instead of batteries.</li>
<li><a href="https://youtu.be/u4zyptPLlJI">How to Use an Oscilloscope</a>: how to use an oscilloscope to look into circuit operation.</li>
</ul>
He also has these videos on electronics basics:<br />
<ul>
<li><a href="https://youtu.be/8jB6hDUqN0Y">Ohm's Law</a></li>
<li><a href="https://youtu.be/z8qfhFXjsrw">What is Voltage?</a></li>
<li><a href="https://youtu.be/kYwNj9uauJ4">What is Electric Current?</a></li>
<li><a href="https://youtu.be/-EB7NVA7rI4">What is a Battery?</a></li>
<li><a href="https://youtu.be/8lMO7VAyEkY">Series and Parallel Circuits</a></li>
<li><a href="https://youtu.be/p8JQTLkV5C8">Electric Power</a></li>
</ul>
These are SparkFun written guides by various contributors:<br />
<ul>
<li><a href="https://learn.sparkfun.com/tutorials/voltage-current-resistance-and-ohms-law">Voltage, Current, Resistance, and Ohm's Law</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/resistors">Resistors</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/capacitors">Capacitors</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/diodes">Diodes</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/light-emitting-diodes-leds">Light-Emitting Diodes (LEDs)</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/transistors">Transistors</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/integrated-circuits">Integrated Circuits</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/how-to-read-a-schematic">How to Read a Schematic</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/analog-vs-digital">Analog vs. Digital</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/switch-basics">Switch Basics</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/working-with-wire">Working with Wire</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/connector-basics">Connector Basics</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/how-to-use-a-breadboard">How to Use a Breadboard</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/pcb-basics">PCB Basics</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/how-to-solder-through-hole-soldering">How to Solder: Through-Hole Soldering</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/how-to-use-a-multimeter">How to Use a Multimeter</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/how-to-power-a-project">How to Power a Project</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/battery-technologies">Battery Technologies</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/logic-levels">Logic Levels</a></li>
<li><a href="https://learn.sparkfun.com/tutorials/pulse-width-modulation">Pulse Width Modulation</a></li>
</ul>
<div>
Another great YouTube resource is Dave Jones' <a href="https://www.youtube.com/user/EEVblog">EEVblog</a>.</div>
</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-62913692001462492402018-08-14T14:49:00.000-07:002018-09-14T04:15:43.248-07:00Review: Test Driven Development for Embedded C, James W. Grenning<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeWKbM3rWFu6MrA4CThz6rEAwQ_cW_K76Zw4psols691PO-0gMGfod_hc617-txavtt7N16eYW4m0g3oi7SpGI4QohDOTKPQcklre7jTAuLIRW3gTNkIXn5lsW9lUF0MWruU-CrGVCiw55/s1600/IMG_20180814_180028.jpg" imageanchor="1"><img border="0" data-original-height="1443" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeWKbM3rWFu6MrA4CThz6rEAwQ_cW_K76Zw4psols691PO-0gMGfod_hc617-txavtt7N16eYW4m0g3oi7SpGI4QohDOTKPQcklre7jTAuLIRW3gTNkIXn5lsW9lUF0MWruU-CrGVCiw55/s400/IMG_20180814_180028.jpg" width="510" /></a><br />
<i><b><br /></b></i>
<i><b>The TL;DR:</b></i><br />
<ul>
<li><i><a href="https://amzn.to/2n03PqX">Test Driven Development for Embedded C</a> by James W. Grenning is an outstanding book.</i> </li>
<li><i>The title says C, but if you work in C, C++, C#, Go, Objective-C, Java, Javascript, or anything else, this is worth reading.</i></li>
<li><i>It says embedded, but if you work in embedded systems, front end web apps, mobile apps, desktop apps, backend servers, or anything else, this is worth reading.</i></li>
<li><i>And it's not just TDD, it's all the concepts that go into good design.</i></li>
<li><i>Get it, read it, USE it. You won't regret it.</i>
</li>
</ul>
<b>Background</b><br />
<br />
I first learned about XP (<a href="https://en.wikipedia.org/wiki/Extreme_programming">eXtreme Programming</a>) concepts in 2007, when I was introduced to Kent Beck's <a href="https://amzn.to/2OCNTHQ">Test-Driven Development: By Example</a>. I used TDD (<a href="https://en.wikipedia.org/wiki/Test-driven_development">Test-Driven Development</a>) to develop a major component on a server system. I learned more in 2013, when I read Michael Feathers' <a href="https://amzn.to/2KfOdZm">Working Effectively With Legacy Code</a>. I used that to apply TDD to an existing server codebase.<br />
<br />
Over the past 3 months, I've been on a reading binge, triggered by reading Robert C. Martin's 2017 book <a href="https://amzn.to/2LUlD4P">Clean Architecture: A Craftsman's Guide to Software Structure and Design</a>. I have an hour-long commuter rail ride, so I have lots of time to read and work on my laptop, plus a little lunchtime reading, and I always have a book open at home.<br />
<br />
I read his <a href="https://amzn.to/2OyI3XW">Clean Code: A Handbook of Agile Software Craftsmanship</a>, <a href="https://amzn.to/2vnnK75">The Clean Coder: A Code of Conduct for Professional Programmers</a>, and am currently in the middle of his <a href="https://amzn.to/2n0lIWA">Agile Software Development, Principles, Patterns, and Practices</a>.<br />
<br />
I read Sandro Mancuso's <a href="https://amzn.to/2mZzadd">The Software Craftsman: Professionalism, Pragmatism, Pride</a>, and am in the middle of Mike Cohn's <a href="https://amzn.to/2LUSUga">Agile Estimating and Planning</a>, both from Martin's series.<br />
<br />
I read Andrew Hunt and David Thomas' <a href="https://amzn.to/2OxHs8T">The Pragmatic Programmer: From Journeyman to Master</a>, and am halfway through Pete McBreen's <a href="https://amzn.to/2v1JaXZ">Software Craftsmanship: The New Imperative</a>; Martin Fowler's <a href="https://amzn.to/2voAUjZ">Refactoring: Improving the Design of Existing Code</a> is waiting on the shelf.<br />
<br />
I've encountered bits and pieces of this material over the years, but this was a chance to go back to primary sources, get the full details and parts I've missed out on, and really understand them. I highly recommend it.<br />
<br />
<b>Review</b><br />
<br />
But maybe you don't have time for all that. Maybe you'd like to cut to the chase and see how to apply their principles in practice.<br />
<br />
<a href="https://amzn.to/2n03PqX">Test Driven Development for Embedded C</a> by James W. Grenning does that. It draws from many of those sources and more, showing you real-world examples to put them into practice.<br />
<br />
Grenning is one of the original authors of the <a href="http://agilemanifesto.org/">Agile Manifesto</a> (as are Beck, Fowler, Hunt, Martin, and Thomas). He contributed the chapter "Clean Embedded Architecture" to <i>Clean Architecture</i>, and is the inventor of the Agile planning poker estimation method.<br />
<br />
The book was published in 2011, so is now 7 years old, but it remains as timely as ever. That's especially true as IoT vastly expands the number of embedded systems that we rely on in our daily lives. Effective testing is critically important. For instance, see <a href="https://flinkandblink.blogspot.com/2017/11/testing-is-how-you-avoid-looking-stupid.html">Testing Is How You Avoid Looking Stupid</a>.<br />
<div>
<br /></div>
If you work on embedded systems in C, this is a must read.<br />
<br />
If you work in a different language besides C, or on a different type of system than embedded systems, you may not think that a book on embedded C programming applies to you. But it's broadly applicable and worth reading.<br />
<br />
The book is organized as an introductory chapter, the remaining chapters grouped into 3 parts, and appendices. I see it as three distinct portions, plus appendices: Chapter 1; Parts I and II (chapters 2-10); and Part III (chapters 11-15).<br />
<br class="Apple-interchange-newline" />
Throughout, Grenning addresses the common concerns people have with applying TDD to embedded systems. Embedded systems are a particular challenge, with particular target system constraints, so people might be skeptical.<br />
<br />
This is a very hands-on, how-to book. I've included a number of lists from it, including those Grenning draws from other sources, because they illustrate the practical, pragmatic, disciplined approach. You can use this as a cheat sheet to remember them after you've read the book.<br />
<br />
It might be tempting to think you can get by just with the information I've provided here and skip the book. But I've included it specifically with the hope that you'll realize you <i>must</i> read the book, and that it will be a worthwhile investment.<br />
<br />
<b>First Portion</b><br />
<br />
This is the motivational portion, the appetizer. Grenning introduces TDD, its benefits in general, and the specific benefits for embedded systems.<br />
<br />
He lists Kent Beck's <i>TDD microcycle</i>:<br />
<ol>
<li>Add a small test.</li>
<li>Run all the tests and see the new one fail, maybe not even compile.</li>
<li>Make the small changes needed to pass the test.</li>
<li>Run all the tests and see the new one pass.</li>
<li>Refactor to remove duplication and improve expressiveness.</li>
</ol>
The microcycle is critically important to the technique, so Grenning reminds you of it several times as he works through examples. This is what makes TDD effective, and I know from my own experience is also what makes it fun and extremely satisfying. He has a sidebar titled "Red-Green-Refactor and Pavlov's Programmer", which is very apt. That Pavlovian drive to take the next step in the cycle draws you into the zone and keeps you cranking.<br />
<br />
For embedded systems, in addition to all the benefits that apply to other types of software, the primary benefits include being able to develop tested, working code when the target hardware isn't available; being able to test off-target (i.e. not on the target embedded system), where you have all the benefits of a general-purpose system and none of the constraints of an embedded one, including speed of development turnaround cycle; being able to isolate hardware/software interactions; and decoupling software from hardware dependencies.<br />
<br />
That last point is part of the <b>Big Lesson</b> (see below) from all this. TDD in general, for any type of software, results in testable and tested software. But more than that, it drives development in a way that improves the design significantly.<br />
<br />
That improved design means a much longer and happier life for the software and the systems that use it. They will be able to adapt to changes much more easily. It's not just about getting V1.0 done. It's about getting to V10.0.<br />
<br />
In <i>Software Craftsmanship</i>, Pete McBreen starts off with the origin of the term <i>software engineering</i>. It was coined by a 1967 NATO study group working on "the problems of software." A 1968 NATO conference identified a <i>software crisis</i> and suggested that software engineering was the best way out of that crisis. They were concerned with very large defense systems. McBreen gives the example of the SAFEGUARD Ballistic Missile Defense System, developed from 1969 through 1975.<br />
<br />
He says, "These really large projects are really <i>systems engineering projects</i>. They are combined hardware and software projects in which the hardware is being developed in conjunction with the software. A defining characteristic of this type of project is that initially the software developers have to wait for the hardware, and then by the end of the project the hardware people are waiting for the software. Software engineering grew up out of this paradox."<br />
<br />
McBreen is questioning the value of that style of large-scale software engineering in the development of commercial products, suggesting that a different approach is needed.<br />
<br />
But doesn't that situation sound familiar? Doesn't that sound like the problem embedded systems developers face all the time, that Grenning is addressing? This was a situation where TDD and off-target testing could have significantly alleviated the software crisis.<br />
<br />
Granted, it was more complicated, since they were also developing the very processors and programming languages they would use, while modern systems rely on COTS (Commercial Off The Shelf) processors and languages. But we see that this has been a pervasive problem for some 50 years.<br />
<br />
All types of systems, from embedded to frontend mobile apps to high-scale backend servers, in all those languages, from C to C++, Objective-C, Go, Java, Javascript, etc., can benefit.<br />
<br />
All that code can be removed from its normal production environment and run off-target, off-platform, in a unit test environment that allows you to exercise every code path you want easily and quickly. That includes the obscure dark corners of the code trying to handle unusual error cases that are hard to produce on the target system.<br />
<br />
For some of my own experience testing off-target, see <a href="https://flinkandblink.blogspot.com/2018/07/off-target-testing-and-tdd-for-embedded.html">Off-Target Testing And TDD For Embedded Systems</a>.<br />
<br />
<b>Second Portion</b><br />
<br />
This portion is the meat of the book, applying TDD to real-world embedded development and going through the mechanics with practical examples.<br />
<br />
Following the lead of Martin's book, Grenning makes restrained use of UML diagrams. While some people dislike UML because they associate it with the heavyweight BDUF (Big Design Up Front) software engineering methodologies that McBreen was talking about, this is a very effective use of it that communicates information quickly. Which is the whole point of UML.<br />
<br />
Grenning presents two unit test harnesses, Unity and CppUTest (of which he is one of the authors). All of the material applies just as well to other test harness tools, such as Google Test/Google Mock. It's equally applicable to other languages and their language-specific test harnesses.<br />
<br />
He uses Gerard Meszaros' <i>Four-Phase Test</i> pattern to structure tests:<br />
<ul>
<li><b>Setup:</b> Establish the preconditions to the test.</li>
<li><b>Exercise:</b> Do something to the system.</li>
<li><b>Verify:</b> Check the expected outcome.</li>
<li><b>Cleanup:</b> Return the system under test to its initial state after the test.</li>
</ul>
The rubber meets the road in his five examples of using TDD to develop embedded code:<br />
<ul>
<li>LED driver</li>
<li>Light scheduler for a home automation system</li>
<li>Circular buffer</li>
<li>Flash driver for ST Microelectronics 16 Mb flash memory device</li>
<li>OS isolation layer (aka OSAL, OS Abstration Layer) for Linux/POSIX, Micrium uC/OS-III, and Win32 (this is actually an appendix and only covers thread control, but establishes the pattern)</li>
</ul>
Clearly, these have real hardware dependencies on both the processor I/O interface and the attached devices, as well as the system clock, and real OS dependencies. Those are critical concerns for the embedded developer. The LED driver is very simple behavior, so makes for a gentle introduction. The others are more complex.<br />
<br />
Grenning discusses driver requirements, then shows the initial tests and code. Notice I said tests first. That's an important concept in TDD. You always write the test first, that uses the code in the way that you want the code to work. Then you write the code that satisfies that usage. He emphasizes the <i>save-make-run</i> cycle that you do repeatedly during this process. Then you repeat for the next test and bit of code. That's how you make fast progress.<br />
<br />
The key concept is faking out portions of the system, so that the Code Under Test (CUT) can run as if it was running on the real system. That's critical for making TDD work off-target and off-platform. There are several strategies for doing this. In the case of the LED driver, he uses <i>virtual registers</i> to simulate memory-mapped I/O. This is simply a variable under the control of the test suite.<br />
<br />
He also talks about test-driving the <i>interface</i> before test-driving the <i>internals</i>. That's another critical concept, integral to the whole design process. That's design-for-change. Because things will change. A product with a long, useful life, that represents an ongoing revenue stream for a company, will change over that time to adapt to changes in underlying technologies, user requirements, and usage. TDD means you can make changes without fear of breaking things (because you'll find and fix breakage as a result of performing the microcycle).<br />
<br />
He talks about the strategy of incremental progress and refactoring as you go. This is in the heat of development. Final code does not flow directly from your fingertips. It evolves in incremental steps as you work. Did you ever look at someone's code and marvel at how clean and easy to follow it was, despite the complexity of the job it was achieving? You might think you could never do something that easily. This process results in that kind of code. Like a novelist in the heat of writing a scene, the first draft is never the final product, and the story arc evolves over time.<br />
<br />
This is where he covers several important guidelines for driving the TDD process effectively. He lists Robert Martin's <i>Three Laws of TDD</i>:<br />
<ul>
<li>Do not write production code unless it is to make a failing unit test pass.</li>
<li>Do not write more of a unit test than is sufficient to fail, and build failures are failures.</li>
<li>Do now write more production code than is sufficient to pass the one failing unit test.</li>
</ul>
He describes Kent Beck's snappy acronym <a href="http://wiki.c2.com/?DoTheSimplestThingThatCouldPossiblyWork">DTSTTCPW</a>: Do The Simplest Thing That Could Possibly Work, which initially means just faking it (for instance, hard code a function to return false in order to get the test that uses it to pass). Then keep tests small and focused, and refactor on green (many unit test setups show a failing result in red, and a passing result in green).<br />
<br />
As this evolves, the faked out code turns into real code (the hard coded false is changed to actual code that does something and returns true or false under the appropriate conditions). That builds out a verified test suite as it builds out verified code.<br />
<br />
This leads to the <i>TDD State Machine</i>, which tells you what to do next. The guidelines above and the state machine take you through the mechanics of working in the TDD style. They answer the questions:<br />
<ul>
<li>How should you start?</li>
<li>What should you do next?</li>
<li>How do you know when you're done?</li>
</ul>
Whenever you write some production code, ask yourself, "Do you have a test for that?". If not, stop, go back, and write the missing test.<br />
<br />
He also covers Dave Thomas and Andrew Hunt's <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a> principle: Don't Repeat Yourself. This mantra helps drive the refactoring so that you keep the code lean and clean. I'll throw in additionally the <a href="https://medium.com/mutual-of-omaha-digital-experience-and-design-team/damp-programming-reviving-readability-d84647cc5b2e">DAMP</a> principle: use Descriptive And Meaningful Phrases, a concept the book applies without calling out by name. This favors readable function and variable names that <i>express intent</i> over cryptic abbreviations and syntax. The result is code that reads with a narrative flow.<br />
<br />
Keeping your code DRY and DAMP makes it easy for others to understand and modify (which might be you when you come back to it six months or a year later). This is the same as Beck's microcycle step 5.<br />
<br />
To some degree this all turns TDD into a very mechanistic process. But that's a good thing. It's not a random, ad hoc process where you're constantly questioning yourself about what do to. Instead it's an orderly stepwise process that makes effective progress. You quickly see and appreciate the value.<br />
<br />
It's also very fun and satisfying, because that mechanistic aspect actually drives your creativity. What's the next thing you can add to it? What's the next test, the next bit of functionality? When you finish, you feel like you've accomplished something, and you have the evidence to prove it. It's addicting.<br />
<br />
That leads to Grenning's <i>Embedded TDD Cycle</i>, which starts with TDD on the development system, then advances to the target processor and eval hardware, then the actual target hardware:<br />
<ul>
<li><b>Stage 1:</b> Write a unit test; make it pass; refactor. This is <i>red-green-refactor</i>, the TDD microcycle on the development platform.</li>
<li><b>Stage 2:</b> Compile unit tests for target processor. This is a build check that verifies toolchain compatibility.</li>
<li><b>Stage 3:</b> Run unit tests on the eval hardware or simulator.</li>
<li><b>Stage 4:</b> Run unit tests on target hardware.</li>
<li><b>Stage 5:</b> Run acceptance tests on target hardware. These are automated and manual tests of the integrated system.</li>
</ul>
This sequence gives you confidence in the code under test quickly, then you can address any hardware-dependent issues that start to arise, such as compiler, library, or primitive data type differences. Next you to start exercising the hardware-dependent code.<br />
<br />
Testing separately on eval hardware and actual target hardware helps shake out hardware issues in the actual target, since the eval hardware is presumably known good. One of the challenges in embedded development is always trying to determine if problems are due to the software or due to the hardware, since both are in active development and haven't had much soak time to prove them out.<br />
<br />
For the other TDD examples, Grenning goes through a progression of different collaborator strategies. These are the <i>test doubles</i>, the fakes, that are substitutable for real components. They stand in for those components to break the test dependencies and allow you to simulate and monitor interactions. An important point is that they are much lighter weight than full-scale simulators. Full simulators can themselves require significant development. These fakes have only enough behavior to support the tests (part of the DTSTTCPW mindset).<br />
<br />
He uses these types of doubles:<br />
<ul>
<li>Spies</li>
<li>Stubs</li>
<li>Mocks</li>
<li>Exploding fakes</li>
</ul>
He goes through the following substitution methods, showing how to do them and discussing when they are appropriate:<br />
<ul>
<li>Link-time substitution</li>
<li>Function pointer substitution</li>
<li>Preprocessor substitution</li>
<li>Combined link-time and function pointer substitution</li>
</ul>
These are fully-worked-out examples, although he starts omitting intermediate steps as he progresses in the interest of brevity. All the code is available online.<br />
<br />
<b>Third Portion</b><br />
<br />
This portion completes the meal, complementing the meat in the second portion. It addresses design issues. This is important because design for testability also means design for flexibility and long product life.<br />
<br />
Grenning starts out with Martin's <a href="https://en.wikipedia.org/wiki/SOLID">SOLID principles</a>:<br />
<ul>
<li><b>S:</b> Single Responsibility Principle (SRP)</li>
<li><b>O:</b> Open Closed Principle (OCP)</li>
<li><b>L:</b> Liskov Substitution Principle (LSP)</li>
<li><b>I:</b> Interface Segregation Principle (ISP)</li>
<li><b>D:</b> Dependency Inversion Principle (DIP)</li>
</ul>
He covers both how the previous chapters have incorporated these principles, and how to use them to guide the development process. TDD is closely intertwined with them.<br />
<br />
Don't be put off by the apparent difference between non-object oriented and object-oriented languages. The specific language used is irrelevant. The syntactic mechanics may be different, but the concerns and concepts are all the same. C can be every bit as object-oriented as Java, it just takes a little more developer discipline. That means that all of the concepts of the various principles above apply.<br />
<br />
He uses the SOLID principles in four module design models of increasing complexity, applicable in different embedded system design cases:<br />
<ul>
<li><b>Single-instance module:</b> Encapsulates a module's internal state when only one instance of the module is needed.</li>
<li><b>Multiple-instance module:</b> Encapsulates a module's internal state and lets you create multiple instance of the module's data.</li>
<li><b>Dynamic interface:</b> Allows a module's interface functions to be assigned at runtime.</li>
<li><b>Per-type dynamic interface:</b> Allows multiple types of modules with the same interface to have unique interface functions.</li>
</ul>
You'll probably recognize more than one of these in the systems you work on. You may also recognize object-oriented concepts, and in fact he shows how to implement, use, and test a C++ virtual function table (vtable) in C.<br />
<br />
Part of good design is adapting to change. He covers Martin Fowler's concepts of refactoring, both the <i>code smells</i> that point to things that need to be refactored, and the strategies for doing it with TDD. He describes a disciplined stepwise process that avoids burning bridges.<br />
<br />
This then leads into Michael Feathers' concepts of working on <i>legacy code</i> (which Feathers defines as "code without tests"). He lists Feathers' legacy code change algorithm:<br />
<ol>
<li>Identify change points.</li>
<li>Find test points.</li>
<li>Break dependencies.</li>
<li>Write tests.</li>
<li>Make changes and refactor.</li>
</ol>
He describes how to apply this to embedded systems. Two important types of unit tests during this process are <i>characterization tests</i> that establish how the legacy code behaves, and <i>learning tests</i> that help you learn how to work with third-party code.<br />
<br />
The final chapter covers test patterns and antipatterns. This is useful for helping to build good, effective unit tests that are maintainable over the long term.<br />
<br />
<b>The Big Lesson</b><br />
<br />
For embedded systems, working with the specific hardware is a critical detail. But as Martin points out in <i>Clean Architecture</i>, it's <i>just</i> a detail. For GUI-based mobile, web, and desktop apps, the GUI is <i>just</i> a detail. For either of these, as well as backend servers, the OS (or lack thereof on a bare metal system) is <i>just</i> a detail. The network is <i>just</i> a detail. The database or the filesystem is <i>just</i> a detail. The frameworks or third-party packages are <i>just</i> details.<br />
<br />
All of those details, critical though they may be, can be isolated and segregated from the code that defines what it is your system is about. That code is called the <i>business logic</i>, which sounds a little too dry for me. But's it's the stuff that makes <i>your</i> system something that other people want to use. So it's the stuff that makes your system drive a meaningful business.<br />
<br />
Your business logic interacts with all those details to make a functioning system. TDD allows you to test that logic, in all its happy, twisty, and unhappy paths, separated from its dependencies on the details. The details are represented by test doubles: dummies, stubs, spies, mocks, and fakes.<br />
<br />
This is where the Gang of Four's concept of <i><a href="https://www.artima.com/lejava/articles/designprinciples.html">programming to an interface, not an implementation</a></i>, stated in their book <a href="https://amzn.to/2LYHwA5">Design Patterns</a>, comes into play. You write your business logic to work to an <i>interface</i> to accomplish the detail interactions. In the production environment, you use the real detail components, the <i>real implementations</i>, with a thin adaptation layer that conforms to the interfaces.<br />
<br />
In the test environment you can substitute test doubles that conform to the interfaces; these are <i>alternate implementations</i>. Since you're in control of the test doubles, you can drive any scenario you need to in order to exercise the business logic.<br />
<br />
That isolation also allows you to substitute in other versions of production details, so it's a design strategy, not just a testability strategy. Maybe you want to use some different hardware in your embedded system, or run your app on a different mobile device with a different GUI, or deploy the system on a different OS, or use a different database.<br />
<br />
By defining your details as <a href="https://en.wikipedia.org/wiki/Abstract_data_type">abstract data types</a> or abstract services, you can drop in replacements, with just the effort of implementing the interface layers.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-78351923220971040172018-07-21T13:38:00.000-07:002018-09-20T17:50:42.146-07:00Off-Target Testing And TDD For Embedded SystemsI've recently started reading things by James Grenning (<a href="https://wingman-sw.com/">Wingman-sw.com</a>), one of the authors of the <a href="http://agilemanifesto.org/">Agile Manifesto</a>. My interest in his work relates to Test-Driven Development (TDD) for embedded systems.<br />
<br />
A copy of his book <a href="https://amzn.to/2NzLKLL">Test Driven Development for Embedded C</a> is currently winging its way to me. His site links to a webinar he gave last summer, <a href="https://go.jamasoftware.com/test-driven-development-for-embedded-software-recording.html">Test-Driven Development for Embedded Software</a>, that makes a great introduction to the topic.<br />
<br />
I found one of his answers on Quora interesting. The question was: <a href="https://www.quora.com/Can-I-perform-a-unit-test-when-creating-C-firmware-for-ARM-Cortex-M-MCUs">Can I perform a unit test when creating C firmware for ARM Cortex-M MCUs?</a> The answer, of course, is yes. Specifically, testing can be done off-target (i.e. not on the target embedded system).<br />
<br />
I wrote a long comment on the answer, and decided it might make an interesting blog post. So the remainder of this post reproduces it substantially as it appears there, with some cleanup. He very kindly asked if I would be interested in adding it to his <a href="https://wingman-sw.com/stories">Stories From The Field</a>.<br />
<br />
<b>My Three Stories</b><br />
<br />
I can offer three anecdotes that show why I give a big thumbs up to off-target testing. Off-target testing puts you on target!<br />
<br />
The first case was back in 1995. I had recently transferred to the DEChub group at Digital Equipment Corporation to work on networking equipment.<br />
<br />
They had a problem with their popular DECbridge 90 product, an office- or departmental-scale stackable Ethernet bridge running an in-house custom RTOS on Motorola 68K, all written in C. It would run for weeks or months at a customer site, then suddenly crash. That would throw the LAN into a tizzy as it went through a <a href="https://en.wikipedia.org/wiki/Spanning_Tree_Protocol">Spanning Tree Protocol</a> (STP) reconfiguration event. Then the LAN would do it again once the bridge came back up and advertised its links.<br />
<br />
So it could be very disruptive to the local network and everyone running on it, completely unpredictable. No one had been able to reproduce the problem in the lab.<br />
<br />
I was tasked with finding and fixing it. This platform had very little in the way of crash dump and debug support, and software update was done by burning and replacing an EPROM. It did have an emulator pod, so that was how most debugging was done.<br />
<br />
The problem here was the long run time between failures. That made trying to collect useful information from repeated test runs, either real or via emulator, impractical to the point of impossibility.<br />
<br />
The one clue we knew from the crash log was that it was an OOM condition (Out Of Memory). The question was why. Other than supporting STP, which is a bit of complex behavior, a bridge is a pretty simple device, just L2 forwarding. Packet comes in, look it up in the bridge tables, forward it out the appropriate interfaces.<br />
<br />
The key dynamic structure was the MAC address table. A bridge is a learning device in that it learns which MAC addresses are attached to which links. It builds up the table as it runs, learning the local network topology and participating in STP. So this table was certainly a prime suspect, but it had capacity for thousands of entries, yet it was crashing in LANs with only tens or hundreds of nodes.<br />
<br />
The table used a <a href="https://en.wikipedia.org/wiki/B-tree">B-tree</a> implementation that was public-domain software from some university. We speculated that it was a memory leak in either the B-tree itself, or our interfacing to it.<br />
<br />
So I pulled out the B-tree code and built a test program for it that would go through tens of thousands of adds and deletes in various simple patterns. This is similar to the type of test fixture that Brian Kernighan and Rob Pike later talked about in their book <a href="https://amzn.to/2NwXMW6">The Practice Of Programming</a>.<br />
<br />
I ran this off-target, on a VAX/VMS. VMS supported a simple Ctrl-T key in the terminal interface that would show the process memory consumption, similar to what the Linux ps command shows. The turnaround time on playing with this setup was minutes, build and run, with the full support of an OS to help me chase things down, including good old printf logging and customized dumping of data structures (VMS also had a debugger similar to gdb).<br />
<br />
With this I could see that under some patterns, memory consumption was monotonically increasing. So yeah, a memory leak. Further exploration allowed me to home in on the right area of the code.<br />
<br />
It was right there in the B-tree memory release code: it would free the main B-tree nodes (the main large data element being managed), but <i>not</i> the associated pointer nodes that it used for bookkeeping. So on every B-Tree node release, it would leak 8 bytes of memory.<br />
<br />
This was a case of a very slow memory leak, that only manifested with lots of table changes. In a customer environment, it could take a long time to chew through the memory. In the lab running on-target, it was even slower, since we didn't know what the cause was, so we didn't know how to trigger and reproduce it.<br />
<br />
Off-target, it took less than an hour to find. Code change: add two lines to free the pointer nodes. This was after many man-weeks of effort for all the people involved in trying to reproduce and chase down the problem, plus all the aggravation caused at customer sites. Ten minutes to code, build, and verify the fix.<br />
<br />
The second case was just recently. I implemented an FSM based directly on the book <a href="https://amzn.to/2JNeMVE">Models to Code: With No Mysterious Gaps</a>, by Leon Starr, Andrew Mangogna, and Stephen J. Mellor (backed up by <a href="https://amzn.to/2LCrPeu">Executable UML: A Foundation for Model Driven Architecture</a>, by Stephen J. Mellor and Marc J. Balcer, and <a href="https://amzn.to/2zZDpig">Executable UML: How to Build Class Models</a> by Leon Starr; I highly recommend the trio of books). Thank you, Leon, Andrew, Stephen, and Marc!<br />
<br />
I was also reading Robert C. Martin's <a href="https://amzn.to/2O6dgl2">Clean Architecture: A Craftsman's Guide to Software Structure and Design</a> and <a href="https://amzn.to/2LBZZie">Clean Code: A Handbook of Agile Software Craftsmanship</a> at the time, which heavily influenced the work (and finally motivated me to fully commit to TDD; Grenning contributed the chapter "Clean Embedded Architecture" to <i>Clean Architecture</i>). Mellor and Martin are both additional Agile Manifesto authors.<br />
<br />
A product of all this reading, the FSM was a hand-built and -translated version of the MDA (<a href="https://en.wikipedia.org/wiki/Model-driven_architecture">Model -Driven Architecture</a>) approach, in C on a PIC32 running bare metal superloop.<br />
<br />
The FSM performed polymorphic control of cellular communication modules connected via a UART. The modules use the old Hayes modem "AT" command set to connect to the cell network and perform TCP/IP communications.<br />
<br />
It was polymorphic because it had to support 4 different modules from 2 different vendors, each with their own variation of AT commands and patterns of asynchronous notifications (URC's, Unsolicited Result Codes).<br />
<br />
If you think LANs and WANs are squirrelly, just wait till you try cellular networks. I could hardly get two test runs to repeat the same path through the FSM. Worse, there were corner cases that the network would only trigger occasionally.<br />
<br />
It was horribly non-deterministic. How can I be sure I've built the right thing when I can't stimulate the system to produce the behavior I want to exercise?<br />
<br />
The solution: build a nearly-full-coverage test suite to run off-target. I built a trivial simulator with fake system clock and UART ISR that I ran on an Ubuntu VM on my Mac. That gave me full support for logging and gdb.<br />
<br />
This wasn't quite TDD, but it was one step away: it was Test-After Development, and instead of Google Test/Mock or some other framework, I built my own ad-hoc fakes and EXPECT handling.<br />
<br />
With this I was able to create scenarios to test every path in the FSM, for all the module variants. Since I had control of the fake clock and ISR, I could drive all kinds of timing conditions and module responses. It did help that the superloop environment was pure RTC (Run To Completion, which coincidentally is required for Executable UML state machines), rather than preemptive multitasking/multithreading. But I could have faked that as well if necessary.<br />
<br />
I was able to fix several bugs that would have been hell to reproduce and debug on-target. In all cases, just as with the B-tree, the code changes were trivial. The time-consuming and hard part is always the debug phase to figure out what's going wrong and what needs to be changed. Doing the actual changes is usually simple.<br />
<br />
That debug phase is where non-TDD approaches run into trouble, especially when they have to be done on-target. It can consume unbounded amounts of development time. The time required to do TDD is far shorter, and for a significant number of problems can either completely eliminate the debug phase, or narrowly direct it to the specific code path of a failing test.<br />
<br />
The third case was this past week, when I did my first true TDD off-target thing for some embedded code. The platform is embedded Linux on ARM, so full OS, with cross-compiled C++.<br />
<br />
I built the code in my Ubuntu VM and used Google Test/Mock, mocking out the file I/O (standard input stream and file output) and system clock. The code wasn’t particularly complex, but it did have a corner case dealing with a full buffer that represented the greatest bug risk.<br />
<br />
I used very thin InputStreamInterface, OutputFileInterface, and ClockInterface classes as the OSAL (Operating System Abstraction Layer) to provide testability (thank you, Robert and James!).<br />
<br />
It was gloriously wonderful and liberating to build it TDD style, red-green-refactor, and I knew I had all the paths in the code covered, including the unusual ones. That instills great confidence in what I did. No more worrying that I got everything right. I was able to demonstrate that I did.<br />
<br />
Did it take a little extra time? Sure, mostly because I’m still on the learning curve getting into the TDD flow. But if I hadn’t used TDD and this code had produced failures, it would take me longer after the fact to chase down the bug. Plus I was able to avoid the impact on all the other people in the organization affected by the development turnaround cycle.<br />
<br />
And just today, I added more behavior to that component using the TDD method. I was able to work fully confident that I wasn't breaking any of the stuff I had already done, and just as confident in the new code.<br />
<br />
So I'm definitely a believer in off-target testing, and from now on I'll be doing it TDD.<br />
<br />
Another benefit of this off-target TDD model? Working out of that Ubuntu VM on my Mac, I'm totally portable. I can work anywhere, at a coffee shop, on the train commuting, at the airport, on a plane, at home. I can be just as productive as if I had my full embedded development environment in front of me. Then once I'm back at my full environment, I have tested, running code ready to go.<br />
<br />
For reference, these are the books that taught me TDD while in different jobs, both highly recommended:<br />
<ul>
<li><a href="https://amzn.to/2NBxLoE">Test Driven Development: By Example</a>, by Kent Beck (yet another Agile Manifesto author). I was introduced to the book and TDD in general by new coworker <a href="https://www.linkedin.com/in/steve-vinoski-b7852/">Steve Vinoski</a> in 2007, whose cred in my eyes went way up when I noticed his name in the acknowledgements of James O. Coplien's <a href="https://amzn.to/2O7kVQ1">Advanced C++ Programming Styles and Idioms</a>.</li>
<li><a href="https://amzn.to/2LghuJ1">Working Effectively With Legacy Code</a>, by Michael C. Feathers. Amazon tells me I bought this in 2013. At the time I used it to start adding unit test coverage to our codebase at work. What makes this book particularly useful is the fact that nearly all software development requires working with legacy code to some degree, even on brand new projects. It also helps you avoid creating a legacy of code that future developers will curse. </li>
</ul>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-80858108924327267242018-05-22T18:28:00.000-07:002018-05-22T19:06:29.022-07:00Review: Make: Electronics and Make:More Electronics<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6s5PHwUHQdNkZkoogLeZyegYJxtgVw6xGDmkeFDs0V0DPaU2HEZE4NiG1Jij80Gqr4CsILXnDWDHBrxwXD4RvO69ezydbn8wKZQlc3uOY41STPpGl84SoR0DB3sSvoejiY8KwhtYWM9sq/s1600/DSCN0205.png" imageanchor="1"><img border="0" data-original-height="882" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6s5PHwUHQdNkZkoogLeZyegYJxtgVw6xGDmkeFDs0V0DPaU2HEZE4NiG1Jij80Gqr4CsILXnDWDHBrxwXD4RvO69ezydbn8wKZQlc3uOY41STPpGl84SoR0DB3sSvoejiY8KwhtYWM9sq/s400/DSCN0205.png" width="560" /></a><br />
<span style="color: #38761d;"><i>Charles Platt's Make: Electronics and Make: More Electronics.</i></span><br />
<br />
Amazon links:<br />
<ul>
<li><a href="https://amzn.to/2rWgpdI">Make: Electronics: Learning Through Discovery 2nd Edition</a></li>
<li><a href="https://amzn.to/2wWW2C7">Make: More Electronics: Journey Deep Into the World of Logic Chips, Amplifiers, Sensors, and Randomicity</a></li>
</ul>
If you're interested in learning electronics, I highly recommend these two books by Charles Platt. They are hands down the best books I have ever seen on the subject, spectacular resources for the beginner.<br />
<br />
Rather than focusing on theory, Platt jumps right into hands-on experimentation. The books are organized as a series of experiments and circuit-building projects that build knowledge incrementally.<br />
<br />
He calls it "learning by discovery". He then follows up with just enough theory to explain what's going on. This is an extremely effective method that avoids getting bogged down.<br />
<br />
I first learned electric theory in high school science. I learned Ohm's Law and basic circuit layout, including the equations for computing series and parallel resistance. I learned further details in college physics.<br />
<br />
But these didn't really cover the practical details of electronics. They didn't address detailed circuit design, combining components into useful projects.<br />
<br />
I started to learn some of those details from the books of <a href="https://amzn.to/2JwzBFQ">Forrest M. Mims III</a> and George Young's book <a href="https://amzn.to/2KnzrRG">Digital Electronics: A Hands-on Learning Approach</a>. The latter introduced me to integrated circuit chips (IC's) and digital logic, as well as breadboard experimentation.<br />
<br />
That constituted the bulk of my electronics knowledge for the past 35 years. But there was still a lot missing, particularly an <i>intuitive</i> understanding of electronics and all those other random parts surrounding the IC's.<br />
<br />
Then I found Platt's books. Platt has a real gift for explaining things at an intuitive level in just a few concise paragraphs and clear diagrams.<br />
<br />
He delves deep into the practical details. No detail is too small. For a beginner trying to learn from a book, this is critical.<br />
<br />
He explains the most basic things so that you know how to wire up a breadboard and check things with a meter. He shows how things work internally, both mechanically and electrically, so a component isn't just an opaque black box.<br />
<br />
The color diagrams are outstanding. One thing I really like is the way he steps from a circuit schematic diagram, to a breadboard-friendly schematic, to a breadboard component and wiring diagram, to a component value diagram, to an under-the-covers diagram illustrating all the electrical paths in the wiring and the breadboard connections hidden beneath.<br />
<br />
He takes several projects from breadboard to final soldered board built into a simple enclosure. This shows how you can turn your experiment into a completed useful or fun gadget.<br />
<br />
The diagrams and project builds really show where other books fall short. Most books show a schematic, and maybe a completed breadboard or a completed wired-up project. But they don't show the stepwise process to get from the start to the end.<br />
<br />
That process is not always obvious and is full of opportunities for mistakes, so having it laid out in detail is a huge benefit. He also covers some of the things that can go wrong and how to diagnose and fix them.<br />
<br />
The clarity of the diagrams and overall layout make the books very readable. This is another improvement over other books.<br />
<br />
If you send a registration email to Platt, he'll add you to his email list for a bonus project and book updates.<br />
<br />
<b>Microcontrollers</b><br />
<br />
Platt doesn't emphasize microcontrollers in these books. In an age where Arduinos, Raspberry Pi's, and other microcontrollers allow you to solve nearly any problem with a little embedded software, he mostly shows you what you can do without them. One experiment does cover using an Arduino.<br />
<br />
He also discusses the pros and cons of replacing the discrete components with microcontrollers in several experiments. This is actually very useful from an engineering standpoint, giving you choices in how to implement things.<br />
<br />
That also ensures that if you do incorporate microcontrollers into your projects, you understand how to integrate them with external components. There are many books on getting started with microcontrollers, but they tend to gloss over the details of those other components, assuming you already understand them. Which you will if you read these books!<br />
<br />
<b>Component Kits</b><br />
<br />
Component kits for all the experiments in the first book are available online. You can certainly gather parts on your own, but the kits offer you one-stop shopping of the correct parts.<br />
<br />
I used ProTechTrader, the supplier he recommends in his email, and I recommend them highly based on my experience. Make sure you get the kits for the 2nd edition.<br />
<br />
The kits are available for the best price <a href="https://www.protechtrader.com/Make-electronics-component-pack-1-2nd-edition">directly from the ProTechTrader website</a>. They offer 3 kits, covering experiments 1-11, 12-24, and 25-34. Each is available in regular and deluxe versions. The deluxe versions add things like a digital multimeter, soldering iron, 9V power supply, and upgraded magnet.<br />
<br />
I purchased the regular version of each kit, since I already had most of the deluxe items, with free economy 3-10-day shipping. The kits arrived in 3 days.<br />
<br />
While you pay a little extra per part for convenience vs. buying everything separately, it was well worth it. The parts are extremely well organized. They're bagged and labeled by value, stored in compartmentalized containers, and identified by experiment.<br />
<br />
Don't underestimate the value of the labor that went into that. Platt dedicates several pages in his book to organization of workspace and parts. That's key to efficient work. Rummaging around in a box of loose parts will make you tear your hair out.<br />
<br />
<b>Additional Books</b><br />
<b><br /></b>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha3eoSzHf8RkQ-FrVh2yheljqTlaFr6fV8utVyLHCzuIBa_ly2wah93v573dHhsQWMVwenXnB-QN13hN2QhBLDaOLEoNK4XmLe9SQjgHOrWMHUXg22G3nt7rbws1fRebN6uuRvJskeHgwx/s1600/DSCN0210.png" imageanchor="1"><img border="0" data-original-height="774" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha3eoSzHf8RkQ-FrVh2yheljqTlaFr6fV8utVyLHCzuIBa_ly2wah93v573dHhsQWMVwenXnB-QN13hN2QhBLDaOLEoNK4XmLe9SQjgHOrWMHUXg22G3nt7rbws1fRebN6uuRvJskeHgwx/s400/DSCN0210.png" width="560" /></a>
<br />
<span style="color: #38761d;"><i>Platt's Make: Tools and 3-volume Encyclopedia of Electronic Components.</i></span><br />
<br />
Platt has several additional books that make useful companions to this pair:<br />
<ul>
<li><a href="https://amzn.to/2LfaFnq">Make: Tools: How They Work and How to Use Them</a></li>
<li><a href="https://amzn.to/2Lg1dA0">Encyclopedia of Electronic Components Volume 1: Resistors, Capacitors, Inductors, Switches, Encoders, Relays, Transistors</a></li>
<li><a href="https://amzn.to/2rZe0ho">Encyclopedia of Electronic Components Volume 2: LEDs, LCDs, Audio, Thyristors, Digital Logic, and Amplification</a></li>
<li><a href="https://amzn.to/2IBd89O">Encyclopedia of Electronic Components Volume 3: Sensors for Location, Presence, Proximity, Orientation, Oscillation, Force, Load, Human Input, Liquid Light, Heat, Sound, and Electricity</a></li>
</ul>
The first book (no, it's not about how to make tools, it's just part of the Make: series) covers the basic hand and small power tools you'll find at home centers and hardware stores, showing how to use them to build small projects. It feature Platt's usual deep attention to practical details.<br />
<br />
The book contains a number of simple projects in wood and plastic. The methods for working with plastic are particularly noteworthy, because while there are many books about woodworking, there aren't many about plastic.<br />
<br />
These are the skills you need to build different styles of enclosures and stands for your electronics projects, and can also be applied to other mechanical aspects such as robotics.<br />
<br />
The remaining books are a 3-volume encyclopedia of electronic components. This is all the information that he didn't have room for in the other books, plus more. Where those books were written as tutorials, this is a reference set.<br />
<br />
He's compiled a vast trove of information culled from manufacturer data sheets, tutorials, reference books, and other sources to create a centralized, practical one-stop resource.<br />
<br />
Need to know pinouts, sample circuits, voltage levels, alternative packages? You can find them here, in Platt's signature level of detail.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-16214308034857929892018-04-29T09:16:00.004-07:002021-07-14T07:52:36.780-07:00How To Ace CalculusThis is the method I used to ace 3 semesters of calculus. Also linear algebra, differential equations, and a semester of physics. It should work for any math or science class, at earlier or later level.<br />
<br />
When I say ace, I mean getting a grade of 100 on most homework, quizzes, tests, midterms, and finals. In all cases, the top grade in the class. Yes, I was the one breaking the curve.<br />
<br />
Sounds arrogant? Well, I didn't start off in that lofty position. So before I give you the recipe for success, let me give you the recipe for failure.<br />
<br />
<b>The Recipe For Failure</b><br />
<br />
In 1978, I entered Northwestern University, in Evanston, IL, as a mechanical engineering major. My dream was to work for NASA.<br />
<br />
This was a major in the Technical Institute, requiring calculus, physics, and mechanics (statics and dynamics).<br />
<br />
The recipe:<br />
<ol>
<li>Show up for all classes and pay attention.</li>
<li>Complete all reading assignments on time.</li>
<li>Complete all assigned homework on time.</li>
<li>Study for tests, reviewing homework.</li>
</ol>
<div>
This was the recipe that had gotten me through high school, where I was usually able to do most of the homework in the last 5 minutes of class allocated for that purpose. Then just a few minutes in "study hall" period or at home to complete it, and another 10 or 15 to read the next section.</div>
<div>
<br /></div>
<div>
Sounds like a pretty good plan, right? Sounds like a good student, right?</div>
<div>
<br /></div>
<div>
The problem was that the material in college was more difficult and faster paced. Doing <i>just</i> the assigned problems was barely enough to keep your head above water, sometimes not even enough for that. It didn't give you enough practice thinking through and performing the work.</div>
<div>
<br /></div>
<div>
Back in algebra, problems were simple, they had one procedure to follow to the solution. Calculus wasn't like that. There were multiple procedures depending on the style of the equation. A good portion of the battle was classifying the equation to determine what approach to bring to it.</div>
<div>
<br /></div>
<div>
Physics was similar. Both topics required a more analytical approach. That meant building up a problem database in your mind so you could pick the approach. That meant <i>experience doing lots of problems</i>.</div>
<div>
<br /></div>
<div>
The result of following that recipe? C's, D's, and finally, an F in physics. Where I had prided myself on my math and science abilities, my favorite subjects, I had failed. Distraught, I dropped out of Northwestern.</div>
<div>
<br /></div>
<div>
<b>The Recipe For Success</b></div>
<div>
<br /></div>
<div>
About 5 years later, I started part-time classes at Richland College, part of the Dallas County Community College District.</div>
<div>
<br /></div>
<div>
Oh sure, you may say, community college. That's easy, it's not a real college.</div>
<div>
<br /></div>
<div>
Negative. Richland used exactly the same textbooks as Northwestern, just the next editions. So it was exactly the same material. And I had Ralph Esparza as instructor for calculus I and III. Ralph was feared among students as a tough math teacher, who cares if it's community college or Ivy League.</div>
<div>
<br /></div>
<div>
I was determined to repeat all those classes in my favorite subjects, and do well in them. Somewhere in hindsight, I had realized the need to do more than the minimum.</div>
<br />
The recipe:<br />
<div>
<ol>
<li>Show up for all classes and pay attention.</li>
<li>Complete all reading assignments on time.</li>
<li>Complete all assigned homework on time.</li>
<li>Complete all remaining odd-numbered problems in the section and check against answers in the back.</li>
<li>Complete all remaining even-numbered problems in the section.</li>
<li>Study for tests (regular tests, midterms, and finals), redoing all problems that the tests cover.</li>
</ol>
<div></div><blockquote><div><i>This is a great way to work in study groups, too. When someone in the group has difficulty, everyone can contribute to helping them understand it. Or maybe the one person in the group who understands it is able to help everyone else.</i></div><div></div></blockquote><div>This boils down to doing <i><b>every</b> problem in <b>every</b> section of the textbook at least twice, <b>more like three or four times</b></i>.</div><div><br /></div></div>
<div>
You may say, that's a lot of work. Yes, it is.<br />
<br />
In Nike ads, athletes show how tough they are. <a href="http://www.complex.com/sneakers/2015/08/nike-just-do-it-history">Just do it</a>. Be tough.</div>
<div>
<br /></div>
<div>
The result? Redemption.</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-76046133892640527152018-04-21T11:06:00.003-07:002021-02-18T04:56:13.344-08:00Sodoto: See One, Do One, Teach OneHere's a useful strategy on this learning path: <b><u>s</u></b>ee <b><u>o</u></b>ne, <u><b>d</b></u>o <b><u>o</u></b>ne, <b><u>t</u></b>each <b><u>o</u></b>ne. <i>Sodoto</i>.<br />
<br />
Sodoto is a learning method and a teaching method rolled into one. It's the cycle of knowledge.<br />
<br />
I'm familiar with it from medicine. My wife is a surgical nurse, and this is the <a href="http://www.ncbi.nim.nih.gov/">traditional method of teaching in surgery</a>. Obviously, safety concerns mean that you don't just watch a brain surgeon at work and then go try it yourself.<br />
<br />
But this forms a useful pattern of mentoring and learning and passing knowledge along. It applies to any kind of knowledge- or skill-based activity.<br />
<br />
It works with a single student at a time, or a whole group. You don't have to be a formal teacher.<br />
<b><br /></b>
<i><b>See one:</b> watch someone do a procedure.</i><br />
<div>
<b><i><br /></i></b></div>
<div>
<i><b>Do one:</b> do what you saw.</i><br />
<div>
<b><i><br /></i></b></div>
<div>
<i><b>Teach one:</b> show it to someone else.</i></div>
<div>
<br /></div>
<div>
Once you learn a procedure, you're primed to teach it. That's how knowledge spreads.<br />
<br />
Here's the real kicker: the teaching step is actually a powerful learning step for you as the teacher. It locks the knowledge into your brain.</div>
<div>
<br />
You have to have sorted out what you're talking about in order to teach it. You can't just vaguely know it and wave your hands in the air glossing over details. Your students will be annoyed and you'll feel stupid.<br />
<br />
The process of getting ready to teach and then doing the teaching forces you to organize your thoughts and chase down details, because you don't want to look stupid, and you want to be prepared for questions.<br />
<br />
That motivates you to dig deeper. As a result, you end up learning more yourself.<br />
<br />
There are two keys to making this work: background knowledge, and the experience of doing it.<br />
<br />
<b>Background Knowledge</b><br />
<br />
Background knowledge applies at each stage of see, do, and teach. Note that "see" can mean live and in person, or on video.<br />
<br />
Whatever the subject, medicine, coding, climbing, sailing, scuba diving, physical fitness, martial arts, building anything from woodworking to electronics, any knowledge you have before seeing the procedure will help you understand it.</div><div><br /></div><div>You can bet that surgeon learning how to do brain surgery brought a huge amount of background knowledge.<br />
<br />
Some things take minimal background, just the random skills and knowledge you already have from life. But more difficult subjects benefit from whatever time you can invest beforehand. Videos, books, blogs, and articles, in print and online, are all good resources, as well as online forums.<br />
<br />
That establishes the background knowledge you'll bring to <i>seeing</i> the procedure.<br />
<br />
Once you've seen the procedure, as you prepare to do it yourself, it's useful to go back to your resources. Now that you know better what to look for, you can get more details. You can reinforce what you saw.<br />
<br />
That expands the background knowledge you'll bring to <i>doing</i> the procedure.<br />
<br />
Once you've done the procedure, as you prepare to teach it, go back to your resources again. As a result of doing, there will be details you want to fill in, and you may understand the material better. You may have run into some things that you wished you knew more about. You may anticipate additional questions from your students.<br />
<br />
That further expands the background knowledge you'll bring to <i>teaching</i> the procedure.<br />
<br />
<b>Experience Of Doing</b><br />
<br />
The experience of <i>doing</i> the procedure is critical. That's where you have the opportunity to work through mistakes and see what works and doesn't work for you. That's where you start to lock it into your brain.<br />
<br />
Don't be afraid to make mistakes! Mistakes are great learning opportunities. As long as there's no injury and no damage, there's no harm done. And a little blood on the deck isn't an injury.<br />
<br />
This is also where you can work out your own changes to the procedure. Just because you saw it done one way doesn't mean that's the only way to do it. That was one way. You can use it as your starting point, and add your own tweaks.<br />
<br />
Or maybe you'll realize what you saw really was a good way and you shouldn't mess with it.<br />
<br />
You might need to do the procedure more than once before teaching it. Some procedures take practice before you feel confident teaching them to someone else.<br />
<br />
The experience of <i>teaching</i> the procedure will be different from the experience of doing it for yourself. Your students may have questions or difficulties that force you to think about things in different ways.<br />
<br />
Plus there's the pressure of performing for an audience. But as you gain experience teaching, that will get easier. It's just a different kind of doing.<br />
<br />
The experience of teaching is where you finish locking it into your brain.<br />
<br />
<b>Teamwork</b><br />
<br />
Sodoto is a great method for dividing up a project. Whether at work, at school, or with your friends, you can divide up the project and have each person take on a part.<br />
<br />
They go off and see how it's done, do it themselves until they feel ready, then bring it back to the team to teach everyone else.<br />
<br />
What if you can't agree how to divide it up because multiple people want to do the same thing? Fine! Let them!<br />
<br />
Each person will have their own take on the experience and teach it slightly differently. That helps explore all the possibilities in the procedure.</div>
</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com1tag:blogger.com,1999:blog-6470064853784686094.post-78416693251227631562018-04-17T08:58:00.000-07:002018-04-20T06:13:05.882-07:00More C+-In <a href="http://flinkandblink.blogspot.com/2018/03/the-case-for-c.html">The Case For C+-</a>, I talked about writing quick tools in a simple C style, but taking advantage of the C++ standard library, primarily the dynamic data structures. It ends up being C++ without any (or just a few) user-defined classes, so is something of a lightweight object-oriented approach (yes, yes, I'm sure OO purists are barfing at the thought). The main benefit is fast coding.<br />
<br />
There I showed as an example the <i>msgresolve</i> tool, which I used to resolve messages logged by an IOT device (the client) and its server. This is a lot of string processing and cross-indexing, with logs containing potentially thousands or tens of thousands of messages.<br />
<br />
Shortly after I had completed <i>msgresolve</i>, I needed to have a tool to help me sift through large text files of server logs, logging the TCP connections made by clients and their subsequent activity. I was chasing down a problem where some of the connections were shutting down sooner than expected.<br />
<br />
I wasn't sure what was causing the early shutdowns, and wasn't even sure initially which connections had experienced it, so I wanted to be able to gather all the lines for a given connection and list them out for tracing through, for each connection.<br />
<br />
That would help me identify the ones that were live at the end of the log sample vs. the ones that had ended early. The log entries for hundreds of connections were all intermixed.<br />
<br />
Armed with the methods I had used in <i>msgresolve.cpp</i>, conceptual design was easy. I wanted an ordered list of connections, and associated with each one, the sequential list of log entries associated with the connection.<br />
<br />
There were also connections with some internal addresses I wanted to ignore. I could have done this filtering with grep, but it was easy enough to build the capability into the program so that it could stand alone. That also helped me explore some additional string processing functions.<br />
<br />
Given that architecture, the data structure I needed was a <i>std::map</i> that mapped a string (the connection identification) to a <i>std::list</i> of strings (the log lines for the connection).<br />
<br />
I had the program working in less than an hour. Then I spent at least another hour screwing around with the timestamps in the log entries, figuring out how to process them and deciding what to do with them. Then a little more time on refactoring and cleanup.<br />
<br />
Throughout, I used a sample log file that had entries for several connections, including addresses I wanted to skip. I used that as a simple unit test to exercise the code paths.<br />
<br />
The resulting code provided the impetus for a simple generalized string processing module, which I'll cover in another post. But you can see some clear patterns emerging in this code.<br />
<br />
Doing quick tools like this is fun and very satisfying. It makes your serotonin flow. You have a problem you need to deal with, so you sit down and spew a bunch of code in a short time, refine it, and use the results.<br />
<br />
This is actually quite different from long-term product development. That kind of work has its intense coding phases, but once the initial version of the product is out, a lot of the work is much smaller surgical changes.<br />
<br />
Even fitting a major new feature in often involves many small bits of code scattered throughout the larger code base, integrating the tendrils. Getting that to work has a different kind of satisfaction.<br />
<br />
<b>Design Choices</b><br />
<br />
These tools also give you a chance to think about different approaches. You can balance the variables of memory consumption, CPU consumption, I/O consumption, time, and code complexity (that is, ease of writing and maintaining the code, and compiled code space consumption, not algorithmic complexity) for a given situation.<br />
<br />
For instance, the log files I was dealing with had over a million lines of data, some 200MB worth covering hundreds of connections.<br />
<br />
That meant I had several choices:<br />
<ol>
<li>I could load all the data into memory and then print it out in an orderly manner. This is a single-pass solution, that consumes large amounts of memory.</li>
<li>I could scan the file once, identifying all the individual connections, then for each connection, scan the file from beginning to end to read and print their lines. This a multi-pass solution that requires little memory but significant file I/O.</li>
<li>I could scan the file once, and for each identified connection, track the file position of the first and last line, then for each connection, just scan that range of the file. This is a multi-pass solution that reduces the total file I/O for a negligible increase in memory.</li>
<li>I could do the same thing, but instead of tracking just the first and last line file positions, build a list of the file position and length of each line, then on each pass, just skip directly to the locations of the lines. This is still multi-pass, but significantly reduces the total file I/O because it only visits each file position twice, requiring a bit more complexity and a bit more memory.</li>
</ol>
The decision on which choice to use is system-dependent. If memory is cheap and plentiful, and file I/O is relatively expensive, either in terms of time or charges to transfer data over a data link (maybe the data is remote, accessed over a cellular link), then the single-pass solution is best, choice #1.<br />
<br />
On a small-memory system, a multi-pass solution is better, and you just have to live with the extra I/O. In that case, choice #4, which is the most complicated code, has the best compromise of low memory and low I/O consumption.<br />
<br />
Although if you're really pressed for code space, the simplest multipass solution that iterates over the entire file for each connection is the better choice, #2.<br />
<br />
Realistically though, you don't run tools such as this locally on small systems. Where possible, you offload the data to a larger system and run the tools offline.<br />
<br />
In this case, I'm running on a Mac with 16GB of memory. Slurping up a 200MB text file and holding everything in memory is nothing. So the single-pass solution is the way to go.<br />
<br />
<b>Loading The Data</b><br />
<br />
The log file may just be a chronological sample of all the activity logged by the server, so some connections will already exist at the start of the log, and some will remain at the end. Meanwhile, connection starts and terminations will appear in the log.<br />
<br />
The program parses the lines from the log to find connection lines (i.e. some activity for a given connection). It identifies them by looking for a connection ID, which consists of an IPv4 address/port pair (a remote socket ID), and may optionally include a hexadecimal client ID.<br />
<br />
It uses just the socket ID as the connection ID, which is the key to the connection map. When it finds a new connection ID, it adds it to the map with an empty list. For each connection line, it finds the connection map entry and appends the line to the list of lines for that connection.<br />
<br />
As it loads connection lines, it filters them against the set of IP addresses to skip (these skip address are due to logging of other types of connections besides the client connections). That helps reduce the noise from a large log file.<br />
<br />
<b>Printing The Data</b><br />
<br />
The program prints the connections by first iterating through the map and printing out a summary of each one: the connection ID, the number of lines, the duration of the connection data found in the log, and how its lifetime relates to the overall log. Since the map is an ordered structure, printing connections is always in sorted order by connection ID (though string sorted, not numeric sorted).<br />
<br />
Then it makes a second pass through the connection map, iterating through each line for each connection and printing it out, with separators between connections. It prints a header and summary line before and after the activity lines for each connection, showing where the connection starts and ends relative to the start and end of the log.<br />
<br />
As a last quick change, I added a threshold time value to clearly identify connections that ended at least 60 seconds before the end of the log. This would be a good candidate for a command-line parameter to override the default threshold.<br />
<br />
All the output lines have unique greppable features or keywords so you can use other tools for additional postprocessing or extraction. For instance, I could grep out the end summary line of each connection, and maybe the last couple of activity lines before it, to see how each connection ended up. I could use the "threshold exceeded" indication to identify the ones that had ended early.<br />
<br />
<b>Some Design Evolution</b><br />
<br />
This program adds the <i>isMember()</i> function, which determines whether a string is a member of a set of strings. Since my usage here was intended to deal with a small set and I had other functions that had similar iterative structure, in the heat of battle I quickly coded it as a linear search of a vector of strings.<br />
<br />
That worked fine here, but as I pulled a bunch of this code out into a general string processing module, I realized that was a bad choice, because it's an O(N) search.<br />
<br />
That became especially bad when I wanted an overload that took a vector of strings and determined if they were all members. That meant an O(M) repetition of O(N) searches: an O(M*N) or effectively O(N^2) algorithm.<br />
<br />
That gets out of hand fast as M and N get larger. Meanwhile, the <i>std::unordered_set</i> is perfect for this, an O(1) algorithm for single searches, and an O(M) algorithm when repeated for M items.<br />
<br />
I've left the original <i>isMember()</i> implementation here as an example of the evolution of a concept as you generalize it for other uses.<br />
<br />
I also threw in a few overloads that I didn't end up using, but that set the stage for running with the concept in the string processing module. More discussion of that in the post containing the module.<br />
<br />
<b>The Result</b><br />
<br />
This program turned out to be another fun exercise in string processing once I had built the basic support functions and could see the problem in those terms. It felt more like working in Python, and in fact just as the dict structure from msgresolve.cpp was inspired by Python, so are the <i>split()</i> and <i>join()</i> functions here.<br />
<br />
The funny thing is that it took doing stuff in Python to make me see this approach. That points out one of the advantages of working in multiple different languages: you start seeing opportunities to apply some of the common idioms from one language in another language.<br />
<br />
Here's <i>logsplit.cpp</i>:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-style: italic;">// Usage: logsplit <serverLog></span>
<span style="color: #008800; font-style: italic;">//</span>
<span style="color: #008800; font-style: italic;">// Splits a server log file by IPv4 connection. Prints a</span>
<span style="color: #008800; font-style: italic;">// summary list of the connections, then the log lines for</span>
<span style="color: #008800; font-style: italic;">// each separate connection.</span>
<span style="color: #008800; font-style: italic;">//</span>
<span style="color: #008800; font-style: italic;">// This is an example of a C++ program that is written mostly</span>
<span style="color: #008800; font-style: italic;">// in plain C style, but that makes use of the container and</span>
<span style="color: #008800; font-style: italic;">// composition classes in the C++ standard library. It is a</span>
<span style="color: #008800; font-style: italic;">// lightweight use of C++ with no user-defined classes.</span>
<span style="color: #008800; font-style: italic;">//</span>
<span style="color: #008800; font-style: italic;">// 2018 Steve Branam <sdbranam@gmail.com> learntocode</span>
<span style="color: #008800;">#include <iostream></span>
<span style="color: #008800;">#include <iomanip></span>
<span style="color: #008800;">#include <sstream></span>
<span style="color: #008800;">#include <vector></span>
<span style="color: #008800;">#include <list></span>
<span style="color: #008800;">#include <map></span>
<span style="color: #aa22ff; font-weight: bold;">enum</span> ARGS
{
ARGS_PROGNAME,
ARGS_SERVER_LOG,
ARGS_REQUIRED,
ARGS_SKIP <span style="color: #666666;">=</span> ARGS_REQUIRED
};
<span style="color: #aa22ff; font-weight: bold;">enum</span> SERVER
{
SERVER_DATE,
SERVER_TIME,
SERVER_THREAD,
SERVER_SEVERITY,
SERVER_FUNC,
SERVER_CONN,
SERVER_TIME_LEN <span style="color: #666666;">=</span> <span style="color: #666666;">16</span>,
SERVER_TIMESTAMP_LEN <span style="color: #666666;">=</span> <span style="color: #666666;">28</span>
};
<span style="color: #aa22ff; font-weight: bold;">enum</span> CONN
{
CONN_IP,
CONN_PORT,
CONN_CLIENT_ID
};
<span style="color: #aa22ff; font-weight: bold;">enum</span>
{
END_TIME_THRESHOLD <span style="color: #666666;">=</span> <span style="color: #666666;">60</span>
};
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>string String;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>vector<span style="color: #666666;"><</span>String<span style="color: #666666;">></span> StringVec;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>list<span style="color: #666666;"><</span>String<span style="color: #666666;">></span> StringList;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>map<span style="color: #666666;"><</span>String, StringList<span style="color: #666666;">></span> ConnMap;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>pair<span style="color: #666666;"><</span>String, StringList<span style="color: #666666;">></span> ConnMapEntry;
<span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span> <span style="color: #666666;">*</span>timeFormat <span style="color: #666666;">=</span> <span style="color: #bb4444;">"%Y-%m-%d %H:%M:%S"</span>;
StringVec skipIps;
<span style="color: #00bb00; font-weight: bold;">size_t</span> lines <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>;
<span style="color: #00bb00; font-weight: bold;">size_t</span> skipped <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>;
String firstTimestamp;
String lastTimestamp;
ConnMap connections;
StringVec <span style="color: #00a000;">split</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str, <span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> delim)
{
<span style="color: #00bb00; font-weight: bold;">char</span> buffer[str.size() <span style="color: #666666;">+</span> <span style="color: #666666;">1</span>];
StringVec strings;
strcpy(buffer, str.c_str());
<span style="color: #00bb00; font-weight: bold;">char</span> <span style="color: #666666;">*</span>token <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>strtok(buffer, delim);
<span style="color: #aa22ff; font-weight: bold;">while</span> (token <span style="color: #666666;">!=</span> <span style="color: #aa22ff;">NULL</span>) {
strings.push_back(token);
token <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>strtok(<span style="color: #aa22ff;">NULL</span>, delim);
}
<span style="color: #aa22ff; font-weight: bold;">return</span> strings;
}
String <span style="color: #00a000;">join</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> strings, <span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> sep,
<span style="color: #00bb00; font-weight: bold;">size_t</span> start <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>, <span style="color: #00bb00; font-weight: bold;">size_t</span> end <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>)
{
String str;
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>end) {
end <span style="color: #666666;">=</span> strings.size();
}
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">size_t</span> i <span style="color: #666666;">=</span> start; i <span style="color: #666666;"><</span> end; <span style="color: #666666;">++</span>i) {
str.append(strings[i]);
<span style="color: #aa22ff; font-weight: bold;">if</span> (i <span style="color: #666666;">+</span> <span style="color: #666666;">1</span> <span style="color: #666666;"><</span> end) {
str.append(sep);
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> str;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isMember</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span>str, <span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> set)
{
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">size_t</span> i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> set.size(); <span style="color: #666666;">++</span>i) {
<span style="color: #aa22ff; font-weight: bold;">if</span> (str <span style="color: #666666;">==</span> set[i]) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #aa22ff; font-weight: bold;">typedef</span> <span style="color: #00a000;">int</span> (<span style="color: #666666;">*</span>CharMatch)(<span style="color: #00bb00; font-weight: bold;">int</span> c);
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isToken</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> token, CharMatch isMatch)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (token.empty()) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">size_t</span> i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> token.size(); <span style="color: #666666;">++</span>i)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>isMatch(token[i])) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isToken</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> tokens, CharMatch isMatch,
<span style="color: #00bb00; font-weight: bold;">size_t</span> start <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>, <span style="color: #00bb00; font-weight: bold;">size_t</span> end <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>end) {
end <span style="color: #666666;">=</span> tokens.size();
}
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">size_t</span> i <span style="color: #666666;">=</span> start; i <span style="color: #666666;"><</span> end; <span style="color: #666666;">++</span>i) {
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>isToken(tokens[i], isMatch)) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isNumeric</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> token)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> isToken(token, isdigit);
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isHex</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> token)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> isToken(token, isxdigit);
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isNumeric</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> tokens,
<span style="color: #00bb00; font-weight: bold;">size_t</span> start <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>, <span style="color: #00bb00; font-weight: bold;">size_t</span> end <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> isToken(tokens, isdigit, start, end);
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isIpv4Address</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str)
{
StringVec tokens(split(str, <span style="color: #bb4444;">"."</span>));
<span style="color: #aa22ff; font-weight: bold;">return</span> ((tokens.size() <span style="color: #666666;">==</span> <span style="color: #666666;">4</span>) <span style="color: #666666;">&&</span>
isNumeric(tokens));
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isIpv4Port</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> ((str.size() <span style="color: #666666;"><=</span> <span style="color: #666666;">5</span>) <span style="color: #666666;">&&</span>
isNumeric(str));
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isIpv4Socket</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> strings)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> ((strings.size() <span style="color: #666666;">>=</span> <span style="color: #666666;">2</span>) <span style="color: #666666;">&&</span>
isIpv4Address(strings[<span style="color: #666666;">0</span>]) <span style="color: #666666;">&&</span>
isIpv4Port(strings[<span style="color: #666666;">1</span>]));
}
<span style="color: #00bb00; font-weight: bold;">time_t</span> <span style="color: #00a000;">getTime</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> strTime, <span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> format)
{
std<span style="color: #666666;">::</span>tm t <span style="color: #666666;">=</span> {};
std<span style="color: #666666;">::</span>istringstream ss(strTime);
ss <span style="color: #666666;">>></span> std<span style="color: #666666;">::</span>get_time(<span style="color: #666666;">&</span>t, format);
<span style="color: #aa22ff; font-weight: bold;">return</span> mktime(<span style="color: #666666;">&</span>t);
}
<span style="color: #00bb00; font-weight: bold;">time_t</span> <span style="color: #00a000;">getTime</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> field)
{
<span style="color: #008800; font-style: italic;">// Skip opening and closing brackets.</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> getTime(field.substr(<span style="color: #666666;">1</span>, SERVER_TIMESTAMP_LEN <span style="color: #666666;">-</span> <span style="color: #666666;">2</span>),
timeFormat);
}
<span style="color: #00bb00; font-weight: bold;">size_t</span> <span style="color: #00a000;">getDuration</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">time_t</span><span style="color: #666666;">&</span> start, <span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">time_t</span><span style="color: #666666;">&</span> stop)
{
<span style="color: #00bb00; font-weight: bold;">size_t</span> seconds(difftime(stop, start));
<span style="color: #aa22ff; font-weight: bold;">return</span> seconds;
}
<span style="color: #00bb00; font-weight: bold;">size_t</span> <span style="color: #00a000;">getDuration</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> start, <span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> stop)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> getDuration(getTime(start), getTime(stop));
}
<span style="color: #00bb00; font-weight: bold;">size_t</span> <span style="color: #00a000;">getDuration</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">time_t</span><span style="color: #666666;">&</span> start, <span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> stop)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> getDuration(start, getTime(stop));
}
<span style="color: #00bb00; font-weight: bold;">size_t</span> <span style="color: #00a000;">getDuration</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> start, <span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">time_t</span><span style="color: #666666;">&</span> stop)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> getDuration(getTime(start), stop);
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isServerTime</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (str.size() <span style="color: #666666;">==</span> SERVER_TIME_LEN) {
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">size_t</span> i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> str.size(); <span style="color: #666666;">++</span>i)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>isdigit(str[i]) <span style="color: #666666;">&&</span>
(str[i] <span style="color: #666666;">!=</span> <span style="color: #bb4444;">':'</span>) <span style="color: #666666;">&&</span>
(str[i] <span style="color: #666666;">!=</span> <span style="color: #bb4444;">'.'</span>) <span style="color: #666666;">&&</span>
(str[i] <span style="color: #666666;">!=</span> <span style="color: #bb4444;">']'</span>)) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isConnId</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str)
{
StringVec fields(split(str, <span style="color: #bb4444;">":"</span>));
<span style="color: #aa22ff; font-weight: bold;">return</span> (isIpv4Socket(fields) <span style="color: #666666;">&&</span>
(fields.size() <span style="color: #666666;"><</span> CONN_CLIENT_ID <span style="color: #666666;">+</span> <span style="color: #666666;">1</span> <span style="color: #666666;">||</span>
isHex(fields[CONN_CLIENT_ID])));
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isServerConn</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> fields)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> ((fields.size() <span style="color: #666666;">></span> SERVER_CONN) <span style="color: #666666;">&&</span>
isServerTime(fields[SERVER_TIME]) <span style="color: #666666;">&&</span>
isConnId(fields[SERVER_CONN]));
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">loadServer</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> fileName)
{
<span style="color: #00bb00; font-weight: bold;">FILE</span><span style="color: #666666;">*</span> file <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>fopen(fileName, <span style="color: #bb4444;">"r"</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (file) {
<span style="color: #00bb00; font-weight: bold;">char</span> buffer[<span style="color: #666666;">1000</span>];
<span style="color: #aa22ff; font-weight: bold;">while</span> (std<span style="color: #666666;">::</span>fgets(buffer, <span style="color: #aa22ff; font-weight: bold;">sizeof</span>(buffer), file) <span style="color: #666666;">!=</span> <span style="color: #aa22ff;">NULL</span>) {
String line(buffer);
StringVec fields <span style="color: #666666;">=</span> split(buffer, <span style="color: #bb4444;">" </span><span style="color: #bb6622; font-weight: bold;">\t</span><span style="color: #bb4444;">"</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (isServerConn(fields)) {
<span style="color: #666666;">++</span>lines;
lastTimestamp <span style="color: #666666;">=</span> line.substr(<span style="color: #666666;">0</span>, SERVER_TIMESTAMP_LEN);
<span style="color: #aa22ff; font-weight: bold;">if</span> (firstTimestamp.empty()) {
firstTimestamp <span style="color: #666666;">=</span> lastTimestamp;
}
strncpy(buffer, fields[SERVER_CONN].c_str(),
<span style="color: #aa22ff; font-weight: bold;">sizeof</span>(buffer));
StringVec conn <span style="color: #666666;">=</span> split(buffer, <span style="color: #bb4444;">":"</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (isMember(conn[CONN_IP], skipIps)) {
<span style="color: #666666;">++</span>skipped;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
String key(conn[CONN_IP]);
key.append(<span style="color: #bb4444;">":"</span>);
key.append(conn[CONN_PORT]);
ConnMap<span style="color: #666666;">::</span>iterator match;
match <span style="color: #666666;">=</span> connections.find(key);
<span style="color: #aa22ff; font-weight: bold;">if</span> (match <span style="color: #666666;">==</span> connections.end()) {
connections.insert(ConnMapEntry(key,
StringList()));
match <span style="color: #666666;">=</span> connections.find(key);
}
match<span style="color: #666666;">-></span>second.push_back(line);
}
}
}
std<span style="color: #666666;">::</span>fclose(file);
<span style="color: #aa22ff; font-weight: bold;">if</span> (connections.empty()) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"No connections found"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Failed to open server file"</span>
<span style="color: #666666;"><<</span> fileName <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">printSeparator</span>()
{
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"=-=-=-=-"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl
<span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">listConnections</span>()
{
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> connections.size() <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" connections "</span>
<span style="color: #666666;"><<</span> firstTimestamp <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"-"</span> <span style="color: #666666;"><<</span> lastTimestamp <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" "</span>
<span style="color: #666666;"><<</span> lines <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" lines, "</span>
<span style="color: #666666;"><<</span> getDuration(firstTimestamp, lastTimestamp) <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" sec:"</span>
<span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">if</span> (skipIps.size()) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"(skipped "</span> <span style="color: #666666;"><<</span> skipped <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" connections with "</span>
<span style="color: #666666;"><<</span> join(skipIps, <span style="color: #bb4444;">", "</span>) <span style="color: #666666;"><<</span> <span style="color: #bb4444;">")"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">for</span> (ConnMap<span style="color: #666666;">::</span>iterator curConn <span style="color: #666666;">=</span> connections.begin();
curConn <span style="color: #666666;">!=</span> connections.end();
curConn<span style="color: #666666;">++</span>) {
String conn(curConn<span style="color: #666666;">-></span>first);
StringList connLogs(curConn<span style="color: #666666;">-></span>second);
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> conn <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"</span><span style="color: #bb6622; font-weight: bold;">\t</span><span style="color: #bb4444;">"</span>
<span style="color: #666666;"><<</span> connLogs.front().substr(<span style="color: #666666;">0</span>, SERVER_TIMESTAMP_LEN)
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"-"</span> <span style="color: #666666;"><<</span> connLogs.back().substr(<span style="color: #666666;">0</span>, SERVER_TIMESTAMP_LEN)
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" "</span> <span style="color: #666666;"><<</span> connLogs.size() <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" lines, "</span>
<span style="color: #666666;"><<</span> getDuration(connLogs.front(), connLogs.back())
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" sec"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
printSeparator();
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">logConnections</span>()
{
<span style="color: #00bb00; font-weight: bold;">time_t</span> timeFirst <span style="color: #666666;">=</span> getTime(firstTimestamp);
<span style="color: #00bb00; font-weight: bold;">time_t</span> timeLast <span style="color: #666666;">=</span> getTime(lastTimestamp);
<span style="color: #aa22ff; font-weight: bold;">for</span> (ConnMap<span style="color: #666666;">::</span>iterator curConn <span style="color: #666666;">=</span> connections.begin();
curConn <span style="color: #666666;">!=</span> connections.end();
curConn<span style="color: #666666;">++</span>) {
String conn(curConn<span style="color: #666666;">-></span>first);
StringList connLogs(curConn<span style="color: #666666;">-></span>second);
<span style="color: #00bb00; font-weight: bold;">size_t</span> duration(getDuration(connLogs.front(), connLogs.back()));
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Connection "</span> <span style="color: #666666;"><<</span> conn
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" "</span> <span style="color: #666666;"><<</span> connLogs.front().substr(<span style="color: #666666;">0</span>, SERVER_TIMESTAMP_LEN)
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"-"</span> <span style="color: #666666;"><<</span> connLogs.back().substr(<span style="color: #666666;">0</span>, SERVER_TIMESTAMP_LEN)
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" "</span> <span style="color: #666666;"><<</span> connLogs.size() <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" lines, "</span>
<span style="color: #666666;"><<</span> duration <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" sec:"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #00bb00; font-weight: bold;">size_t</span> seconds <span style="color: #666666;">=</span> getDuration(timeFirst, connLogs.front());
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> firstTimestamp <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" Starts "</span> <span style="color: #666666;"><<</span> seconds
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" sec after start of log."</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">for</span> (StringList<span style="color: #666666;">::</span>iterator curLog <span style="color: #666666;">=</span> connLogs.begin();
curLog <span style="color: #666666;">!=</span> connLogs.end();
curLog<span style="color: #666666;">++</span>) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #666666;">*</span>curLog;
}
seconds <span style="color: #666666;">=</span> getDuration(connLogs.back(), timeLast);
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> lastTimestamp
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" "</span> <span style="color: #666666;"><<</span> connLogs.size() <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" lines, "</span>
<span style="color: #666666;"><<</span> duration <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" sec. Ends "</span>
<span style="color: #666666;"><<</span> seconds <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" sec before end of log."</span>;
<span style="color: #aa22ff; font-weight: bold;">if</span> (seconds <span style="color: #666666;">></span> END_TIME_THRESHOLD) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" Exceeds threshold."</span>;
}
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
printSeparator();
}
}
<span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">main</span>(<span style="color: #00bb00; font-weight: bold;">int</span> argc, <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> argv[])
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (argc <span style="color: #666666;"><</span> ARGS_REQUIRED <span style="color: #666666;">||</span>
String(argv[<span style="color: #666666;">1</span>]) <span style="color: #666666;">==</span> <span style="color: #bb4444;">"-h"</span>) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Usage: "</span> <span style="color: #666666;"><<</span> argv[ARGS_PROGNAME]
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" <serverLog> [<skipIps>]"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Where <skipIps> is comma-separated list "</span>
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"of IP addresses to skip."</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">if</span> (argc <span style="color: #666666;">></span> ARGS_REQUIRED) {
skipIps <span style="color: #666666;">=</span> split(argv[ARGS_SKIP], <span style="color: #bb4444;">","</span>);
}
<span style="color: #aa22ff; font-weight: bold;">if</span> (loadServer(argv[ARGS_SERVER_LOG])) {
listConnections();
logConnections();
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_SUCCESS;
}
</pre>
</td></tr>
</tbody></table>
</div>
<br />
The sample log file, <i>conns.log</i>:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25</span></pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;">[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] start() <span style="color: #666666;">10.1.180.206:30450</span> TCP socket receive_buffer_size<span style="color: #666666;">=117708</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469678</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] start() <span style="color: #666666;">10.1.180.206:30450</span> TCP socket send_buffer_size<span style="color: #666666;">=43520</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.867381</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] set_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450</span> set idle send timeout to <span style="color: #666666;">60</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.867394</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] set_idle_receive_timeout() <span style="color: #666666;">10.1.180.206:30450</span> set idle receive timeout to <span style="color: #666666;">120</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.867450</span>] [<span style="color: #666666;">0x00007fb3ff92a700</span>] [info] handle_connected() <span style="color: #666666;">10.1.180.206:30450</span> remote connected [<span style="color: #666666;">2423/8951</span>]
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.959877</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] install_connection() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 linkType <span style="color: #666666;">0</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.966599</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0x4a2137a6</span>, <span style="color: #666666;">231</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.966935</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0xa11f878a</span>, <span style="color: #666666;">35</span> bytes, msg type <span style="color: #666666;">3</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:29.967117</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x1c35386e</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">1</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:29.967228</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">59</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:30.086722</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0xf6e3f3bf</span>, <span style="color: #666666;">29</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:30.086813</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0xfb208d9f</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">17</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086722</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.2.80.206:3050:00000</span>bd4 RX sum <span style="color: #666666;">0xf6e3f3bf</span>, <span style="color: #666666;">29</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086813</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.2.80.206:3050:00000</span>bd4 TX sum <span style="color: #666666;">0xfb208d9f</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">17</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:30.139377</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0xa0f8a8e1</span>, <span style="color: #666666;">78</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:29.867494</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_receive_timeout() <span style="color: #666666;">127.0.0.1:32450:00003</span>bd4 <span style="color: #666666;">60</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:29.967315</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">0</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.086988</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x8fcf53f6</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">1</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.087101</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">59</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.197029</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0x515f1d4e</span>, <span style="color: #666666;">29</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.197120</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x6827d190</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">17</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.249027</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0xda66c5c7</span>, <span style="color: #666666;">78</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.087189</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">0</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.139486</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_receive_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">60</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x812afb16</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">1</span>
</span></pre>
</td></tr>
</tbody></table>
</div>
<br />
Sample run:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47</span></pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;"><span style="border: 1px solid #FF0000;">$</span> g<span style="color: #666666;">++</span> logsplit.cpp <span style="color: #666666;">-</span>o logsplit
<span style="border: 1px solid #FF0000;">$</span> .<span style="color: #666666;">/</span>logsplit conns.log <span style="color: #666666;">127.0.0.1</span>,<span style="color: #666666;">3.3.3.3</span>
<span style="color: #666666;">2</span> connections [<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>]<span style="color: #666666;">-</span>[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] <span style="color: #666666;">25</span> lines, <span style="color: #666666;">181</span> sec<span style="color: #666666;">:</span>
(skipped <span style="color: #666666;">1</span> connections with <span style="color: #666666;">127.0.0.1</span>, <span style="color: #666666;">3.3.3.3</span>)
<span style="color: #666666;">10.1.180.206:30450</span> [<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>]<span style="color: #666666;">-</span>[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] <span style="color: #666666;">22</span> lines, <span style="color: #666666;">181</span> sec
<span style="color: #666666;">10.2.80.206:3050</span> [<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086722</span>]<span style="color: #666666;">-</span>[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086813</span>] <span style="color: #666666;">2</span> lines, <span style="color: #666666;">0</span> sec
<span style="color: #666666;">=-=-=-=-</span>
Connection <span style="color: #666666;">10.1.180.206:30450</span> [<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>]<span style="color: #666666;">-</span>[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] <span style="color: #666666;">22</span> lines, <span style="color: #666666;">181</span> sec<span style="color: #666666;">:</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>] Starts <span style="color: #666666;">0</span> sec after start of log.
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] start() <span style="color: #666666;">10.1.180.206:30450</span> TCP socket receive_buffer_size<span style="color: #666666;">=117708</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469678</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] start() <span style="color: #666666;">10.1.180.206:30450</span> TCP socket send_buffer_size<span style="color: #666666;">=43520</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.867381</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] set_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450</span> set idle send timeout to <span style="color: #666666;">60</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.867394</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] set_idle_receive_timeout() <span style="color: #666666;">10.1.180.206:30450</span> set idle receive timeout to <span style="color: #666666;">120</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.867450</span>] [<span style="color: #666666;">0x00007fb3ff92a700</span>] [info] handle_connected() <span style="color: #666666;">10.1.180.206:30450</span> remote connected [<span style="color: #666666;">2423/8951</span>]
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.959877</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] install_connection() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 linkType <span style="color: #666666;">0</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.966599</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0x4a2137a6</span>, <span style="color: #666666;">231</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.966935</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0xa11f878a</span>, <span style="color: #666666;">35</span> bytes, msg type <span style="color: #666666;">3</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:29.967117</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x1c35386e</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">1</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:29.967228</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">59</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:30.086722</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0xf6e3f3bf</span>, <span style="color: #666666;">29</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:30.086813</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0xfb208d9f</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">17</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:30.139377</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0xa0f8a8e1</span>, <span style="color: #666666;">78</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:29.967315</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">0</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.086988</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x8fcf53f6</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">1</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.087101</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">59</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.197029</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0x515f1d4e</span>, <span style="color: #666666;">29</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.197120</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x6827d190</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">17</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:18:30.249027</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 RX sum <span style="color: #666666;">0xda66c5c7</span>, <span style="color: #666666;">78</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.087189</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_send_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">0</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.139486</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] handle_idle_receive_timeout() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 <span style="color: #666666;">60</span> seconds
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.1.180.206:30450:00003</span>bd4 TX sum <span style="color: #666666;">0x812afb16</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">1</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] <span style="color: #666666;">22</span> lines, <span style="color: #666666;">181</span> sec. Ends <span style="color: #666666;">0</span> sec before end of log.
<span style="color: #666666;">=-=-=-=-</span>
Connection <span style="color: #666666;">10.2.80.206:3050</span> [<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086722</span>]<span style="color: #666666;">-</span>[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086813</span>] <span style="color: #666666;">2</span> lines, <span style="color: #666666;">0</span> sec<span style="color: #666666;">:</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:16:29.469659</span>] Starts <span style="color: #666666;">71</span> sec after start of log.
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086722</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] receive() <span style="color: #666666;">10.2.80.206:3050:00000</span>bd4 RX sum <span style="color: #666666;">0xf6e3f3bf</span>, <span style="color: #666666;">29</span> bytes
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:17:40.086813</span>] [<span style="color: #666666;">0x00007fb3ff129700</span>] [debug] send() <span style="color: #666666;">10.2.80.206:3050:00000</span>bd4 TX sum <span style="color: #666666;">0xfb208d9f</span>, <span style="color: #666666;">29</span> bytes, msg type <span style="color: #666666;">17</span>
[<span style="color: #666666;">2018-04-03</span> <span style="color: #666666;">13:19:30.197322</span>] <span style="color: #666666;">2</span> lines, <span style="color: #666666;">0</span> sec. Ends <span style="color: #666666;">110</span> sec before end of log. Exceeds threshold.
<span style="color: #666666;">=-=-=-=-</span>
</span></pre>
</td></tr>
</tbody></table>
</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-77237882726136012452018-04-14T17:16:00.000-07:002018-04-20T06:13:54.225-07:00First Use Of New Tools<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5zEvt1j8yY5LOHmuN1WvI8Zb3RD3zDuJ5Rr_Uujt7L_8cEY_nW5fIJW4NXliCsugV1zZBwrCVKSQyVvppT90V_mFlWNUP2zDwpDGAeXhr5QkQ4TH_tnCI4TqaI8I3VuDhwIKD8F2H2qhx/s1600/DSCN0195.png" imageanchor="1"><img border="0" data-original-height="1497" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5zEvt1j8yY5LOHmuN1WvI8Zb3RD3zDuJ5Rr_Uujt7L_8cEY_nW5fIJW4NXliCsugV1zZBwrCVKSQyVvppT90V_mFlWNUP2zDwpDGAeXhr5QkQ4TH_tnCI4TqaI8I3VuDhwIKD8F2H2qhx/s640/DSCN0195.png" width="560" /></a>
<br />
<i><span style="color: #38761d;">Using my new tools to fix my Droid MAXX, lying open front and center.</span></i><br />
<br />
Sometimes life just amazes me. Three days after receiving <a href="http://flinkandblink.blogspot.com/2018/04/limor-fried-is-my-new-hero.html">my Adafruit tools</a>, I needed them. <i>Twice</i>.<br />
<br />
This isn't the first time that I've learned something completely new out of the blue, and almost immediately needed it. It's just incredibly serendipitous.<br />
<br />
<b>First Problem</b><br />
<br />
Yesterday, Friday, I worked from home. The project I'm working on is embedded system firmware to control a daughter board. The main smart chip on the board was shutting down unexpectedly, which I could tell initially from the timeouts reading responses from it, and then from its status output LED going off.<br />
<br />
There's a power enable line for the chip in the cable pigtail that I control via software, and scouring through my code assured me that no code path should be shutting it off during the scenario I was exercising.<br />
<br />
Yet the LED was going off and the chip was unresponsive. Maybe it was a bug in the chip, and my commands to it were triggering the shutdown.<br />
<br />
Or maybe something besides my code was affecting the power enable. So I pulled out my new meter, opened up the cable to the daughter card, and clipped the meter into the ground and enable lines, switched to VDC.<br />
<br />
With the code in the state where power was enabled, the LED was on, the chip was responsive, and the meter showed 3.2V. With the code in the state where power was disabled, the LED was off, the chip was unresponsive, and the meter showed 0.0V. Ok, good, correct setup.<br />
<br />
Running my tests, with additional debug logging verifying the code was NOT running through the parts that shutoff power, I saw the LED go off and the meter drop from 3.2 to 0. So something unrelated was indeed dropping the power enable.<br />
<br />
After more testing, I noticed a pattern in the logging, unrelated to my code, that correlated with the shutoff. Now, correlation does not imply causation. But it's a hint about where to look.<br />
<br />
So running a couple more debug commands that didn't exercise my code path at all, I was able to confirm that the correlated activity did indeed result in disabling power to the chip.<br />
<br />
Yes! Phew, it's not the chip crashing on me, and it's not a problem in the code I just wrote.<br />
<br />
Monday, I'll get together with the hardware guys and chase the problem down. First, I'll try it with another unit.<br />
<br />
For all I know, my unit is damaged, since it's all pulled apart on my desk to connect the daughter board. Open frame hardware is always vulnerable.<br />
<br />
That's an example of diagnosing a problem and ruling out two large potential root cause areas.<br />
<blockquote class="tr_bq">
<i>Monday update: Another unit showed the same problem. So it wasn't damage in my unit.</i> </blockquote>
<blockquote class="tr_bq">
<i>I discussed </i><i>the evidence in the logging</i><i> with the hardware engineer. We identified some register accesses used to control GPIO (General Purpose Input/Output) pins on the processor that were affecting more pins than intended. These were causing the GPIO pin controlling the power enable to be cleared.</i></blockquote>
<blockquote class="tr_bq">
<i>This shows one of the challenges inherent in embedded systems. You're working at a much lower, more direct level in the hardware, with lots of opportunities to create strange conflicts between different parts of the system.</i> </blockquote>
<b>Second Problem</b><br />
<br />
Today, Saturday, I was down in the basement, and as I put my phone in my pocket, it slipped and fell flat on its face on the concrete floor with a sharp thud. DOHH!<br />
<br />
It's a Droid Maxx in a case that goes in a holster. The case protected the screen and phone from overall damage, but the power had shut off (yeah, another power problem). I held the power button in, but it wouldn't power on, even after several attempts.<br />
<br />
I plugged the charger cord in. The display came on, showing 0% power. It tried to boot, then quickly shutoff. It just repeated this cycle continuously. So there was some level of system function, but no power.<br />
<br />
The shock of impact had apparently knocked something loose inside, which I figured was most probably the battery connection. Dead phone. Stupid reason. Words were said.<br />
<br />
And this is a sealed phone, not a removable battery like older models. More words were said.<br />
<br />
I emailed my wife who was visiting some friends that my phone was dead. Ugh. I look forward to going to the cell phone store the way I look forward to going to a used car lot (although I'm being unfair, my last several experiences with them have been excellent).<br />
<br />
But Internet to the rescue, I Googled "access battery in droid maxx" and found a page showing how to do it, plus someone in the comments had pointed to a YouTube video they liked better.<br />
<br />
I read <a href="https://www.ifixit.com/Guide/Motorola+DROID+MAXX+Battery+Replacement/33737">the page</a> and watched <a href="https://youtu.be/HJ0989c6TXI">the video</a>. Yep, I can do that. Worst case, the phone is already dead, I can't make it any worse.<br />
<br />
Since I didn't have a spudger or plastic pry tool as shown in the tutorials, I improvised one from a piece of 1/8" thick padauk, which is a very hard tropical wood. I split off a strip about 1/2" wide and beveled the end of it with a block plane.<br />
<br />
Between that and a small flat screwdriver, I was able to get the case open. The metal driver tip did break a couple bits of plastic off the edge of the back, but no major damage.<br />
<br />
This also shows why I ordered multiple tool sets (I've added an iFixit set to the shopping list on my Adafruit post, because it has pry tools). I used the smallest flat screwdriver from one set for lifting tabs, the T5 from another set to remove most of the screws, and the even tinier flat tip from a third set to get into the points of a T4 screw.<br />
<br />
I used the compartments in the base of my Panavise to hold all the loose parts.<br />
<br />
With all the screws removed, I separated the board from the display, and sure enough, I could see immediately that the tiny connector on the flat battery cable was loose on the back of the board. This wasn't the first time I had dropped it, so it probably loosened up progressively. Everything inside the phone is secured in place pretty tightly.<br />
<br />
I pressed on it and it clicked positively into place. I pulled it back off to look at it, then clicked it back in. So I was confident that it was secure.<br />
<br />
As I was handling the phone with everything folded together loosely, I accidentally held the power button in. I noticed the display come on, so I let it continue. It booted up (SIM removed), so power appeared to be restored.<br />
<br />
I shut it down and reassembled the phone. The tweezers were a big help dropping the tiny screws back into their holes.<br />
<br />
Popped the case back together, reinserted the SIM, and held the power button in. Voila! It powered on, connected to the network, and I was back in business, with a couple of quick tests to verify that things were working right and it was staying on. I texted my wife that I had fixed it.<br />
<br />
That's an example of digging in and not being afraid to play around and see what's going on. It's all a learning experience!Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-56588719835599598042018-04-08T17:17:00.003-07:002018-09-22T03:01:52.717-07:00Limor Fried Is My New Hero<iframe allow="encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/_pyt7-86x0c?rel=0" width="560"></iframe>
<br />
<i><span style="color: #38761d;">Meet Limor Fried, founder of Adafruit.</span></i><br />
<br />
I'm cross-posting this to both my woodworking blog <a href="http://www.closegrain.com/">www.CloseGrain.com</a> and my software engineering blog <a href="http://flinkandblink.blogspot.com/">FlinkAndBlink.blogspot.com</a> (under the <a href="http://flinkandblink.blogspot.com/search/label/LearnToCode">LearnToCode</a> label), because even though there's no woodworking in it, this is all about building stuff, so it bridges the worlds. It's the maker ethos.<br />
<br />
If you're interested in <a href="http://flinkandblink.blogspot.com/p/learntocode.html">learning to code</a>, and actually building the stuff that you're coding on, this is for you. This is all about working on embedded systems, from the hobby level to the professional.<br />
<br />
<b>In Which I Find Out About Limor Fried</b><br />
<br />
Allow me a moment of self-indulgent gushing admiration here. Or you can skip down to the real information that starts at the <b>Electronics Learning Resources</b> heading.<br />
<br />
I admit to instant and total nerd-crush. Limor Fried, who goes by the name Ladyada online (for <a href="https://en.wikipedia.org/wiki/Ada_Lovelace">Lady Ada Lovelace</a>, The First Programmer) is the founder of <a href="https://www.adafruit.com/">Adafruit</a>.<br />
<br />
Adafruit is a small electronics manufacturing company in Manhattan, NY, that focuses on teaching electronics to makers of all ages. You can read about them <a href="https://www.adafruit.com/about">here</a>.<br />
<br />
Electronics is another of those hobbies that I wanted to pursue as a teenager, but never could due to lack of funds. Fortunately I've advanced beyond that impecunious stage of life, and seeing this has fired instant obsession (hence the shopping list below!).<br />
<br />
I'm familiar with that feeling of obsession settling on my shoulders. It propelled me into hand tool woodworking, turning into a <a href="http://www.closegrain.com/2017/11/hand-tool-basics-book-available-for.html">book</a>. It propelled me into <a href="http://www.closegrain.com/2015/10/greenish-luthiery.html">violinmaking</a>. It propelled me into <a href="http://www.closegrain.com/2017/01/when-i-grow-up.html">boatbuilding</a>.<br />
<br />
Each time, the pattern is the same. I buy a bunch of books, watch a bunch of videos, dig through a bunch of blogs and forums, then buy a bunch of tools and start playing. Last year it propelled me into small engine repair and oxy-acetylene welding after I found <a href="https://www.youtube.com/channel/UC2Cd9kKYXFakV7ChvU_rjKw">Taryl Dactyl</a> (yes, blog posts will be forthcoming).<br />
<br />
Now, in my copious free time (that's a joke, son), I'll finally be realizing that dream to get my hands dirty with electronics.<br />
<br />
I owe this to Matt Pandina, whom we recently hired at work. It quickly turned out that Matt is a maker and likes sharing information. He has some nice stuff on Google Groups under the moniker <a href="https://sites.google.com/site/artcfox/">artcfox</a> (in fact, one of his articles was coincidentally the answer to the embedded systems programming problem I use when interviewing candidates!).<br />
<br />
He made a comment about how Adafruit is doing manufacturing in Manhattan, and I asked, "Who's Adafruit?". That was all it took. Thanks, Matt!<br />
<br />
I was tickled to read Fried's favorite quote in the <a href="https://blog.adafruit.com/2012/12/18/adafruits-ladyada-limor-fried-named-entrepreneur-of-the-year-by-entrepreneur-magazine-entmagazine/">Entrepreneur Magazine</a> article about her:<br />
<blockquote class="tr_bq">
<em style="background-color: #eeeeee; box-sizing: border-box; font-family: "proxima nova", "lucida grande", "lucida sans unicode", "lucida sans", Geneva, Verdana, sans-serif; font-size: 15px; letter-spacing: 0.28px; margin: 0px; padding: 0px;">“We are what we celebrate.”</em><span style="background-color: #eeeeee; font-family: "proxima nova" , "lucida grande" , "lucida sans unicode" , "lucida sans" , "geneva" , "verdana" , sans-serif; font-size: 15px; letter-spacing: 0.28px;"> —entrepreneur and inventor Dean Kamen.</span></blockquote>
Kamen is one of my other heroes. She whose hero is my hero is my hero!<br />
<br />
I managed to score his autograph at the <a href="https://www.massmedic.com/2015/04/22/massmedics-19th-annual-conference-only-two-weeks-away/">2015 MassMEDIC conference</a>. I was at the <a href="http://my.presentations.techweb.com/events/esc/boston/2015/conference">2015 Embedded Systems Conference</a> (ESC Boston), which was being held concurrently at the Boston Convention Center.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi87RC92tHJTLfSreI2wwIwuZ5-oiFqAf-ahTH-Qgc4q-q9cYANL7sq2MfrzQxXwnjT6NRgFvVXKRUpCx-SoRxrrOG5S5MvNhte15fgKkWSoY7jYfrG7FkdjMBaMTKhdt9AYnz3YuLpi-Yv/s1600/Dean+Kamen+May+6+2015.jpg" imageanchor="1"><img border="0" data-original-height="1600" data-original-width="995" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi87RC92tHJTLfSreI2wwIwuZ5-oiFqAf-ahTH-Qgc4q-q9cYANL7sq2MfrzQxXwnjT6NRgFvVXKRUpCx-SoRxrrOG5S5MvNhte15fgKkWSoY7jYfrG7FkdjMBaMTKhdt9AYnz3YuLpi-Yv/s640/Dean+Kamen+May+6+2015.jpg" width="396" /></a><br />
<i><span style="color: #38761d;">When I saw Kamen listed as keynote speaker, I scooted down early and got a chance to talk to him and tell him I wanted to work for him (he probably gets a lot of stalker geeks like that!). Came close the following year, but logistics didn't work out.</span></i><br />
<br />
<b>Electronics Learning Resources</b><br />
<br />
On the business side, Adafruit <a href="https://www.adafruit.com/categories">sells kits, parts, tools, and books</a>. That's pretty cool (along with being able to pull off a manufacturing operation in Manhattan). But what's truly spectacular about them is their online learning resources.<br />
<br />
Fried is a big proponent of open source, sharing the knowledge. So the Adafruit website is <a href="https://learn.adafruit.com/">chock full of information</a>. There's also an extensive <a href="https://www.youtube.com/adafruit">YouTube channel</a>.<br />
<br />
You'll also finds lots of cross-pollination with others in the maker community. There are magazines, blogs, and videos by the score, by independent makers like Matt, and by larger organizations.<br />
<br />
I've just barely begun to scratch the surface. This is great, because I know how to program embedded systems, but I don't know much about the components that go into them and connect to them. It's the combination of hardware and software that really makes something work.<br />
<br />
Pretty much everything I know about digital electronics I owe to <a href="https://amzn.to/2JwzBFQ">Forrest M. Mims</a> and <a href="https://amzn.to/2KnzrRG">George Young</a> 35 years ago. Now, after that brief hiatus, I can take the next step.<br />
<br />
<b>Basic Electronics Lab Skills</b><br />
<br />
<iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/w0c3t0fJhXU?rel=0" width="560"></iframe><br />
<i><span style="color: #38761d;">Step into Collin's lab!</span></i><br />
<br />
Among the resources is a series of very accessible quick guides and videos by <a href="https://learn.adafruit.com/category/collins-lab">Collin Cunningham</a>. Of particular interest to the electronics beginner such as myself is this set of basic electronics lab skills (you can scan through all these for quick grok of the big picture by setting the speed in the YouTube window settings (the gear icon) to 2x, then come back and watch at normal speed for a second pass):<br />
<ul>
<li><a href="https://learn.adafruit.com/collins-lab-soldering">Soldering and Desoldering</a>: how to solder components together properly, and how to pull them apart for salvage and rework.</li>
<li><a href="https://youtu.be/QzoPxvIM2qE">Surface Mount Soldering</a>: how to solder surface-mount components.</li>
<li><a href="https://learn.adafruit.com/collins-lab-multimeters">Multimeters</a>: how to use a meter for basic measurements.</li>
<li><a href="https://youtu.be/ThrK2spjrLs">Oscilloscopes</a>: how to use an oscilloscope for advanced measurements and waveforms.</li>
<li><a href="https://youtu.be/J-1phA_vKDg">Hand Tools</a>: the basic hand tools used for assembling and disassembling electronics.</li>
<li><a href="https://youtu.be/9cps7Q_IrX0">Schematics</a>: how to read schematics (no, they're not Greek!).</li>
<li><a href="https://youtu.be/w0c3t0fJhXU">Breadboards and Perfboards</a>: how to combine the parts on a schematic into a functioning circuit.</li>
<li><a href="https://youtu.be/-mHLvtGjum4">Ohm's Law</a>: understanding the relationship between voltage, current, and resistance.</li>
</ul>
Once you have these skills, you are unleashed. Just like hand tool woodworking, it takes a little investment in tools and equipment, and a little time practicing with them.<br />
<br />
These form the basis of the shopping list below. And of course they lead to lots of other interesting videos, like Collin's videos on the basics of various components:<br />
<ul>
<li><a href="https://youtu.be/8ftkn7nZSDo">Batteries</a>: the basics of using batteries to supply DC power to projects.</li>
<li><a href="https://youtu.be/RRW4pBCzXBA">Solar Cells</a>: using solar cells to keep the batteries charged.</li>
<li><a href="https://youtu.be/qPt0X5s-1JU">Power Supplies</a>: using an AC power supply to supply DC power to projects.</li>
<li><a href="https://youtu.be/Lf7JJAAZxEU">Pulse Width Modulation</a>: using a PWM converter to change DC input voltage to lower effective DC voltage, or as a simple digital-to-analog converter (DAC).</li>
<li><a href="https://youtu.be/S2AHimvbovI">Switches</a>: understanding the different types of switches for manually controlling projects. </li>
<li><a href="https://youtu.be/-td7YT-Pums">The Transistor</a></li>
<li><a href="https://youtu.be/ZYH9dGl4gUE">The Capacitor</a></li>
<li><a href="https://youtu.be/AqzYsuTRVRc">The Diode</a></li>
<li><a href="https://youtu.be/STDlCdZnIsw">The Inductor</a></li>
<li><a href="https://youtu.be/VPVoY1QROMg">The Resistor</a></li>
<li><a href="https://youtu.be/P3PDLsJQcGI">The LED</a></li>
<li><a href="https://youtu.be/uSRIc-sEgPw">The Integrated Circuit (IC)</a></li>
<li><a href="https://youtu.be/pnf8ojsK6S4">The Arduino</a></li>
</ul>
There are also a number of other introductory Adafruit written guides by various contributors (as well as oceans of more specialized and advanced guides, <a href="https://learn.adafruit.com/">check them out</a>!):<br />
<ul>
<li><a href="https://learn.adafruit.com/adafruit-guide-excellent-soldering">Adafruit Guide To Excellent Soldering</a></li>
<li><a href="https://learn.adafruit.com/multimeters">Multimeters</a></li>
<li><a href="https://learn.adafruit.com/wires-and-connections">Wires and Connections</a></li>
<li><a href="https://learn.adafruit.com/all-about-batteries">All About Batteries</a></li>
<li><a href="https://learn.adafruit.com/power-supplies">Power Supplies</a></li>
<li><a href="https://learn.adafruit.com/all-about-leds">All About LEDs</a></li>
</ul>
<b>Shopping List</b><br />
<br />
These are the tools, equipment, supplies, and books to do the work. With the exception of the oscilloscope and logic analyzer, these are all links to the Adafruit shopping pages. Prices as of April 8, 2018.<br />
<br />
Tools and equipment:<br />
<ul>
<li><a href="https://www.adafruit.com/product/1204">Hakko FX-888D Digital Soldering Iron</a>, $109.95</li>
<li><a href="https://www.adafruit.com/product/1286">Hakko Soldering Tip: T18-D24 Screwdriver</a>, $7.95</li>
<li><a href="https://www.adafruit.com/product/1248">Hakko Soldering Tip: T18-C2 Hoof</a>, $7.95</li>
<li><a href="https://www.adafruit.com/product/1249">Hakko Soldering Tip: T18-S4 Fine SMD</a>, $9.95</li>
<li><a href="https://www.adafruit.com/product/2452">Panavise Multi-Purpose Work Center</a>, $99.95</li>
<li><a href="https://www.adafruit.com/product/3019">Third Hand Pana Hand Workstation Add-On</a>, $54.95</li>
<li><a href="https://www.adafruit.com/product/291">Helping Third Hand Magnifier W/Magnifying Glass</a>, $6.00</li>
<li><a href="https://www.adafruit.com/product/152">Flush diagonal cutters</a>, $7.25</li>
<li><a href="https://www.adafruit.com/product/146">Simple pliers</a>, $3.00</li>
<li><a href="https://www.adafruit.com/product/527">Hakko Professsional Quality 20-30 AWG Wire Strippers</a>, $14.95</li>
<li><a href="https://www.adafruit.com/product/3284">Adafruit Pocket Screwdriver</a>, $1.50</li>
<li><a href="https://www.adafruit.com/product/424">Precision screwdriver set (6 pieces)</a>, $7.95</li>
<li><a href="https://www.adafruit.com/product/452">Precision Torx Screwdriver Set (6 pieces)</a>, $6.95</li>
<li><a href="https://www.adafruit.com/product/829">65 Piece Ratchet Screwdriver and Tool Bit Set</a>, $29.95</li>
<li><a href="https://www.adafruit.com/product/3445">iFixit Essential Electronics Toolkit</a>, $19.95</li>
<li><a href="https://www.adafruit.com/product/1599">Super Scissors</a>, $14.95</li>
<li><a href="https://www.adafruit.com/product/3720">Solar Digital Calipers</a>, $14.95</li>
<li><a href="https://www.adafruit.com/product/421">Fine tip straight tweezers - ESD safe</a>, $3.95</li>
<li><a href="https://www.adafruit.com/product/422">Fine tip curved tweezers - ESD safe</a>, $3.95</li>
<li><a href="https://www.adafruit.com/product/1209">ESD-Safe PCB Cleaning Brush</a>, $2.95</li>
<li><a href="https://www.adafruit.com/product/148">Solder sucker</a>, $5.00</li>
<li><a href="https://www.adafruit.com/product/1598">Professional IC Extraction Tool</a>, $14.95</li>
<li><a href="https://www.adafruit.com/product/239">Full sized breadboard</a>, $5.95</li>
<li><a href="https://www.adafruit.com/product/153">Breadboarding wire bundle</a>, $4.95</li>
<li><a href="https://www.adafruit.com/product/1008">Small Alligator Clip Test Lead (set of 12)</a>, $3.95</li>
<li><a href="https://www.adafruit.com/product/1328">2.1mm DC Barrel Jack to Alligator Clips</a>, $1.95</li>
<li><a href="https://www.adafruit.com/product/1125">In-line power switch for 2.1mm barrel jack</a>, $2.50</li>
<li><a href="https://www.adafruit.com/product/276">5V 2A (2000mA) switching power supply</a>, $7.95</li>
<li><a href="https://www.adafruit.com/product/63">9 VDC 1000mA regulated switching power adapter</a>, $6.95</li>
<li><a href="https://www.adafruit.com/product/308">Extech EX330 12-function autoranging multimeter</a>, $59.95</li>
<li><a href="https://amzn.to/2Ew5MBM">Rigol DS1054Z Digital Oscilloscope - Bandwidth: 50 Mhz, Channels: 4</a>, $349.00 (this is an Amazon link, since the 4 channel scope is less than the Adafruit 2 channel Rigol. Sorry, Limor!)</li>
<li><a href="http://saleae-usd.myshopify.com/?rfsn=1807544.21a5e9">Saleae Logic 8 Logic Analyzer, 8 D/A Inputs, 100 MS/s</a>, $199 with discount code that <a href="https://blog.saleae.com/saleae-discounts/">you can request</a> and apply to your cart when checking out (this is a direct Saleae link, since they offer an awesome "enthusiast/student" discount of $200 off through their site; thanks, guys!)</li>
</ul>
<div>
Consumable supplies:</div>
<div>
<ul>
<li><a href="https://www.adafruit.com/product/2670">Bakelite Universal Perfboard Plates - Pack of 10</a>, $4.95</li>
<li><a href="https://www.adafruit.com/product/3174">Hook-up Wire Spool Set - 22AWG Solid Core - 10 x 25ft</a>, $27.50</li>
<li><a href="https://www.adafruit.com/product/3175">Hook-up Wire Spool Set - 22AWG Stranded-Core - 10 x 25ft</a>, $27.50</li>
<li><a href="https://www.adafruit.com/product/145">Mini Solder spool - 60/40 lead rosin-core solder 0.031" diameter - 100g</a>, $7.95</li>
<li><a href="https://www.adafruit.com/product/1886">Solder Wire - 60/40 Rosin Core - 0.5mm/0.02" diameter - 50 grams</a>, $5.95</li>
<li><a href="https://www.adafruit.com/product/2473">Solder Wire - RoHS Lead Free - 0.5mm/.02" diameter - 50g</a>, $11.95</li>
<li><a href="https://www.adafruit.com/product/1930">Solder Wire - SAC305 RoHS Lead Free - 0.5mm/.02" diameter - 50g</a>, $14.95</li>
<li><a href="https://www.adafruit.com/product/2660">Chip Quik SMD Removal Kit</a>, $16.00</li>
<li><a href="https://www.adafruit.com/product/2668">Chip Quik SMD Removal Kit with Lead-Free Alloy</a>, $17.00</li>
<li><a href="https://www.adafruit.com/product/149">Solder wick - 3S 5ft.</a>, $3.00</li>
<li><a href="https://www.adafruit.com/product/344">Heat Shrink Pack (24 pieces)</a>, $4.95</li>
<li><a href="https://www.adafruit.com/product/1649">Multi-Colored Heat Shrink Pack (30 pieces)</a>, $4.95</li>
<li><a href="https://www.adafruit.com/product/373">Breadboard-friendly 2.1mm DC barrel jack</a>, $0.95</li>
<li><a href="https://www.adafruit.com/product/2975">Adafruit Parts Pal</a>, $19.95</li>
</ul>
</div>
<div>
Books:</div>
<ul>
<li><a href="https://www.adafruit.com/product/203">Make: Electronics (Charles Platt) - 2nd Edition</a>, $34.95 (spoiler alert: this is a fantastic book for beginners and those with a little knowledge!)</li>
<li><a href="https://www.adafruit.com/product/1939">Make: More Electronics by Charles Platt</a>, $39.95</li>
<li><a href="https://www.adafruit.com/product/517">Getting Started in Electronics by Forrest M. Mims III</a>, $19.95 (remember him?)</li>
<li><a href="https://www.adafruit.com/product/1261">Practical Electronics for Inventors, Fourth Edition, by Paul Scherz and Simon Monk</a>, $40.00</li>
<li><a href="https://www.adafruit.com/product/1339">Hacking Electronics by Simon Monk</a>, $29.95</li>
<li><a href="https://www.adafruit.com/product/3638">Learn Electronics with Arduino - by Jody Culkin and Eric Hagan</a>, $24.95</li>
<li><a href="https://www.adafruit.com/product/3325">Python for Microcontrollers: Getting Started with MicroPython by Donald Norris</a>, $19.95</li>
<li><a href="https://www.adafruit.com/product/1089">Programming the Raspberry Pi: Getting Started with Python - 2nd Edition by Simon Monk</a>, $15.00</li>
</ul>
Finally, here are some additional random useful items that they don't carry, all via Amazon:<br />
<ul>
<li><a href="https://amzn.to/2EC4IMC">StarTech.com 24x27.5-Inch Desktop Anti-Static Mat</a>, $17.44. Anti-static mats are important for ESD safety, to avoid damaging sensitive components. Use this larger one as you primary work surface.</li>
<li><a href="https://amzn.to/2qnXfM5">Velleman AS4 Anti-Static Mat with Ground Cable - Desktop static dissipative mat - 11.8" x 22"</a>, $11.18. Smaller secondary mat for second work area.</li>
<li><a href="https://amzn.to/2JAWIiP">Rosewill ESD Anti-Static Wrist Strap Components</a>, $5.99, one per mat.</li>
<li><a href="https://amzn.to/2HEF64j">Elenco Electronics LP-560 Logic Probe</a>, $18.00. For checking logic levels on IC pins, including catching quick pulses that you would miss using a meter.</li>
<li><a href="https://amzn.to/2jgnBwg">Silvertronic 501784CS Solid Copper Alligator Clip w/Barrel (10 pieces)</a>, $19.99. Because copper absorbs heat well, these are used as heat sinks on component leads when soldering to avoid heat damage.</li>
<li><a href="https://amzn.to/2FoHpGt">Elenco Electronics TL-21 Minigrabber to Minigrabber 5 pc Test Lead Set</a>, $8.95. Minigrabbers are good for grabbing onto closely-spaced IC and header pins without shorting to adjacent pins.</li>
<li><a href="https://amzn.to/2EAKMtJ">3M Scotch #35 Electrical Tape Value Pack (5 colors)</a>, $10.10, because how can you build electrical stuff without electrical tape, color coded?</li>
<li><a href="https://amzn.to/2IHG13Y">Scotch Super 33+ Vinyl Electrical Tape (black)</a>, $3.98</li>
<li><a href="https://amzn.to/2EBSsvv">Sharpie Fine Point Asst Colors (8 colors)</a>, $6.30, for color coded marking.</li>
<li><a href="https://amzn.to/2HqzWt9">Permatex 80050 Clear RTV Silicone Adhesive Sealant, 3 oz</a>, $11.18. RTV is a universal technician's friend. Gobs of it serve as adhesive, sealers, hole plugs, gaskets, wire holders, vibration dampers, etc. There are a number of different formulations.</li>
</ul>
Total cost: $1567 for everything (I ordered 2 spools leaded solder and 1 leaded Chip Quik, no lead-free items, 10 DC barrel jacks, and all the screwdriver/tool sets, since you never know which tips and shanks will fit, and some cases need special access tools to open), with free shipping from both Adafruit and Amazon, $10 for Saleae. Plus Adafruit threw in a free half-size breadboard and a Circuit Playground Express.<br />
<br />
Back in my teenage days, $10 was a major expenditure, and $100 was simply inconcievable. This is starting to add up to some real money, but it will leave you armed with the tools, knowledge, and skills sufficient to launch a career.<br />
<br />
The really nice thing is that Adafruit provides a curated list of things to choose from, so you're getting the benefit of their experience and recommendations, all guided by that maker ethos. That was a big plus for me.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBzC9dv7abvQPPvYJxK9iapHQhcwkx5F5YzNVSWYUE4YkWgcpIVdEcCierg0Fbw6aVlSkeuGZSN28L5UTFKWxthHab25DAJQbg8SyHVkQ5Z0dZE8zyShwhq8YzyZjd8XjmOGwK54o_NBSc/s1600/Electronics+Workshop.png" imageanchor="1"><img border="0" data-original-height="1146" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBzC9dv7abvQPPvYJxK9iapHQhcwkx5F5YzNVSWYUE4YkWgcpIVdEcCierg0Fbw6aVlSkeuGZSN28L5UTFKWxthHab25DAJQbg8SyHVkQ5Z0dZE8zyShwhq8YzyZjd8XjmOGwK54o_NBSc/s640/Electronics+Workshop.png" width="560" /></a>
<br />
<i><span style="color: #38761d;">Bridging three centuries of maker technology in my workshop.</span></i><br />
<br />
You can read about my <a href="http://flinkandblink.blogspot.com/2018/04/first-use-of-new-tools.html">first use of these tools</a>, since I needed them almost immediately.<br />
<br />
For a review of the outstanding Charles Platt books listed above, see <a href="http://flinkandblink.blogspot.com/2018/05/review-make-electronics-and-makemore.html">Review: Make: Electronics and Make:More Electronics</a>.<br />
<div>
<br /></div>
<div>
For a useful set of resources to help you learn electronics, see <a href="http://flinkandblink.blogspot.com/2018/09/learning-about-electronics-and.html">Learning About Electronics And Microcontrollers</a>.</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-19964366559878521622018-03-28T20:32:00.000-07:002018-04-20T06:12:46.832-07:00The Case For C+-No, that's not a typo. I really do mean C+-.<br />
<br />
When using C++, there's a tendency for C programmers to think they have to use all the facilities of the language at once. Particularly user-defined classes, since C++ takes C into object-oriented programming. "If I'm using C++, I have to define some classes."<br />
<br />
The danger in that is a risk of over-engineering, over-complicating things, by forcibly looking for ways to use classes when there may not be a need for that.<br />
<br />
In a large, complex piece of software, there are many places that benefit from user-defined classes. In a bigger beast like that, the user-defined object-oriented approach helps abstract the problem.<br />
<br />
But in small, quick tools, that's not necessarily the case, so plain C with a few simple structs is probably sufficient. However, there's still a lot of value to be found in the C++ standard library.<br />
<br />
Specifically, the managed string and container classes. One of the big complaints about C is the need for explicit memory management. Because of that need, the C language and runtime library don't offer any native containers other than the statically-sized array, with the simple character array as an implementation of strings. If you need any dynamic structures, you have to implement your own, with explicit memory management on top of <i>malloc()</i> and <i>free()</i>.<br />
<br />
There are no native or standard library lists, hash tables, trees, or other dynamic structures. There are no dynamically-sized arrays or strings. There's no automatic deallocation of heap when you're finished using it.<br />
<br />
As a result, C usage has been plagued by decades of buffer overflows and memory leaks. It also means a lot of time required to roll your own basic dynamic structures (and iron out the buffer overflows and memory leaks in their implementation).<br />
<br />
But the C++ standard library provides all of those things. It also provides building blocks that can be used to layer more complex structure on them. That provides a lot of opportunities to build things without having to define any classes of your own.<br />
<br />
I'm not saying there's anything wrong with classes. I'm just saying there's a whole class of programs that don't need any extra classes. The C++ standard library already provides a rich set of resources to choose from, often sufficient on their own to build useful programs that are faster to implement and debug than if you did everything in C.<br />
<br />
You probably already treat templates that way. Even though C++ offers the ability to define templates, you may write tons of code without ever defining your own templates. Just because the language offers a feature doesn't mean you have to define any of your own things with it. Yet you still probably make extensive of templates through the library.<br />
<br />
And so it is with classes. You can write tons of code without ever defining your own classes. Doing lots of string processing, common in software tools? The C++ standard library provides a whole host of classes that will help, starting with <i>std::string</i>.<br />
<br />
Need to keep lists of those strings, in the order you got them? How about a <i>std::list<std::string></i>? Need fast associative storage keyed off the string, or a portion of it? How about a <i>std::unordered_map<std::string, yourThingHere></i>? Need to keep a set of sorted strings? How about a <i>std::map<yourThingHere, std::string></i>? Or something sorted by the strings? How about a <i>std::map<std::string, yourThingHere></i>?<br />
<br />
Then if you need something a little more complex than simple strings and structs, you can use <i>std::pair<thingA, thingB></i> or <i>std::bind<callableThing, args></i>.<br />
<br />
The other benefit to this is that at some point you may realize that perhaps there are some user-defined classes that would make sense in your progam after all, it's not just strings and structs and pairs and binds. The infrastructure you've already built into the program is OO-ready. And you have <i>std::shared_ptr<yourClassHere></i> to automate memory management and support RAII, avoiding memory leaks.<br />
<br />
So making the switch to a more heavily object-oriented program is a small step, a refinement, rather than throwing it all out and starting over again.<br />
<br />
Meanwhile, you're already in the mindset of using just the minimum of appropriate user-defined classes, and not going overboard trying to beat everything into the shape of an OO nail just because you have an OO hammer.<br />
<br />
That's adding just the amount of design and implementation complexity necessary to help abstract the appropriate parts of the problem, while maintaining a simple, pared-down elegance. Make things only as complex as you need to, and no more (as well as following Einstein's advice to make things as simple as possible, but no simpler).<br />
<br />
Meanwhile, you're relying on a large body of fully-implemented and debugged composable, modular elements to speed the job to completion. In many ways, that right there is going a long way to meeting the promise of the "software IC".<br />
<br />
So that's what I'm calling C+-. It's C++ minus the user-defined classes. Which is more than just writing plain C that you compile with the C++ compiler. It's simply object-oriented code that relies entirely on someone else's classes.<br />
<br />
You can argue about whether that's a good thing or a bad thing in the grand scheme of things, but I see it as just another practical tool in your toolbox.<br />
<br />
There are three situations where this approach is useful:<br />
<ul>
<li>Quick tools where you need to get it done as fast as possible so you can use it to help you get on with your main work.</li>
<li>Competitive programming, where you're working under the gun.</li>
<li>Coding interviews, which are essentially competitive programming under a time limit, whether on a whiteboard, in a shared editing session, or in an automated coding assessment system.</li>
</ul>
As an example of this, here's a tool I've been wanting to have for a while. I work on IOT projects, distributed systems where small embedded system client devices communicate with large backend servers.<br />
<br />
Debugging these can be challenging as you try to sift through the logs each side produces. Because many IOT systems lack real-time clocks, they may not know what actual time it is, so it's hard to match up activity in the client log with the activity in the server, especially when there are communication errors and voluminous logs.<br />
<br />
The tool below, msgresolve, resolves the messages logged by a client IOT device and its server. The client tracks time since booted, in msec, and logs that timestamp on each line. The server tracks real time GMT to msec resolution, logging the data and time on each line.<br />
<br />
The example logs here contain a very small amount of data, but it's not unreasonable for a log to have hundreds or thousands of messages.<br />
<br />
In order for this to work, the message logging must have a way of identifying each message uniquely. This is known as the message signature, a short string that summarizes the message contents. The signature may be a cryptographic hash or message digest such as MD5, or a checksum or polynomial such as Fletcher or CRC.<br />
<br />
The messages must have some degree of randomization in the the contents so that no two messages in the same direction every produce the same signature (at least for the duration of the logging). This randomization might be due to encryption, some incrementing field such as a timestamp or counter, or a randomized nonce.<br />
<br />
Users of git will be familiar with this concept. The commit hash acts as the identifier for changes to file content, and is affected by only a single-byte change in the file contents.<br />
<br />
Here the signature is formed from the message hash and the message length. Appending the length adds a little insurance in case messages of different lengths, with different contents, hash to the same value, known as a hash collision. Two messages of the same length should always hash to different values if at least one bit is different in them, so the hash conditioned by the length ensures a unique signature.<br />
<br />
I had a couple thoughts on how to approach the algorithm. One was to treat it as a difference-matching problem, such as the <a href="https://en.wikipedia.org/wiki/Diff_utility">Unix diff utility</a>. The other was a kind of match-and-merge approach. However that seemed like it might head toward an O(N^2) algorithm (for each client message, run down the list of server messages to find a match), which would rapidly get too slow for large logs.<br />
<br />
But that made me think about an indexed lookup method, where a faster lookup method would make that approach manageable.<br />
<br />
Part of what made it tricky is the fact that even though the two logs have parallel, time-ordered sets of messages, there might be lost or corrupted messages, and the two logs might not cover the exact same range of time. So just because a message appeared in one log, there was no guarantee that it opposite appeared exactly as is in the other log.<br />
<br />
The other thing that helped crystallize it was the realization that matching up a set of parallel ordered log entries could be viewed as three parts from the perspective of the client messages:<br />
<ul>
<li>Handle any messages in the server log that preceded the messages matching the client messages.</li>
<li>Handle all the messages in the client log, which may or may not have matching server messages (along with intervening server messages that didn't have any matching client messages).</li>
<li>Handle any messages in the server log that followed the messages matching the client messages.</li>
</ul>
So this algorithm uses a hash table (<i>std::unordered_map</i>) to index a list (<i>std::list</i>) of log entries. The hash table (which I call a dict, as in a Python dict) is indexed by message signature. Ideally, for every transmitted message, there is a received message with matching signature. That's the basis of the lookup. Iterating linearly through the time-ordered list deals with the unmatched server messages. Reordered messages can produce some interesting results.<br />
<br />
For every message, lookup the signature in the other side's dict to find its matching message. That makes it an O(N) algorithm (the hash lookup done for each message is O(1)).<br />
<br />
I did have to separate transmit from receive messages for each side, since it's possible for a received message to have the same signature as a transmitted message if all the randomizing factors are the same in both directions. Thus the signature on a client TX message would be used to lookup the corresponding message in the server RX message dict.<br />
<br />
The actual string storage for the log lines for each side is in the list, which is a time-ordered list. The dict entries contain references to those strings, so a dict is simply the index, by signature, of the list of strings.<br />
<br />
All of this can be managed with standard library objects, using <i>std::pairs</i> to bind cross reference information with the log strings. For simple composition, this works well. As you need to compose more complex objects, navigating pairs of pairs rapidly gets out of hand, so that's when to define some structs, or maybe some simple data classes.<br />
<br />
The other thing that was very useful was to define a <i>split()</i> function, equivalent to the <i>split()</i> function in Python. I use <i>split()</i> and <i>join()</i> quite a bit in Python for similar text processing tools. They really speed up string processing, allowing you to tear apart and reassemble strings easily. That also crosses the C/C++ string boundary: <i>split()</i> takes a C-style character array and splits it into a vector of strings (<i>std::vector<std::string></i>).<br />
<br />
I used a number of typedefs of the standard objects as syntactic sugar. That's a big help when declaring an iterator for an unordered map of composed pairs.<br />
<br />
With the <i>split()</i> function and the typedefs of the standard objects acting as power tools, the code was straightforward.<br />
<br />
The resulting output of the tool makes it easy to navigate the logs and correlate activity. One useful modification would be to have it group all the other non-message logs line with the nearest message (though that brings up the problem of deciding whether the lines should be grouped with the nearest subsequent message, or the nearest previous message). That would be especially useful behind a GUI like tkdiff (see, there's that diff thinking again...).<br />
<br />
For another example of code like this, see <a href="http://flinkandblink.blogspot.com/2018/04/more-c.html">More C+-</a>.<br />
<br />
The source, <i>msgresolve.cpp</i> (I had to do a little odd line-folding to make it fit in the width below):<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-style: italic;">// Usage: msgresolve <clientLog> <serverLog></span>
<span style="color: #008800; font-style: italic;">//</span>
<span style="color: #008800; font-style: italic;">// Resolve client/server logs from the client perspective. That</span>
<span style="color: #008800; font-style: italic;">// treats the total sequence of messages as 3 sections:</span>
<span style="color: #008800; font-style: italic;">// 1) Initial unmatched server messages.</span>
<span style="color: #008800; font-style: italic;">// 2) Client messages that may be matched or unmatched,</span>
<span style="color: #008800; font-style: italic;">// interspersed with unmatched server messages.</span>
<span style="color: #008800; font-style: italic;">// 3) Remaining unmatched server messages.</span>
<span style="color: #008800; font-style: italic;">//</span>
<span style="color: #008800; font-style: italic;">// This is an example of a C++ program that is written mostly</span>
<span style="color: #008800; font-style: italic;">// in plain C style, but that makes use of the container and</span>
<span style="color: #008800; font-style: italic;">// composition classes in the C++ standard library. It is a</span>
<span style="color: #008800; font-style: italic;">// lightweight use of C++ with no user-defined classes.</span>
<span style="color: #008800; font-style: italic;">//</span>
<span style="color: #008800; font-style: italic;">// 2018 Steve Branam <sdbranam@gmail.com> learntocode</span>
<span style="color: #008800;">#include <iostream></span>
<span style="color: #008800;">#include <vector></span>
<span style="color: #008800;">#include <list></span>
<span style="color: #008800;">#include <unordered_map></span>
<span style="color: #008800;">#define SERVER_PREFIX " "</span>
<span style="color: #aa22ff; font-weight: bold;">enum</span> ARGS
{
ARGS_PROGNAME,
ARGS_CLIENT_LOG,
ARGS_SERVER_LOG,
ARGS_REQUIRED
};
<span style="color: #aa22ff; font-weight: bold;">enum</span> CLIENT
{
CLIENT_TIMESTAMP,
CLIENT_FILE,
CLIENT_LINE,
CLIENT_SEVERITY,
CLIENT_DIRECTION,
CLIENT_HASH_KEYWORD,
CLIENT_HASH,
CLIENT_LEN,
CLIENT_BYTES_KEYWORD,
CLIENT_TIMESTAMP_LEN <span style="color: #666666;">=</span> <span style="color: #666666;">10</span>
};
<span style="color: #aa22ff; font-weight: bold;">enum</span> SERVER
{
SERVER_DATE,
SERVER_TIME,
SERVER_THREAD,
SERVER_SEVERITY,
SERVER_FUNC,
SERVER_CLIENT,
SERVER_DIRECTION,
SERVER_HASH_KEYWORD,
SERVER_HASH,
SERVER_LEN,
SERVER_BYTES_KEYWORD,
SERVER_TIME_LEN <span style="color: #666666;">=</span> <span style="color: #666666;">16</span>
};
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>string String;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>vector<span style="color: #666666;"><</span>String<span style="color: #666666;">></span> StringVec;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>pair<span style="color: #666666;"><</span>String, String<span style="color: #666666;">></span> StringPair;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>list<span style="color: #666666;"><</span>StringPair<span style="color: #666666;">></span> MsgList;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>unordered_map<span style="color: #666666;"><</span>String, String<span style="color: #666666;">&></span> MsgDict;
<span style="color: #aa22ff; font-weight: bold;">typedef</span> std<span style="color: #666666;">::</span>pair<span style="color: #666666;"><</span>String, String<span style="color: #666666;">&></span> MsgDictEntry;
MsgList clientTimestamps;
MsgDict clientReceives;
MsgDict clientTransmits;
MsgList serverTimestamps;
MsgDict serverReceives;
MsgDict serverTransmits;
StringVec <span style="color: #00a000;">split</span>(<span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> str, <span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> delim)
{
StringVec strings;
<span style="color: #00bb00; font-weight: bold;">char</span> <span style="color: #666666;">*</span>token <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>strtok(str, delim);
<span style="color: #aa22ff; font-weight: bold;">while</span> (token <span style="color: #666666;">!=</span> <span style="color: #aa22ff;">NULL</span>) {
strings.push_back(token);
token <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>strtok(<span style="color: #aa22ff;">NULL</span>, delim);
}
<span style="color: #aa22ff; font-weight: bold;">return</span> strings;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isClientTimestamp</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (str.size() <span style="color: #666666;">==</span> CLIENT_TIMESTAMP_LEN) {
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">int</span> x <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; x <span style="color: #666666;"><</span> str.size(); <span style="color: #666666;">++</span>x)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>isdigit(str[x])) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isServerTime</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> String<span style="color: #666666;">&</span> str)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (str.size() <span style="color: #666666;">==</span> SERVER_TIME_LEN) {
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">int</span> x <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; x <span style="color: #666666;"><</span> str.size(); <span style="color: #666666;">++</span>x)
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>isdigit(str[x]) <span style="color: #666666;">&&</span>
(str[x] <span style="color: #666666;">!=</span> <span style="color: #bb4444;">':'</span>) <span style="color: #666666;">&&</span>
(str[x] <span style="color: #666666;">!=</span> <span style="color: #bb4444;">'.'</span>) <span style="color: #666666;">&&</span>
(str[x] <span style="color: #666666;">!=</span> <span style="color: #bb4444;">']'</span>)) {
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isClientRxTx</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> fields)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> ((fields.size() <span style="color: #666666;">></span> CLIENT_BYTES_KEYWORD) <span style="color: #666666;">&&</span>
isClientTimestamp(fields[CLIENT_TIMESTAMP]) <span style="color: #666666;">&&</span>
(fields[CLIENT_DIRECTION] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"RX"</span> <span style="color: #666666;">||</span>
fields[CLIENT_DIRECTION] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"TX"</span>) <span style="color: #666666;">&&</span>
(fields[CLIENT_HASH_KEYWORD] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"hash"</span>) <span style="color: #666666;">&&</span>
(fields[CLIENT_BYTES_KEYWORD] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"bytes</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span> <span style="color: #666666;">||</span>
fields[CLIENT_BYTES_KEYWORD] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"bytes,"</span>));
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">isServerRxTx</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> StringVec<span style="color: #666666;">&</span> fields)
{
<span style="color: #aa22ff; font-weight: bold;">return</span> ((fields.size() <span style="color: #666666;">></span> SERVER_BYTES_KEYWORD) <span style="color: #666666;">&&</span>
isServerTime(fields[SERVER_TIME]) <span style="color: #666666;">&&</span>
(fields[SERVER_DIRECTION] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"RX"</span> <span style="color: #666666;">||</span>
fields[SERVER_DIRECTION] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"TX"</span>) <span style="color: #666666;">&&</span>
(fields[SERVER_HASH_KEYWORD] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"hash"</span>) <span style="color: #666666;">&&</span>
(fields[SERVER_BYTES_KEYWORD] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"bytes</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span> <span style="color: #666666;">||</span>
fields[SERVER_BYTES_KEYWORD] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"bytes,"</span>));
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">loadClient</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> fileName)
{
<span style="color: #00bb00; font-weight: bold;">FILE</span><span style="color: #666666;">*</span> file <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>fopen(fileName, <span style="color: #bb4444;">"r"</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (file) {
<span style="color: #00bb00; font-weight: bold;">char</span> buffer[<span style="color: #666666;">1000</span>];
<span style="color: #aa22ff; font-weight: bold;">while</span> (std<span style="color: #666666;">::</span>fgets(buffer, <span style="color: #aa22ff; font-weight: bold;">sizeof</span>(buffer), file) <span style="color: #666666;">!=</span> <span style="color: #aa22ff;">NULL</span>) {
String line(buffer);
StringVec fields <span style="color: #666666;">=</span> split(buffer, <span style="color: #bb4444;">" "</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (isClientRxTx(fields)) {
<span style="color: #008800; font-style: italic;">// Remove trailing comma.</span>
fields[CLIENT_HASH].pop_back();
String key(fields[CLIENT_HASH]);
key.append(fields[CLIENT_LEN]);
String xref(fields[CLIENT_DIRECTION]);
xref.append(key);
clientTimestamps.push_back(StringPair(xref, line));
<span style="color: #aa22ff; font-weight: bold;">if</span> (fields[CLIENT_DIRECTION] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"RX"</span>) {
clientReceives.insert(MsgDictEntry(key,
clientTimestamps.back().second));
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
clientTransmits.insert(MsgDictEntry(key,
clientTimestamps.back().second));
}
}
}
std<span style="color: #666666;">::</span>fclose(file);
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Failed to open client file "</span>
<span style="color: #666666;"><<</span> fileName <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #00bb00; font-weight: bold;">bool</span> <span style="color: #00a000;">loadServer</span>(<span style="color: #aa22ff; font-weight: bold;">const</span> <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> fileName)
{
<span style="color: #00bb00; font-weight: bold;">FILE</span><span style="color: #666666;">*</span> file <span style="color: #666666;">=</span> std<span style="color: #666666;">::</span>fopen(fileName, <span style="color: #bb4444;">"r"</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (file) {
<span style="color: #00bb00; font-weight: bold;">char</span> buffer[<span style="color: #666666;">1000</span>];
<span style="color: #aa22ff; font-weight: bold;">while</span> (std<span style="color: #666666;">::</span>fgets(buffer, <span style="color: #aa22ff; font-weight: bold;">sizeof</span>(buffer), file) <span style="color: #666666;">!=</span> <span style="color: #aa22ff;">NULL</span>) {
String line(buffer);
StringVec fields <span style="color: #666666;">=</span> split(buffer, <span style="color: #bb4444;">" "</span>);
<span style="color: #aa22ff; font-weight: bold;">if</span> (isServerRxTx(fields)) {
<span style="color: #008800; font-style: italic;">// Remove trailing comma.</span>
fields[SERVER_HASH].pop_back();
String key(fields[SERVER_HASH]);
key.append(fields[SERVER_LEN]);
String xref(fields[SERVER_DIRECTION]);
xref.append(key);
serverTimestamps.push_back(StringPair(xref, line));
<span style="color: #aa22ff; font-weight: bold;">if</span> (fields[SERVER_DIRECTION] <span style="color: #666666;">==</span> <span style="color: #bb4444;">"RX"</span>) {
serverReceives.insert(MsgDictEntry(key,
serverTimestamps.back().second));
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
serverTransmits.insert(MsgDictEntry(key,
serverTimestamps.back().second));
}
}
}
std<span style="color: #666666;">::</span>fclose(file);
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">true</span>;
}
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Failed to open server file"</span>
<span style="color: #666666;"><<</span> fileName <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff;">false</span>;
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">printRxSeparator</span>()
{
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" /"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" <"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">printTxSeparator</span>()
{
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">" </span><span style="color: #bb6622; font-weight: bold;">\\</span><span style="color: #bb4444;">"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" >"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">printTransactionSeparator</span>()
{
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"---------"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl
<span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
<span style="color: #008800; font-style: italic;">// Find next server match for client processing, processing any</span>
<span style="color: #008800; font-style: italic;">// unmatched server messages along the way.</span>
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">findNextServerMatch</span>(MsgList<span style="color: #666666;">::</span>iterator<span style="color: #666666;">&</span> curServer)
{
<span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">bool</span> found <span style="color: #666666;">=</span> <span style="color: #aa22ff;">false</span>;
<span style="color: #666666;">!</span>found <span style="color: #666666;">&&</span> curServer <span style="color: #666666;">!=</span> serverTimestamps.end();) {
std<span style="color: #666666;">::</span>string<span style="color: #666666;">&</span> xref(curServer<span style="color: #666666;">-></span>first);
std<span style="color: #666666;">::</span>string key(xref.substr(<span style="color: #666666;">2</span>));
std<span style="color: #666666;">::</span>string<span style="color: #666666;">&</span> server(curServer<span style="color: #666666;">-></span>second);
<span style="color: #aa22ff; font-weight: bold;">if</span> (xref[<span style="color: #666666;">0</span>] <span style="color: #666666;">==</span> <span style="color: #bb4444;">'R'</span>) {
found <span style="color: #666666;">=</span> (clientTransmits.find(key) <span style="color: #666666;">!=</span>
clientTransmits.end());
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>found) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Client transmit not found"</span>
<span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
printTxSeparator();
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> SERVER_PREFIX <span style="color: #666666;"><<</span> server;
}
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
found <span style="color: #666666;">=</span> (clientReceives.find(key) <span style="color: #666666;">!=</span>
clientReceives.end());
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>found) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> SERVER_PREFIX <span style="color: #666666;"><<</span> server;
printRxSeparator();
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Client receive not found"</span>
<span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
}
}
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: #666666;">!</span>found) {
printTransactionSeparator();
curServer<span style="color: #666666;">++</span>;
}
}
}
<span style="color: #008800; font-style: italic;">// Process all client messages, checking for unmatched server</span>
<span style="color: #008800; font-style: italic;">// messages along the way.</span>
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">processClient</span>(MsgList<span style="color: #666666;">::</span>iterator<span style="color: #666666;">&</span> curServer)
{
<span style="color: #aa22ff; font-weight: bold;">for</span> (MsgList<span style="color: #666666;">::</span>iterator curClient <span style="color: #666666;">=</span> clientTimestamps.begin();
curClient <span style="color: #666666;">!=</span> clientTimestamps.end();
curClient<span style="color: #666666;">++</span>) {
std<span style="color: #666666;">::</span>string<span style="color: #666666;">&</span> xref(curClient<span style="color: #666666;">-></span>first);
std<span style="color: #666666;">::</span>string key(xref.substr(<span style="color: #666666;">2</span>));
std<span style="color: #666666;">::</span>string<span style="color: #666666;">&</span> client(curClient<span style="color: #666666;">-></span>second);
MsgDict<span style="color: #666666;">::</span>iterator match;
<span style="color: #aa22ff; font-weight: bold;">if</span> (xref[<span style="color: #666666;">0</span>] <span style="color: #666666;">==</span> <span style="color: #bb4444;">'R'</span>) {
match <span style="color: #666666;">=</span> serverTransmits.find(key);
<span style="color: #aa22ff; font-weight: bold;">if</span> (match <span style="color: #666666;">==</span> serverTransmits.end()) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> SERVER_PREFIX
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Server transmit not found"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> SERVER_PREFIX <span style="color: #666666;"><<</span> match<span style="color: #666666;">-></span>second;
}
printRxSeparator();
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> client;
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> client;
printTxSeparator();
match <span style="color: #666666;">=</span> serverReceives.find(key);
<span style="color: #aa22ff; font-weight: bold;">if</span> (match <span style="color: #666666;">==</span> serverReceives.end()) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> SERVER_PREFIX
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Server receive not found"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> SERVER_PREFIX <span style="color: #666666;"><<</span> match<span style="color: #666666;">-></span>second;
}
}
printTransactionSeparator();
<span style="color: #aa22ff; font-weight: bold;">if</span> (match <span style="color: #666666;">!=</span> serverReceives.end()) {
<span style="color: #008800; font-style: italic;">// Matched, advance server iterator and find next</span>
<span style="color: #008800; font-style: italic;">// matching server msg.</span>
findNextServerMatch(<span style="color: #666666;">++</span>curServer);
}
}
}
<span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">resolve</span>()
{
MsgList<span style="color: #666666;">::</span>iterator curServer <span style="color: #666666;">=</span> serverTimestamps.begin();
<span style="color: #008800; font-style: italic;">// Handle any initial unmatched server messages.</span>
findNextServerMatch(curServer);
<span style="color: #008800; font-style: italic;">// Handle client messages interspersed with any unmatched</span>
<span style="color: #008800; font-style: italic;">// server messages.</span>
processClient(curServer);
<span style="color: #008800; font-style: italic;">// Handle any remaining unmatched server messages.</span>
findNextServerMatch(curServer);
}
<span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">main</span>(<span style="color: #00bb00; font-weight: bold;">int</span> argc, <span style="color: #00bb00; font-weight: bold;">char</span><span style="color: #666666;">*</span> argv[])
{
<span style="color: #aa22ff; font-weight: bold;">if</span> (argc <span style="color: #666666;"><</span> ARGS_REQUIRED <span style="color: #666666;">||</span>
String(argv[<span style="color: #666666;">1</span>]) <span style="color: #666666;">==</span> <span style="color: #bb4444;">"-h"</span>) {
std<span style="color: #666666;">::</span>cout <span style="color: #666666;"><<</span> <span style="color: #bb4444;">"Usage: "</span> <span style="color: #666666;"><<</span> argv[ARGS_PROGNAME]
<span style="color: #666666;"><<</span> <span style="color: #bb4444;">" <clientLog> <serverLog>"</span> <span style="color: #666666;"><<</span> std<span style="color: #666666;">::</span>endl;
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">if</span> (loadClient(argv[ARGS_CLIENT_LOG]) <span style="color: #666666;">&&</span>
loadServer(argv[ARGS_SERVER_LOG])) {
resolve();
} <span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_SUCCESS;
}
</pre>
</td></tr>
</tbody></table>
</div>
<br />
Sample client log (the 0x11111111 hashes are ones I deliberately changed to break the match):
<!-- HTML generated using hilite.me --><br />
<br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7</pre>
</td><td><pre style="line-height: 125%; margin: 0;">0345604820 comm.c, 1529, D: TX hash 0x47e21fdd, 185 bytes, msg type 3
0345605799 comm.c, 1426, D: RX hash 0xd331bb95, 35 bytes
0345605916 comm.c, 1529, D: TX hash 0x2f66bbd6, 180 bytes, msg type 15
0345606875 comm.c, 1426, D: RX hash 0x11111111, 28 bytes
0345607011 comm.c, 1529, D: TX hash 0x6924ebfd, 69 bytes, msg type 16
0345607146 comm.c, 1426, D: RX hash 0x183d710c, 33 bytes
0345607215 comm.c, 1529, D: TX hash 0x5c4b78f4, 504 bytes, msg type 18
</pre>
</td></tr>
</tbody></table>
</div>
<br />
Sample server log:
<!-- HTML generated using hilite.me --><br />
<span style="font-size: xx-small;"><br /></span>
<br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;"> 1
2
3
4
5
6
7
8
9
10
11
12</span></pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;">[2018-02-05 20:50:04.093798] [0x00007f3412dc8700] [debug] send_msg() 00000062 TX hash 0xfcf3f009, 33 bytes, msg type 19
[2018-02-05 20:50:04.101101] [0x00007f3412dc8700] [debug] send_msg() 00000062 TX hash 0xca5c8aea, 53 bytes, msg type 15
[2018-02-05 20:51:45.796547] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x47e21fdd, 185 bytes
[2018-02-05 20:51:45.812284] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0xd331bb95, 35 bytes, msg type 3
[2018-02-05 20:51:46.894310] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x2f66bbd6, 180 bytes
[2018-02-05 20:51:46.894661] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0x7495ff13, 29 bytes, msg type 17
[2018-02-05 20:51:46.894829] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0x183d710c, 33 bytes, msg type 19
[2018-02-05 20:51:46.903009] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0xc1575ef6, 53 bytes, msg type 15
[2018-02-05 20:51:47.894246] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x11111111, 68 bytes
[2018-02-05 20:51:48.732482] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x5c4b78f4, 504 bytes
[2018-02-05 20:52:39.990683] [0x00007f34125c7700] [debug] handle_msg() :00000062 RX hash 0x15667979, 185 bytes
[2018-02-05 20:52:39.999387] [0x00007f34125c7700] [debug] send_msg() :00000062 TX hash 0x3b1bf5ec, 35 bytes, msg type 3
</span></pre>
</td></tr>
</tbody></table>
</div>
<span style="font-size: xx-small;"><br /></span>
Sample output:
<!-- HTML generated using hilite.me --><br />
<span style="font-size: xx-small;"><br /></span>
<br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98</span></pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="font-size: xx-small;">$ ./msgresolve client.log server.log
[2018-02-05 20:50:04.093798] [0x00007f3412dc8700] [debug] send_msg() 00000062 TX hash 0xfcf3f009, 33 bytes, msg type 19
/
<
Client receive not found
---------
[2018-02-05 20:50:04.101101] [0x00007f3412dc8700] [debug] send_msg() 00000062 TX hash 0xca5c8aea, 53 bytes, msg type 15
/
<
Client receive not found
---------
0345604820 comm.c, 1529, D: TX hash 0x47e21fdd, 185 bytes, msg type 3
\
>
[2018-02-05 20:51:45.796547] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x47e21fdd, 185 bytes
---------
[2018-02-05 20:51:45.812284] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0xd331bb95, 35 bytes, msg type 3
/
<
0345605799 comm.c, 1426, D: RX hash 0xd331bb95, 35 bytes
---------
0345605916 comm.c, 1529, D: TX hash 0x2f66bbd6, 180 bytes, msg type 15
\
>
[2018-02-05 20:51:46.894310] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x2f66bbd6, 180 bytes
---------
[2018-02-05 20:51:46.894661] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0x7495ff13, 29 bytes, msg type 17
/
<
Client receive not found
---------
Server transmit not found
/
<
0345606875 comm.c, 1426, D: RX hash 0x11111111, 28 bytes
---------
0345607011 comm.c, 1529, D: TX hash 0x6924ebfd, 69 bytes, msg type 16
\
>
Server receive not found
---------
[2018-02-05 20:51:46.894829] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0x183d710c, 33 bytes, msg type 19
/
<
0345607146 comm.c, 1426, D: RX hash 0x183d710c, 33 bytes
---------
[2018-02-05 20:51:46.903009] [0x00007f34135c9700] [debug] send_msg() :00000062 TX hash 0xc1575ef6, 53 bytes, msg type 15
/
<
Client receive not found
---------
Client transmit not found
\
>
[2018-02-05 20:51:47.894246] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x11111111, 68 bytes
---------
0345607215 comm.c, 1529, D: TX hash 0x5c4b78f4, 504 bytes, msg type 18
\
>
[2018-02-05 20:51:48.732482] [0x00007f34135c9700] [debug] handle_msg() :00000062 RX hash 0x5c4b78f4, 504 bytes
---------
Client transmit not found
\
>
[2018-02-05 20:52:39.990683] [0x00007f34125c7700] [debug] handle_msg() :00000062 RX hash 0x15667979, 185 bytes
---------
[2018-02-05 20:52:39.999387] [0x00007f34125c7700] [debug] send_msg() :00000062 TX hash 0x3b1bf5ec, 35 bytes, msg type 3
/
<
Client receive not found
---------</span>
</pre>
</td></tr>
</tbody></table>
</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-35067342925579873052018-03-17T10:18:00.000-07:002019-11-26T04:34:00.780-08:00We Need To Build Security In<b>The Old Priorities</b><br />
<br />
For a long time, the basic priorities for software were:<br />
<ul>
<li><b>Functional: </b>does it work right?</li>
<li><b>Performance:</b> is it fast enough?</li>
</ul>
<br />
The first is obvious. If software doesn't work right, it isn't going to be useful. It has to be a correct design, correctly implemented.<br />
<br />
Once the first has been achieved, the second has become critically important. As systems scale up, performance has become an enormous driver. Do everything you can to make it fast, as long as it still works right (and there's an argument to be made for putting performance first, then make it work right subject to maintaining performance).<br />
<br />
This is often expressed as "first make it <i>work</i>, then make it <i>fast</i>".<br />
<br />
Failure to achieve either of these can mean failure in the marketplace.<br />
<br />
In a few cases, security was also a requirement. But often, it wasn't. Or it was a distant third priority or an afterthought, always on the losing side of compromises for the first two.<br />
<br />
The result is that we've sacrificed security on the three-legged altar of time to market, convenience, and performance. We've built everything with the assumption that everyone out there is well-behaved, using things only as they were intended to be used.<br />
<br />
I've got some bad news for you sunshine, there are bad people out there. They're all too happy to slip into our insecure systems and have their way. These are people who actively search out ways to abuse, confuse, and misuse systems for their own purposes.<br />
<br />
They have a variety of motivations and goals, with a variety resources at their disposal, from the single script kiddie just wanting to impress his friends to criminals, terrorists, and state-sponsored cyberespionage and cyberwarfare groups.<br />
<br />
The potential consequences of these attacks range from minor annoyances to financial disaster to service outages to outright physical destruction, ranging in scope from personal to national. They can ruin lives.<br />
<br />
We see real cases of this daily in the news, in data breaches; botnet recruiting (taking over legitimate machines for use as bots); DDOS attacks; identity theft; account takeovers; social media fake news, fake accounts, and fake followers; ATM and POS skimming, siphoning, and jackpotting; all manner of large and small financial attacks and scams; ransomware of critical data systems; industrial espionage; and other attacks and disruptions.<br />
<br />
Just Google any of those terms if you want some depressing reading. Every new technology just seems to bring a whole new raft of attack opportunities.<br />
<br />
At the risk of sounding overly alarmist, we've built an incredibly fragile house of cards, completely permeable to bad actors. The Big Bad Wolf doesn't even need to huff or puff. All he has to do is inhale to bring it down.<br />
<br />
It's equivalent to doing all your banking by storing your money in grocery bags outside your front door.<br />
<br />
And yet our lives increasingly depend on these systems. We've made ourselves completely vulnerable. We've left ourselves completely exposed.<br />
<br />
<b>Security Needs To Be The New Top Priority</b><br />
<br />
Especially with the adoption of ubiquitous network connectivity over the past decade, that needs to change. Security needs to be the primary requirement, and the other two need to compromise to support it:<br />
<ul>
<li><b>Security:</b> is it secure?</li>
<li><b>Functional:</b> does it work right?</li>
<li><b>Performance:</b> is it fast enough?</li>
</ul>
<br />
Now getting it to work right and performance need to be subject to security. Does it work right, and still maintain security? Do everything you can to make it fast, as long as it's still secure and still works right.<br />
<br />
First make it <i>secure</i>, then make it <i>work</i>, then make it <i>fast</i>. And make sure it <i>stays</i> secure.<br />
<br />
That means when making design and implementation decisions, they need to done in such a way as to favor security. There are choices and ways of doing things that lead to insecure software. Make the choices that lead to secure software.<br />
<br />
Security has really been a wholly overlooked critical segment of software engineering. In retrospect, that's irresponsible.<br />
<br />
In other types of engineering, safety is the analogous property. In automobile or aircraft design, safety is a critical area. Imagine what would happen to a car company that ignored safety.<br />
<br />
We need to add a fourth leg to that altar: security, time to market, convenience, and performance.<br />
<br />
<b>Build Security In</b><br />
<br />
Here I'm adopting <a href="https://www.garymcgraw.com/">Gary McGraw's</a> mantra: build security in. That means you address security first, then achieve proper functioning and performance while maintaining it.<br />
<br />
I'll temper that with <a href="https://www.schneier.com/">Bruce Schneier's</a> key point: security is a trade-off. That means there's no such thing as absolute security, and you get security by giving something up.<br />
<br />
I look at the combination of the two like this: we must focus on security from the start, but we have to realize that it can only get us so far within the context of the larger environment, and we're going to have to give up something in functional convenience and performance.<br />
<br />
I'm not a security expert. I'm a student of security, so that I can become a practitioner. That's what we all need to do, become students of security so that we can become practitioners, looking to experts like McGraw and Schneier to guide us in the appropriate practices.<br />
<br />
Real security engineering requires you to think from both sides of the fence. You need to think like a good guy defender ("white hat") and a bad guy attacker ("black hat").<br />
<br />
On the white hat side, you need to know the proper security practices to follow. On the black hat side, you need to know what attacks will be arrayed against you; otherwise you end up creating the software version of the Maginot line, an ineffective defense against the actual attack.<br />
<br />
Security isn't something you bolt on after the fact. There is no "security layer". It has to be built in from the beginning. It has to be interwoven throughout, part of the raw fabric.<br />
<br />
And just because one part is secure doesn't mean that all the rest is safe. It's all too easy to undermine the security by not maintaining vigilance system-wide, throughout all uses of the system and the data it produces, in all environments and contexts, over its entire life.<br />
<br />
Security is easy to get wrong and hard to get right, and easy to get wrong again once you get it right. There are a lot of details. Understanding those details and how they all fit together takes effort. That's why you have to study the literature and learn how to apply the techniques properly.<br />
<br />
Some of the recommendations may seem arbitrary. For instance, a recommendation not to use a particular library function, because it's been the source of many security vulnerabilities in the past. You can say, well, I'm going to use it correctly in my code so that doesn't happen.<br />
<br />
But what about a year from now, when you've moved on to another project, or you've left the company, and someone else comes in and has to make some changes to add a new feature? Or they lift your code out to a different context. They may not notice the potential for a problem and end up making your formerly safe code unsafe.<br />
<br />
Borrowing a line from the top 10 security design flaws document in the reading list below, designing for security should take into account that code typically evolves over time, resulting in the risk that gaps in security are introduced in later stages of the software life-cycle.<br />
<br />
<b>What Causes Vulnerabilities?</b><br />
<br />
<i>Vulnerabilities</i> are problems that can be exploited by attackers. They are the unlocked doors that allow entry. Not all software problems result in security vulnerabilities. But software problems are a rich ground for finding vulnerabilities. What causes them?<br />
<br />
We can look at software correctness in two dimensions, design and implementation. Each can be either correct or incorrect. Adopting McGraw's terminology, "flaws" are problems in design. "Bugs" are problems in implementation.<br />
<blockquote class="tr_bq">
<i>Note that I'm lumping requirements in with design, so incorrect requirements implies incorrect design. You could treat requirements as a third independent dimension that can be correct or incorrect, but the results are really the same for this discussion.</i></blockquote>
This gives us four quadrants into which software may fall:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivaNEMHdndXA3hoVpWM8iFKEC3-L0PipS7lM6FXeTvpB1Pt0JY9MFp2R1zJwZRxOzCRDv54bME5WbfUuuipJ6tE3GjX5oTNk7z6oawzfxhrXn9rE9-9Vxgb3vtbEUU3mswimVJcHdaqLeo/s1600/SoftwareProblemQuadrants.png" imageanchor="1"><img border="0" data-original-height="170" data-original-width="377" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivaNEMHdndXA3hoVpWM8iFKEC3-L0PipS7lM6FXeTvpB1Pt0JY9MFp2R1zJwZRxOzCRDv54bME5WbfUuuipJ6tE3GjX5oTNk7z6oawzfxhrXn9rE9-9Vxgb3vtbEUU3mswimVJcHdaqLeo/s1600/SoftwareProblemQuadrants.png" /></a>
<br />
<br />
Software is problem-free in only one quadrant: <i>correct design</i> (free of flaws), and <i>correct implementation</i> of that design (free of bugs).<br />
<br />
It's very important to realize that in two of the quadrants where one dimension is correct, you are still doomed to have software problems. You can have a correct design, but incorrect implementation. Or, you can have a perfect, bug-free implementation, but of an incorrect design.<br />
<blockquote class="tr_bq">
<i>If you treat requirements as a third dimension, that produces a cube of eight octants. You can see that this discussion generalizes to the same thing. Software is problem-free in only one octant: correct requirements, correct design to meet those requirements, and correct implementation of that design. If the requirements are incorrect, no matter how perfect the design and implementation, the software has problems.</i> </blockquote>
<blockquote class="tr_bq">
<i>So for simplicity, we can collapse it down to the two-dimensional discussion. Just be aware that if you get the requirements wrong, the design is by definition incorrect (since it is designed for the wrong thing, no matter how perfectly done).</i></blockquote>
What all this means is that there are many opportinities to create a problem, and a potential vulnerability.<br />
<br />
That's part of what I mean when I say security is hard to get right, and easy to get wrong. The other part is that there are lots of subtle details, and getting any single one wrong risks undermining all the rest.<br />
<br />
That's what real engineering is about, dealing with all that, being rigorous and thorough and getting it all right top to bottom, beginning to end. That's what it means to be a responsible professional. Yeah, it's complicated. Yeah, it's hard work.<br />
<br />
Are the odds really as bad as just a 1 in 4 chance of getting it right, or even 1 in 8? That may be abusing probability and statistics to overstate the situation, but it does show that the odds are against you.<br />
<br />
And if you aren't testing for security vulnerabilities, you can bet that those bad people are. They're out there actively searching for your systems and probing them for vulnerabilities. They will find them. Then all you've done is added to the problem.<br />
<br />
The tools for evaluating and implementing security are useful to both defenders and attackers. Regardless of how you use those tools to improve security (if at all), adversaries are using them to pick your systems apart.<br />
<br />
That's why you need to learn how to use them, and why you have to put on the black hat and think that way. Where attackers will use the results to attack your system, you can use those same results to feed back into the development process to improve the design and implementation of the system from a security standpoint.<br />
<br />
<b>Next Steps</b><br />
<br />
The first step is awareness. That's what this post is about. The second step is learning. The third step is putting the knowledge into practice. The fourth step is maintaining continuous vigilance.<br />
<br />
It starts with us, the developers. It also ends with us, because no one else is going to do it.<br />
<br />
<b>Reading List</b><br />
<br />
This is the reading list I've accumulated for the second step, learning, that I'm working my way through. There's some overlap here with my reading list from <a href="http://flinkandblink.blogspot.com/2017/11/testing-is-how-you-avoid-looking-stupid.html">Testing Is How You Avoid Looking Stupid</a>. Once again, the market in used books helps keep the cost down.<br />
<br />
Interestingly, most of these are over 10 years old. Yet they remain as timely as ever. The same vulnerabilities still show up repeatedly. But their potential impact on our real daily lives has grown significantly. These are no longer abstract problems.<br />
<br />
There are two nice starting points. They help set the background necessary to appreciate the others:<br />
<ul>
<li>First, because it's free, relatively brief, and a nice overview of security design flaws, is the IEEE Computer Society Center for Secure Design paper <a href="https://www.computer.org/cms/CYBSI/docs/Top-10-Flaws.pdf">AVOIDING THE TOP 10 SOFTWARE SECURITY DESIGN FLAWS</a>.</li>
<li>Second, another free overview of common security risks, the Open Web Application Security Project paper <a href="https://www.owasp.org/images/7/72/OWASP_Top_10-2017_%28en%29.pdf.pdf">OWASP Top 10 Application Security Risks - 2017</a>.</li>
<li>Third, McGraw's book <a href="https://www.amazon.com/gp/product/0321356705/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321356705&linkCode=as2&tag=closegrain-20&linkId=d19041db1e63fcbb9d74a9c9a62520f6" target="_blank">Software Security: Building Security In</a><img alt="" border="0" height="1" src="https://ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0321356705" style="border: none !important; margin: 0px !important;" width="1" />, 2006, is a nice overview of the various considerations and methods for incorporating security, including the white hat/black hat duality.</li>
</ul>
Here's the remainder of the list, in no particular order, which will no doubt lead to many others:<br />
<ul>
<li><a href="https://www.amazon.com/gp/product/020172152X/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=020172152X&linkCode=as2&tag=closegrain-20&linkId=41888d73c1fd54eca6b7a9f5b9f3df53" target="_blank">Building Secure Software: How to Avoid Security Problems the Right Way</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=020172152X" style="border: none !important; margin: 0px !important;" width="1" />, 2001, John Viega, Gary McGraw.</li>
<li><a href="https://www.amazon.com/gp/product/0201786958/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0201786958&linkCode=as2&tag=closegrain-20&linkId=252ca5384814af167573ac50f492170c" target="_blank">Exploiting Software: How to Break Code</a><img alt="" border="0" height="1" src="https://ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0201786958" style="border: none !important; margin: 0px !important;" width="1" />, 2004, Greg Hoglund, Gary McGraw.</li>
<li><a href="https://www.amazon.com/gp/product/0470395354/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0470395354&linkCode=as2&tag=closegrain-20&linkId=5c49f305adbab3a6b27900134835793a" target="_blank">Schneier on Security</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0470395354" style="border: none !important; margin: 0px !important;" width="1" />, 2008, Bruce Schneier.</li>
<li><i>Update Oct 1, 2018:</i> <a href="https://amzn.to/2IsL75a">Click Here to Kill Everybody: Security and Survival in a Hyper-connected World</a>, 2018, Bruce Schneier. This is Schneier's newest book, focused on IOT-related security, a must-read given how pervasive Internet-connected systems have become to our daily lives.</li>
<li><a href="https://www.amazon.com/gp/product/0470068523/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0470068523&linkCode=as2&tag=closegrain-20&linkId=d0c7e25d314e7b0569476a4212f504bc" target="_blank">Security Engineering: A Guide to Building Dependable Distributed Systems</a><img alt="" border="0" height="1" src="https://ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0470068523" style="border: none !important; margin: 0px !important;" width="1" />, 2008, Ross Anderson.</li>
<li><a href="https://www.amazon.com/gp/product/0735617228/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0735617228&linkCode=as2&tag=closegrain-20&linkId=e400d7c0f7b88814421a2d6cad4001c1" target="_blank">Writing Secure Code: Practical Strategies and Proven Techniques for Building Secure Applications in a Networked World (Developer Best Practices)</a><img alt="" border="0" height="1" src="https://ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0735617228" style="border: none !important; margin: 0px !important;" width="1" />, 2004, Michael Howard, David LeBlanc.</li>
<li><a href="https://resources.sei.cmu.edu/downloads/secure-coding/assets/sei-cert-c-coding-standard-2016-v01.pdf">SEI CERT C Secure Coding Standard: Rules for Developing Safe, Reliable, and Secure Systems (free downloadable PDF)</a>, 2016, multiple authors.</li>
<li><a href="https://www.amazon.com/gp/product/0321822137/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321822137&linkCode=as2&tag=closegrain-20&linkId=5ca61bdb692be3ea57e2eac82357f49e" target="_blank">Secure Coding in C and C++ (2nd Edition) (SEI Series in Software Engineering)</a><img alt="" border="0" height="1" src="https://ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0321822137" style="border: none !important; margin: 0px !important;" width="1" />, 2013, Robert C. Seacord. Seacord is one of the authors of the SEI CERT C Secure Coding Standard above.</li>
<li><a href="https://www.amazon.com/gp/product/047008023X/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=047008023X&linkCode=as2&tag=closegrain-20&linkId=2c8e37234fcbf9d9ad16807f9a7acedc" target="_blank">The Shellcoder's Handbook: Discovering and Exploiting Security Holes</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=047008023X" style="border: none !important; margin: 0px !important;" width="1" />, 2nd Ed. 2007, Chris Anley, John Heasman, Felix Lindner, Gerardo Richarte.</li>
<li><a href="https://www.amazon.com/gp/product/020163497X/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=020163497X&linkCode=as2&tag=closegrain-20&linkId=b99a47db6fc71b004f785488ea5550d6" target="_blank">Forensic Discovery</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=020163497X" style="border: none !important; margin: 0px !important;" width="1" />, 2005, Dan Farmer, Wietse Venema.</li>
</ul>
<div>
For a little perspective on the nature of vulnerabilities, see C.A.R. "Tony" Hoare's presentation on null references, what he calls his <a href="https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare">"billion dollar mistake"</a> (though perhaps karma and cost balance out, since he also invented the quicksort algorithm, among many other brilliant contributions to computer science).<br />
<br />
In addition to McGraw's and Schneier's websites, several good sources for security-related news and information:</div>
<div>
<ul>
<li><a href="https://catless.ncl.ac.uk/Risks/">Risks Digest</a>, <i>Forum on Risks to the Public in Computers and Related Systems</i>, ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator. This is where it all starts for me, fascinating reading (in the way watching a train wreck is fascinating).</li>
<li><a href="https://www.sei.cmu.edu/research-capabilities/cybersecurity/index.cfm">CMU SEI Cybersecurity</a>, Carnegie Mellon University Software Engineering Institute cybersecurity main page.</li>
<li><a href="https://www.sei.cmu.edu/about/divisions/cert/">CMU SEI CERT Division</a>, CMU SEI Computer Emergency Response Team.</li>
<li><a href="https://krebsonsecurity.com/">Krebs On Security</a>, Brian Krebs.</li>
<li><a href="https://threatpost.com/">Threatpost</a>.</li>
<li><a href="https://www.owasp.org/">Open Web Application Security Project (OWASP)</a></li>
<li>Others? Probably, but also be aware that this is a topic ripe for abuse, so CHECK YOUR SOURCES AND CORROBORATE YOUR INFORMATION. 'Nuff said.</li>
</ul>
</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-47388138799665135772017-12-10T13:04:00.001-08:002018-04-19T18:58:59.714-07:00First Code(Go back to <a href="http://flinkandblink.blogspot.com/2017/12/learn-to-code.html">Learn To Code Introduction</a>)<br />
<br />
Let's jump into some code. I'll be spewing terms right and left here. I'll deal with some quickly, just enough detail to get by, but defer others for later discussion.<br />
<br />
I'm also going to be a bit wordy, saying the same things in different ways so that you pick up the terminology and the different ways people express these things. Forgive me if I beat a concept to death. As you'll see, many terms also get reused for different things in a mix of formal and informal usage. That's why terminology can be so confusing.<br />
<br />
And for everything I say, there are exceptions, arguments, counter-arguments, and many more details. I'll address some of those in later posts. For now just bear with me so we don't get too far off into the weeds.<br />
<br />
For every characteristic of a given language, there are those who think it's great, and those who think it's terrible. Good or bad, some things are certainly a source of confusion. I take a neutral approach. A language is a tool, it is what it is. Understand the pros and cons and bear them in mind when using it.<br />
<br />
An important point to remember is that there are always multiple ways to do something. From the formatting of source code to data structures, design, and organization of a program, you have virtually infinite choices. These rapidly escalate into religious wars. Some choices are worth arguing over. Some aren't.<br />
<br />
<b>The C Language</b><br />
<br />
C is a <i>high-level language</i> (as opposed to a <i>low-level language</i>), which means it is a human-readable text language where <i>source code</i> specifies the instructions for how a program runs. However, computers don't execute high-level code, they execute binary <i>machine instructions</i>, also known as <i>machine code</i>. At some point, high-level source code needs to be translated to machine code so that it can be executed.<br />
<br />
This brings up the critically important concept of <i>abstraction</i>, which will show up in many ways in this series. It takes multiple machine instructions to carry out each source instruction.<br />
<br />
High-level languages elevate your thinking to a higher level of abstraction, allowing you to <i>abstract</i> the low-level details. This is a huge benefit, because instead of having to think about all the tiny details of machine instructions, you can focus on the higher-level concepts of your program logic.<br />
<br />
Contrast this to <i>assembly language</i>, which is a human-readable low-level language. Assembly language statements translate one for one to machine code, so you have to deal with all those low-level details.<br />
<br />
C is a <i>compiled language</i> (as opposed to an <i>interpreted language</i>), which means a software tool called a <i>compiler</i> translates source code into machine code, also called <i>object code</i>. The compiler <i>compiles</i> source code instructions and <i>generates</i> object code corresponding to their logic.<br />
<br />
Each <i>source file</i> produces an <i>object file</i>. These files are often referred to simply as <i>sources</i> and <i>objects</i> (but don't confuse this use of the term object with its use in <i>object-oriented programming (OOP)</i>).<br />
<br />
A tool called a <i>linker</i> then <i>links</i> your objects with objects from pre-built <i>runtime libraries</i> to produce the final complete program. The result is an <i>executable</i>, also known simply as a <i>binary</i>.<br />
<br />
This set of tools and libraries comprise the <i>toolchain</i>, and are specific to the type of system where you'll be running the binary. That's why there are separate versions of programs for Mac and for Windows.<br />
<br />
Once you've built the binary with this <i>build process</i>, you can run it any number of times without having to run it through the toolchain again. You only have to <i>rebuild</i> if you make a change to the source.<br />
<br />
You can distribute the binary to other people who have the same type of system without having to give them your source code. They don't need to have the toolchain to be able to use the binary.<br />
<br />
C is a <i>statically-typed language</i> (as opposed to a <i>dynamically-typed language</i>, static meaning constant, fixed, unchanging, and dynamic meaning varying, changing), which means you have to <i>declare</i> an item of data in a <i>type declaration</i> to tell the compiler what <i>type </i>it is (integer numeric, floating point numeric, character string, etc.) before you can use it, and you can only store that type of data in it (that's the static part, the fact that the type is fixed ahead of time).<br />
<br />
C has rules about what words are reserved as part of the language, known as <i>key words</i>, how you can name your own things as <i>user-defined names</i>, punctuation, and how to form <i>statements</i>. This is the <i>syntax</i> of the language, just as grammar and spelling rules are the syntax of spoken languages.<br />
<br />
Those statements have some particular meaning and cause the program to behave in a particular way. This is the <i>semantics</i> of the language, just as the meaning and implications of sentences are the semantics of spoken languages.<br />
<br />
And like spoken languages, you can construct statements that are syntactically correct, but semantically incorrect, such as saying, "The sky is fast." That's a perfectly legal sentence, but it doesn't make any sense. Similarly, you can write code that is legal, but doesn't do what you want.<br />
<br />
To make a program that does what you want, you have to write code that is both syntactically correct, and semantically correct. The compiler will tell you if you make syntax errors, so you correct the code and try again, but once you have correct syntax, it can't tell you anything about the semantics. You have to run the program and test it to tell if you got the semantics right.<br />
<br />
That's the real challenge of software development. The compiler will quickly help you find and correct syntax errors. But a complex piece of software can have many behaviors, and testing and verifying them can be as much work as writing it in the first place.<br />
<br />
Further complicating things, while there's only one set of correct behaviors that you want it to to, there's an infinite variety of random incorrect things it can do if the semantics are wrong. When it does strange and unexpected things, you have figure it out so you can correct the semantics. This can be time-consuming and frustrating. "Well, I know what I <i>wanted</i> it do, but what did it <i>actually</i> do? And why?"<br />
<br />
The classic book <i>The C Programming Language</i> by Kernighan and Ritchie (known as K&R) established the tradition of the "hello, world" program before getting into slightly more complex examples. I'll buck that tradition by skipping right to the latter. Like their examples, this provides the framework to start presenting lots of details.<br />
<br />
K&R described the initial version of the C language, defining the original syntax and semantics. Over the years, the language has been changed to improve and standardize it. The current version is known as C11, for the 2011 standard. The previous version was C99.<br />
<br />
You need to know which standard your compiler supports so that you write the code it will understand. Some language version differences are minor, making the syntax a little more convenient, and some are major, changing the way you do things. I'll use C11 here, but I have a tendency to use older style when I don't think about it, and you'll run into code written that way out in the real world.<br />
<br />
<b>Source Control</b><br />
<br />
You can find all the <i>source code</i> for this series in a public <i>GitHub repository</i> at <a href="https://github.com/sdbranam/learntocode">https://github.com/sdbranam/learntocode</a>. GitHub is an online <i>version control system (VCS)</i>, a place that stores source code, also known as a <i>source control</i> system. It can store multiple versions of the code. There are other VCS's besides GitHub, which is actually an online version of <i>git</i>.<br />
<br />
A VCS has two main purposes: protecting code against loss, and sharing code among multiple developers. Source code can be lost two ways: by deleting a file (losing the entire file), or by changing its contents (losing that particular version of the code).<br />
<br />
By storing multiple versions, a VCS allows any version to be recovered. It also allows tracing specific changes to specific developers, so you can tell who did what to the code, known as the <i>annotate</i> or <i>blame</i> function.<br />
<br />
Sharing allows multiple people to work on code, or distribution of code from the authors to others, as I'm doing here. Settings on the repository, known as a <i>repo</i>, control who is allowed to see its contents (you might want it to be private within your company), and who is allowed to make changes to it (you might only want authorized developers to be able to change it).<br />
<br />
This is my repo, that I've made publicly accessible. Anyone can create their own public repo for free, and I'll show you how to do that later, since you'll want to have one for working on this series. A public repo is also a good way to showcase your work to others.<br />
<br />
<b>The Program</b><br />
<br />
This is file <a href="https://github.com/sdbranam/learntocode/blob/master/printargs.c">printargs.c</a> , containing the entire program:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-style: italic;">/* </span>
<span style="color: #008800; font-style: italic;"> * Printargs prints the list of arguments from the command line.</span>
<span style="color: #008800; font-style: italic;"> * It returns EXIT_SUCCESS if at least one argument was specified,</span>
<span style="color: #008800; font-style: italic;"> * or EXIT_FAILURE if no arguments were specified.</span>
<span style="color: #008800; font-style: italic;"> *</span>
<span style="color: #008800; font-style: italic;"> * 2017 Steve Branam <sdbranam@gmail.com> learntocode</span>
<span style="color: #008800; font-style: italic;"> */</span>
<span style="color: #008800;">#include <stdio.h></span>
<span style="color: #008800;">#include <stdlib.h></span>
<span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">main</span>(<span style="color: #00bb00; font-weight: bold;">int</span> argc, <span style="color: #00bb00; font-weight: bold;">char</span> <span style="color: #666666;">*</span>argv[])
{
<span style="color: #00bb00; font-weight: bold;">int</span> i;
<span style="color: #aa22ff; font-weight: bold;">if</span> (argc <span style="color: #666666;"><</span> <span style="color: #666666;">2</span>) {
printf(<span style="color: #bb4444;">"Usage: %s <arguments></span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>
<span style="color: #bb4444;">"Prints command line arguments.</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>,
argv[<span style="color: #666666;">0</span>]);
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">for</span> (i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> argc; <span style="color: #666666;">++</span>i) {
printf(<span style="color: #bb4444;">"%d: %s</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>, i, argv[i]);
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_SUCCESS;
}
</pre>
</td></tr>
</tbody></table>
</div>
<br />
Note the different colors. This is called <i>syntax highlighting</i>, which helps in visually navigating through code. Many source code editors do this, along with other convenience features that help speed up working on code.<br />
<br />
I used the online <i>source code formatter</i> <a href="http://hilite.me/">http://hilite.me/</a> to produce this listing, with CSS: "border:solid gray;border-width:1px 1px 1px 1px;padding:.1em .2em;" and Style: "emacs" (although it appears the blog settings override the 1px border). I ran the output of that through a crude Python script to extract specific lines below.<br />
<br />
The way to read code is to scan visually looking for the blocks, to get a feel for the overall shape. And when I say shape, I mean that literally. The way it's spaced out and indented is a visual guide to its logic.<br />
<br />
Spacing and indentation help you follow that structure. Code without spacing or indentation is very hard to follow. It's like trying to read a book where all the text is jammed together. Paragraphs help break up the page. Spacing and indentation help break up the source code.<br />
<br />
The C compiler ignores spacing and indentation. It simply reads through the file, skipping over them as it <i>parses</i> the statements. With a couple of exceptions that I'll cover below, line boundaries are irrelevant, so you can arrange the code any way you want.<br />
<br />
Identify the boundaries between blocks, then pick the ones to dig into further. That's more of that abstraction, allowing you to focus on the bigger things before you get into the finer details, like seeing the forest before you see the trees.<br />
<br />
The first blocks to look for are the <i>functions</i>. These are the modular building blocks of code. They provide the overall separation of logic into manageable chunks.<br />
<br />
Deciding how to divide things up into those chunks is a major part of the art of coding. It's one of those things that can be difficult to describe, but you know it when you see it. When you do see it, look for more by that person. Good code, like good art or good music, is something to be appreciated and emulated.<br />
<br />
There are many guidelines for what constitutes good structuring of functions. For now, think in terms of division of labor. Rather than one big chunk that does it all from start to finish, divide up the work, like delegating a big job to a team of workers, each with their own responsibility, with some of them providing helper services that the others can use.<br />
<br />
And just as it's a bad idea to overload an individual worker with too much, it's a bad idea to make a function too long. Break up the work into smaller functions that the larger function can <i>call</i>.<br />
<br />
Just like a large team of workers that needs to be organized into a hierarchy of different levels that depend on each other to carry out their responsibilities, organize functions into a hierarchy where they depend on each other to get their work done.<br />
<br />
A top level function depends on the next level of functions for the overall program, and those functions depend on a third level of functions for their responsibility, and so on, as deeply <i>nested</i> as necessary. That's how you manage complexity in real software.<br />
<br />
Now I'll break down the different sections in the file, known as <i>snippets</i>. This is how I'll go through code throughout this series. I'll go into excrutiating gory detail on this one because there are so many concepts to introduce.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-style: italic;">/* </span>
<span style="color: #008800; font-style: italic;"> * Printargs prints the list of arguments from the command line.</span>
<span style="color: #008800; font-style: italic;"> * It returns EXIT_SUCCESS if at least one argument was specified,</span>
<span style="color: #008800; font-style: italic;"> * or EXIT_FAILURE if no arguments were specified.</span>
<span style="color: #008800; font-style: italic;"> *</span>
<span style="color: #008800; font-style: italic;"> * 2017 Steve Branam <sdbranam@gmail.com> learntocode</span>
<span style="color: #008800; font-style: italic;"> */</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Lines 1-7:</b> a <i>comment,</i> free-form text that helps the reader understand the code. The comment is <i>delimited</i> by the /* and */ markers. Everything between theses <i>comment delimiters</i> is ignored by the compiler.<br />
<br />
Another style of comment delimiter is the double slash //. Everything from the double slash to the end of the line is a comment. This is one place where line boundaries mean something to the compiler.<br />
<br />
This particular comment describes the program. It also lists author information. A brief <i>header comment</i> like this at the top of a file is a big help to readers sifting through files.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 9
10</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #008800;">#include <stdio.h></span>
<span style="color: #008800;">#include <stdlib.h></span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Lines 9-10:</b> <i>preprocessor directives</i> that direct the <i>preprocessor</i> to include two <i>system header files</i> at this point in the file. These are also known as <i>system headers</i>, <i>header files</i>, or simply <i>headers</i>. File inclusion is a way to pull other code into the file, breaking things up into modular parts.<br />
<br />
These headers are from the <i>standard library</i>, which contains predefined code required to make your source a complete program. The headers themselves contain various declarations, including <i>forward declarations</i> that tell the compiler about the functions in the library.<br />
<br />
File stdio.h contains declarations for the standard <i>input/ouput (I/O)</i> functions. File stdlib.h contains declarations for various <i>constants</i>, fixed data values that don't change as the program runs (i.e. they remain constant).<br />
<br />
Preprocessor directives are the other place where line boundaries are significant. The preprocessor is actually an initial stage of the compiler that processes the source code text before compiling, executing directives as it finds them.<br />
<br />
Each preprocessor directive takes one line, although that can be extended by putting a backslash \ at the end of the line to form a multi-line directive.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">main</span>(<span style="color: #00bb00; font-weight: bold;">int</span> argc, <span style="color: #00bb00; font-weight: bold;">char</span> <span style="color: #666666;">*</span>argv[])
{
<span style="color: #00bb00; font-weight: bold;">int</span> i;
<span style="color: #aa22ff; font-weight: bold;">if</span> (argc <span style="color: #666666;"><</span> <span style="color: #666666;">2</span>) {
printf(<span style="color: #bb4444;">"Usage: %s <arguments></span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>
<span style="color: #bb4444;">"Prints command line arguments.</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>,
argv[<span style="color: #666666;">0</span>]);
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">for</span> (i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> argc; <span style="color: #666666;">++</span>i) {
printf(<span style="color: #bb4444;">"%d: %s</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>, i, argv[i]);
}
}
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_SUCCESS;
}
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Lines 12-28:</b> the <i>main</i> function, the top level function in the function call hierarchy. C requires one function in the program to be named main(), which defines the program <i>entry point</i>. This is where the program starts when you run it. Note that I use the function name with empty parenthesis when I refer to it informally.<br />
<br />
You can define other functions with any name you want as long as they conform to the C naming syntax.<br />
<br class="Apple-interchange-newline" />
C encloses things in braces {}. They delimit the function body itself, and blocks within the function. Any number of lines may appear in a block.<br />
<br />
There are places where the braces aren't required, when a block contains only one line, but I put them in anyway because a common source of bugs is expanding a one-line block into a multi-line block and forgetting to add the braces.<br />
<div>
<br /></div>
Look at the shape of the function. You can see it's outline and the shape of the blocks inside, hinting at its logical flow. After identifying the blocks, look at the details inside them. This on is pretty simple, but others can get more complex, with blocks nested within blocks.<br />
<br />
Just as the functions work in layers, the code inside them does. As with the hierarchical layers of functions, these are layers of logic, like peeling back the layers of an onion.<br />
<br />
But unlike layers of functions that can nest to any depth, you don't want to have too many layers within a single function, or it can be difficult to follow. Code that's difficult to follow has a higher likelihood of bugs (or you end up adding bugs when you try to change it).<br />
<br />
Now lets dig down a level into the function.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">12</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">main</span>(<span style="color: #00bb00; font-weight: bold;">int</span> argc, <span style="color: #00bb00; font-weight: bold;">char</span> <span style="color: #666666;">*</span>argv[])
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Line 12:</b> the <i>function declaration</i>, telling the compiler what the function's <i>call interface</i> is. This is how other code must <i>call</i> the function to invoke it.<br />
<br />
The function main() has an interface that's predefined by C, but other functions allow you to specify the interface yourself.<br />
<br />
The items separated by commas in the parenthesis () are the function <i>parameters</i>. These are a type of <i>variable</i> that holds the <i>arguments</i> passed into the function when it is <i>called</i>.<br />
<div>
<br /></div>
<div>
Each parameter is identified by its <i>data type</i>, shown in bold green, and its name, shown in black. The asterisk * and square brackets [] indicate characteristics of the arguments. The asterisk means <i>pointer</i>. The brackets mean <i>array</i>. I'll cover pointers later; they're often a source of confusion, but once you understand how to visualize them, they're easy.</div>
<br />
A data type specifies what kind of values are used. The int type means integer numbers, such as 0, 1, and -2. The char type means a character, as in a letter, digit, or punctuation mark.<br />
<br />
An <i>array</i> is a contiguous block of <i>elements</i> of the same data type. An array of characters forms a <i>character string</i>, or simply <i>string</i>.<br />
<br />
Because these types are predefined by the language, they are known as <i>primitive types</i>. You can use these as building blocks to define your own <i>user-defined types</i>.<br />
<br />
In the case of main(), the parameters are the <i>command line arguments</i> that are passed to the program itself when you run it. This is one way to get information from the outside world into the program.<br />
<br />
The argc parameter is the argument count, and argv is the argument <i>vector</i> that contains the list of all the arguments, including the program name. Vector is another term for an array. Thus argv is an array of pointers to characters.<br />
<br />
Because of the way strings work in C, a pointer to a character is often interpreted as a pointer to a whole string of characters, not just a single character. I'll cover that more when I talk about pointers. But that means argv is an array of pointers to strings.<br />
<br />
Once you've built the program, typing "printargs hello world" on the command line results in running the program and calling main() with argc containing the value 3, and argv containing the strings "printargs", "hello", and "world". Notice that the first string is the name of the program as it appeared on the command line.<br />
<br />
What about that very first int on line 12? That's the <i>function return type</i>, indicating the type of value the function returns to whatever called it. So just as the parameters had a data type and name, the function has a data type and name.<br />
<br />
Since main() is called by the <i>operating system (OS)</i>, through some extra layers that we won't worry about now, the int return type means that main() returns an integer value to the OS. Since this is the value the program returns when it <i>exits</i>, this is called the <i>exit code</i>.<br />
<br />
The term <i>caller</i> refers to whatever code is calling the function, and the term <i>callee</i> refers to the function being called. The caller calls the function with specific arguments. The function, as callee, receives those specific argument values in its parameters, and returns a value to the caller. The caller may use the return value in some way.<br />
<br />
Some functions exist purely to produce a value to return to the caller, such as square(x), which computes the mathematical square of x and returns it. The value produced is the primary purpose of the function, and any work performed producing it is a consequence of achieving that goal.<br />
<br />
Other functions are intended to do some sort of work, and then return a result indicating the status of the work. The work performed is the primary purpose of the function, and the value returned from it is merely a report of what happened.<br />
<br />
The first type of function is closer to the mathematical concept of functions. In its purest form, such a function has no <i>side effects</i>, meaning it does not affect anything else. The only output of the function is the return value, which is based solely on its input parameters. There is an entire field of <i>functional programming</i> based on this.<br />
<br />
The second type of function is intended to produce side effects. Such a function is intended to affect other things in the program or the outside world. In this case, main() has the side effect of printing something out. The value returned by main() indicates the success or failure of the program.<br />
<br />
A return value that is intended as a status indicator is often referred to as a <i>status code</i>. Some status codes simply indicate a binary status, "true" or "false", "yes" or "no", "success" or "failure". Other status codes may convey more detailed information, often used to discriminate various errors, such as "success", "failure, bad filename", "failure, full disk", or "failure, not authorized".<br />
<br />
It's also possible to have a function that doesn't return a value. The side effects of running the function are the only thing that happens. In this case, the return type is declared as <i>void</i>, so this is known as a <i>void function</i>.<br />
<br />
What line 12 means is "Main is a function that accepts an integer argument count and an array of character pointers to argument strings, and returns an integer." Everything in line 12, except for the parameter names, forms the <i>function signature</i>. That is, the signature consists of the function name, its parameter types, and its return type.<br />
<br />
Saying a signature out loud is a mouthful, so in informal usage you just use the name. But when you write code that calls the function, you have to know the precise signature so that you call the function the right way.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">14
15
16
17
18
19
20
21
22
23
24
25
26</pre>
</td><td><pre style="line-height: 125%; margin: 0;"> <span style="color: #00bb00; font-weight: bold;">int</span> i;
<span style="color: #aa22ff; font-weight: bold;">if</span> (argc <span style="color: #666666;"><</span> <span style="color: #666666;">2</span>) {
printf(<span style="color: #bb4444;">"Usage: %s <arguments></span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>
<span style="color: #bb4444;">"Prints command line arguments.</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>,
argv[<span style="color: #666666;">0</span>]);
<span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_FAILURE;
}
<span style="color: #aa22ff; font-weight: bold;">else</span> {
<span style="color: #aa22ff; font-weight: bold;">for</span> (i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> argc; <span style="color: #666666;">++</span>i) {
printf(<span style="color: #bb4444;">"%d: %s</span><span style="color: #bb6622; font-weight: bold;">\n</span><span style="color: #bb4444;">"</span>, i, argv[i]);
}
}
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Lines 14-26:</b> the function <i>body</i>, everything enclosed in the braces that follow the signature. This <i>defines</i> the function. It's where the work of the function gets done.<br />
<br />
This function uses two <i>control structures</i>, an <i>if-else decision</i> in lines 16-26, and a <i>for-loop</i> in lines 23-25, nested in the else block of the decision. Control structures direct the flow of control of execution.<br />
<br />
An if-else decision checks some <i>condition</i>, in this case whether the argument count is less than 2, and does something based on the result. It goes one way if the condition is <i>true</i>, and the other way if the condition is <i>false</i>.<br />
<br />
If there's nothing to do when the condition is false, you can omit the else portion. You use a simple <i>if decision</i>, that only doing something if the condition is true.<br />
<br />
A for-loop repeats the block it contains for some number of times. Each repetition cycle is known as an <i>iteration</i>. It is therefore often used for <i>iterating</i> through something, cycling through it. Iterating through the elements of an array is a common use of for-loops.<br />
<br />
Line 14 is another variable of type int, named simply i. This one is a <i>local variable</i>, a variable that is local to the function; it exists only within the <i>scope</i> of the function. After the function returns, it no longer exists and no longer has a value.<br />
<br />
This line is both a <i>variable definition</i> and a<i> variable declaration</i>. It declares the type and name of the variable, and defines the memory for it.<br />
<br />
The function parameters are also local variables, the difference being that their values are set by the arguments that are passed in.<br />
<br />
What exactly is a variable? It's a small portion of memory that contains a value that can change over time based on what the program does. The fact that it can change is what makes it a variable.<br />
<br />
C uses <i>call by value</i>, meaning that it passes the values of things into function parameters. But pointers provide a way to <i>call by reference</i>.<br />
<br />
The name "i" is very simple and doesn't convey much meaning. However, it's common to use single-letter names for local variables used as simple for-loop controls. For other variables, used in more complex ways, it's better to use more descriptive names.<br />
<br />
Line 16 tests the value of argc to see if it's less than 2. If so, it calls <i>library function</i> printf() to print a formatted message. This function was <i>forward-declared</i> in stdio.h, so the compiler knows its signature and can check that I used it correctly (syntactically, not necessarily semantically).<br />
<br />
The arguments to printf() are a string that describes the format of the message, and the data values to be formatted.<br />
<br />
In this case, the string is a <i>hard-coded</i> constant, meaning the actual string is coded right there where it's used. It's delimited by double-quotes. The \n at the end of each line is an <i>escape sequence</i> that contains a control character called <i>newline</i>. Newline causes a new line to be started in the program output. The %s is a <i>conversion specification</i> that shows where the value of another string should be substituted into the output; the process of substituting values for markers in a string is called <i>string substitution</i>.<br />
<br />
There's another subtle thing going on with this function call. Notice that the arguments in a function call are separated by commas. But the comma is missing after the first string in line 17. This is a syntactic convenience called <i>string concatenation</i>, where the compiler joins together all the strings in the source that aren't separated by commas or semi-colons into a single string. This allows you to break up long strings in the source code for readability. So lines 17 and 18 only contain a single string, the first argument to printf().<br />
<br />
The second argument, argv[0], is the first element (i.e. the first entry) of the of the argv array. The square brackets [] contain the index of the element, which is 0. You might think that the first one would be 1, but C uses 0-based indexing. It's like the years in a century; the first year of a century is the 0 year, such as 1900 or 2000.<br />
<br />
Recall that argv was declared to be an array of pointers to strings. The first element is a therefore a single pointer to a string, so it matches up with the %s conversion specification.<br />
<br />
If you run the program with just the name on the command line, no arguments, argc will be 1. When line 16 checks that argc is less than 2, the condition will be true, and the function will execute the block in lines 17-21. The printf() will print out:<br />
<blockquote class="tr_bq">
<pre style="background-color: #f8f8f8; line-height: 16.25px;"><span style="color: #bb4444;">Usage: printargs <arguments></span>
<span style="color: #bb4444;">Prints command line arguments.</span></pre>
</blockquote>
A <i>usage message</i> like this is a common way to inform the user that they didn't supply all the command line arguments expected, or that the arguments were in some way unacceptable.<br />
<br />
After printing that message, line 20 will return from the function, with the value EXIT_FAILURE. This is a <i>symbolic constant</i> that was defined in stdlib.h. It's symbolic because we don't know its actual value here, all we know is a symbolic name that's been given to it. This indicates the program completed with some kind of error.<br />
<br />
<!-- HTML generated using hilite.me -->If you run the program with additional arguments on the command line, the condition in line 16 will be false, and the function will execute the else block in lines 22-26. This consists of the for-loop in lines 23-25.<br />
<br />
The for-loop iterates through the items in array argv, printing each one with printf().<br />
<br />
The for-loop uses i as the <i>control variable</i>, which it also uses as the index into the array. The for statement has three <i>control expressions</i> in the parenthesis, separated by semicolons, that control how it runs:<br />
<ul>
<li>The loop initialization, executed once before starting the loop, here initializing i to 0.</li>
<li>The loop condition, executed at the beginning of each cycle, here checking that i is less than argc.</li>
<li>The loop update, executed at the end of each cycle, here <i>pre-incrementing</i> i by 1.</li>
</ul>
As long as the condition is true, the loop keeps executing. Here, with i starting at 0 and incrementing on each iteration, it will execute until i reaches whatever count is in argc.<br />
<br />
You can have an empty initialization expression, if the condition that is being checked is already initialized before the for-loop. You can have an empty update expression, if the condition that is being checked is updated within the loop.<br />
<br />
The format string for the printf() in line 24 has a %d conversion specification, which means to substitute a decimal integer value, and a %s for a string. The remaining printf() arguments are the array index, and the array value at that index. So the printf() prints out a number and a string.<br />
<br />
It's important to be aware of how the 0-based indexing relates to the specific check in the for-loop condition. Otherwise the loop may not execute enough times, or may execute one time too many. This is a common source of <i>off-by-one bugs</i>.<br />
<br />
Incorrect control expressions can also cause <i>dead loops</i>, that never run through any iterations, or <i>infinite loops</i>, that never end.<br />
<br />
The easiest way to figure this out is to step through the iterations yourself, remembering that this update expression increments i after every cycle. If the command line is "printargs hello world", argc will be 3. Therefore:<br />
<ul>
<li>On the first iteration, i will be 0, so the condition is true, and it will print "0: printargs".</li>
<li>On the second iteration, i will be 1, so the condition is true, and it will print "1: hello".</li>
<li>On the third iteration, i will be 2, so the condition is true, and it will print "2: world".</li>
<li>On the fourth iteration, i will be 3, so the condition is false, and the loop terminates.</li>
</ul>
A simple way to model this on paper is with a table that steps through the index values <b>I</b>, the actual values used in the condition <b>C</b>, the condition result <b>R</b> (t for true, f for false), and the resulting value <b>V</b> represented by that iteration:<br />
<blockquote class="tr_bq">
<table border="1">
<tbody>
<tr>
<th>I</th>
<th>C</th>
<th>R</th>
<th>V</th>
</tr>
<tr>
<td>0</td>
<td>0 < 3</td>
<td>t</td>
<td>printargs</td>
</tr>
<tr>
<td>1</td>
<td>1 < 3</td>
<td>t</td>
<td>hello</td>
</tr>
<tr>
<td>2</td>
<td>2 < 3</td>
<td>t</td>
<td>world</td>
</tr>
<tr>
<td>3</td>
<td>3 < 3</td>
<td>f</td>
<td></td>
</tr>
</tbody></table>
</blockquote>
<div>
Drawing things out like this and stepping through the code yourself is a great way to work out the details, even on a simple example, so that you get the <i>initial conditions</i> and <i>termination conditions</i> right. It's even more helpful when the initialization is something other than 0, or the condition or resulting value is more complex.<br />
<br />
Notice also that i isn't used anywhere except in the for-loop, yet I declared it at the top of the function, where any parts of the function could access it (maybe when they shouldn't). A reader might reasonably wonder why I did it that way.<br />
<br />
This is one of those cases where my old-version C habits take over when I'm not thinking. That was a requirement of old C. New hotness allows i to be declared where used. So I could have put it right in the for statement:<br />
<blockquote class="tr_bq" style="background-color: #f8f8f8; line-height: 16.25px;">
<!-- HTML generated using hilite.me -->
<br />
<div style="background: #f8f8f8; border-width: 0.1px 0.1px 0.1px 0.1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">for</span> (<span style="color: #00bb00; font-weight: bold;">int</span> i <span style="color: #666666;">=</span> <span style="color: #666666;">0</span>; i <span style="color: #666666;"><</span> argc; <span style="color: #666666;">++</span>i) {
</pre>
</div>
</blockquote>
</div>
<div>
That limits the scope of code that can access it, and also makes it clear that this simply-named variable is just the loop control, not used for anything else. That's just one of the subtleties of coding to minimize the potential for errors and maximize understanding, especially as a function gets complex.<br />
<br />
The moral here is not only to keep up to date on language versions, but to remember to take advantage of them!<br />
<br /></div>
<div style="background: #f8f8f8; border-width: 1px 1px 1px 1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">27</pre>
</td><td><pre style="line-height: 125%; margin: 0;"> <span style="color: #aa22ff; font-weight: bold;">return</span> EXIT_SUCCESS;
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Line 27:</b> if the function reaches this point, it returns EXIT_SUCCESS, indicating it successfully printed out the arguments.<br />
<br />
It's important to remember that since you declared the function as returning a value, you must make sure that every possible return from the function actually does return a value. It's possible for the function to "run off the end" and return without explicitly returning a value. The result is that the caller will get back some random value.<br />
<br />
This can be a nasty type of bug, because sometimes that random value might be acceptable to the caller as a valid return value, even though it has no relation to what the function actually did. This can cause very mystifying behavior.<br />
<br />
For a function that returns a value, always put an explicit return statement at the end of the function. For a void function, which doesn't have a return value, you can simply let the function run off the end and return implicitly; you can also use a return statement with no value at the end of the function, but that's considered redundant and unnecessary.<br />
<br />
C also allows you to have different return points in a function (for a void function, these would be return statements without values). Some people like to code that way, as I did here, with two return statements. Others prefer to have only one return statement at the end of a function, using a local variable to keep track of the value to return.<br />
<br />
That's an awful lot to talk about a mere 28 lines of code. But now you're armed with a lot of terminology that will make getting through subsequent code faster. Some of the concepts may be a little shaky, but they'll firm up as we proceed.<br />
<br />
<b>Building And Running The Program</b><br />
<br />
The toolchain I'm using is GCC, the GNU C Compiler. It both compiles and links the program. It comes with Linux and Mac OS systems.<br />
<br />
There are also free online tools that allow you to build and run C code (and other languages) on a server. These are <i>sandboxed</i> environments that allow you to play around with code without any risk of affecting anyone else. These are useful if you're using a Chromebook or OS that's not setup for software development.<br />
<br />
One example of such an <i>online IDE (Integrated Development Environment)</i> is <a href="https://www.tutorialspoint.com/codingground.htm">CodingGround</a>. Some include complete online courses, and some include integration with GitHub. My experience with them shows that they're a great way to practice coding, but some can be a bit buggy, resulting in lost coding sessions or other problems. So don't rely on them to save your code.<br />
<br />
That's another good reason to <a href="https://help.github.com/articles/signing-up-for-a-new-github-account/">create an account on GitHub</a>, which is free to use for public and open source projects, and create your own public repo like my learntocode repo. You can upload and download your files, or even edit them directly on GitHub. If you use an online tool that doesn't have GitHub integration, you can copy-and-paste from a GitHub window into the tool window.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1px 0.1px 0.1px 0.1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8</pre>
</td><td><pre style="line-height: 125%; margin: 0;">$ <span style="color: #00a000;">gcc -v</span>
Configured with: --prefix=/Library/Developer/CommandLineTools/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
$ <span style="color: #00a000;">gcc printargs.c -o printargs</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Lines 1-6:</b> show the version of gcc. I'm building and running on a Mac, so the compiler and standard library themselves are built to run under Mac OS X on an x86 processor, generating code that will run under Mac OS X on an x86 processor.<br />
<br />
<b>Line 8:</b> the build command. This is about the simplest possible build command, directing gcc to compile source file printargs.c and output an executable in file printargs. Builds can get quite complex, allowing you to construct software from a number of parts.<br />
<br />
If gcc finds a syntax error, it prints it out along with the line number, and won't produce an executable. It may also print warnings, which indicates things that are at risk of being an error. If there are warning but no errors, it will go ahead and produce an executable.<br />
<br />
Depending on the severity of an error, the compilation may end prematurely. But the compiler will try to get as far as it can, reporting as many errors as it can.<br />
<br />
That's both good and bad. It's nice to get all the errors at once so you can fix them all. But some errors can have a cascading effect that causes the compiler to report many other things as errors because incorrect syntax has thrown it off. With a little experience, you'll learn to pick out the real errors quickly.<br />
<br />
Sometimes error messages are obscure. The compiler may be trying to report a very technical issue with your code, but it's not clear what the error means, so you don't know how to fix it. Googling error messages is a useful way to get help interpreting them. You're probably not the first person who had that problem.<br />
<br />
Once you have a successful build, indicated by the absence of error messages, you can run it. Here's where it's useful to start thinking about test cases. Because you want to make sure your code works, right? You'll feel stupid if you let someone else run it and it doesn't work properly.<br />
<br />
In fact, you should think about test cases before you even write the code, so that you write the code in a way that makes it easy to test. <i>Testability</i> affects how you design the code.<br />
<br />
How many possible paths are there through the code? Ideally, you should run the program in a way that exercises each one to test it. That's easy for a simple program like this.<br />
<br />
For more complex software, however, that becomes a big job, and it can be difficult to achieve that ideal. Just identifying all the paths can be tricky. Then coming up with inputs that guarantee you cover all those paths is even trickier.<br />
<br />
It's further complicated by the fact that some paths are meant to handle error conditions that are hard to produce. And what if the code has poor testability? These issues get off into the whole art of testing.<br />
<br />
For this program there are two test cases, going through the two possible paths of the if-else decision:<br />
<ol>
<li>You run the program with no additional arguments.</li>
<li>You run the program with additional arguments.</li>
</ol>
<div>
There's no need to worry about differentiating between 1 additional argument, 2 additional arguments, 3, etc. They all generalize to the single case "with additional arguments". That simplifies testing so you don't have to keep going with 98 additional arguments, 99 additional arguments, 100...</div>
<div>
<br /></div>
<div>
You can identify test cases by the different control structures in the code, the various decisions and loops that it contains. These create a combinatorial set of possible execution paths. That set gets large quickly, known as <i>combinatorial explosion</i>, because real code has lots of control structures to deal with the various combinations of inputs the many functions may handle.</div>
<div>
<br /></div>
<div>
What about the for-loop in this program? Well, we can see logically that the loop gets executed only if there are additional arguments. Testing loops often includes test cases where the loop has nothing to do. In this program, we can see that such a scenario is impossible. So all the possible loop test cases are already covered by the if-else test cases.</div>
<div>
<br /></div>
<div>
Coming up with a suitable set of test cases is very much an art form. An incomplete set of cases risks missing some bug, that then shows up when someone else uses the code. But excess test cases that are redundant just waste time without telling you any further useful information.</div>
<div>
<br /></div>
<div>
I'll cover more about testing later, because it's an important part of being an RPSD. If you do a poor job of testing, it can have consequences for your career. It can also have consequences for the people who depend on your code. It becomes a matter of being ethical and responsible.</div>
<div>
<br /></div>
<div>
If you think that's overblown, think about the problems caused by software failures you've experienced or heard about in the news. The security breaches, the software crashes, the system failures, and the inconvenience, aggravation, frustration, and misery heaped on people's lives as a result.<br />
<br />
I'll be covering more about testing in later posts, but for a separate presentation on it, see <a href="http://flinkandblink.blogspot.com/2017/11/testing-is-how-you-avoid-looking-stupid.html">Testing Is How You Avoid Looking Stupid</a>.</div>
<div>
<br /></div>
<div>
I can exercise the two test cases for this program by running it with and without additional arguments:</div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1px 0.1px 0.1px 0.1px; border: solid gray; overflow: auto; padding: 0.1em 0.2em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9</pre>
</td><td><pre style="line-height: 125%; margin: 0;">$ <span style="color: #00a000;">printargs</span>
Usage: printargs <arguments>
Prints command line arguments.
$ <span style="color: #00a000;">printargs hello world everywhere</span>
0: printargs
1: hello
2: world
3: everywhere
</pre>
</td></tr>
</tbody></table>
</div>
<br />
<b>Lines 1-3:</b> test case 1. As expected, the program prints the usage message, referring to the program name correctly.<br />
<br />
<b>Lines 5-9:</b> test case 2. As expected, the program prints each of the three additional arguments that were on the command line, with the correct 0-based indices. It would have been sufficient to use just one extra argument.<br />
<br />
The process of manually running each test case like this and examining their results is called <i>manual testing</i>. Manual testing a simple program is pretty easy, but even that can get awfully tedious if something goes wrong and you keep having to repeat the tests as you chase down and correct the bugs. That's especially true if you have to carefully scrutinize every line being printed out to check for an unexpected result.<br />
<br />
An RPSD quickly moves from manual testing to <i>automated testing</i>, using some form of <i>test automation</i>. That creates an efficient workflow and gives you a way to repeat the testing later without having to remember all the cases. But that's a topic for another post.<br />
<br />
For now, I've proven to myself that the code works as expected.Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-42879915049890811012017-12-03T15:48:00.000-08:002018-04-19T19:12:47.172-07:00Learn To Code IntroductionThis will be an ongoing series of posts on learning to code, tagged with the label <a href="http://flinkandblink.blogspot.com/search/label/LearnToCode">LearnToCode</a>. You can always get to the series home page by clicking on the <a href="http://flinkandblink.blogspot.com/p/learntocode.html">Learn To Code</a> button in the bar at the top. It has the complete outline and map of the series.<br />
<br />
For <a href="http://flinkandblink.blogspot.com/search/label/Beginner">beginner</a> posts, I'll assume you have no background in computers other than general use. Therefore I'll briefly explain a lot of terms that may be familiar to those with more experience. For <a href="http://flinkandblink.blogspot.com/search/label/Intermediate">intermediate</a> and <a href="http://flinkandblink.blogspot.com/search/label/Advanced">advanced</a> posts, that won't be necessary.<br />
<br />
There are lots of other "learn to code" resources available already, and I encourage you to use them. It's always good to learn from multiple source, because each one has a slightly different emphasis and take on the material.<br />
<br />
The main difference here is that I'm going to teach multiple different languages at once. In fact, I'm going to cover 8 languages. That's <i>my</i> particular take on the material.<br />
<br />
What?!? 8 Languages? Isn't that crazy? Won't that be confusing?<br />
<br />
Yes, that's certainly ambitious and risks confusion. But I'll take advantage of the fact that there are a lot of common concepts between languages. I'll show you how to apply those concepts in the different languages, taking pains to point out the commonality and the differences.<br />
<br />
Why multiple languages? The real secret to a long career is versatility. If you only know how to do one thing, the hot technology of the day, that's great until it's no longer the hot technology of the day. But if you know how to do multiple things, in multiple different ways, you can roll with it and adapt.<br />
<br />
That will also help teach you how to acquire a new language. As I said in my <a href="http://flinkandblink.blogspot.com/2017/11/welcome.html">welcome post</a> to this blog, the most important skill you can develop is the ability to learn new skills.<br />
<br />
There's another very practical reason. As a software developer, you often need to work in multiple different languages at once. Different parts of what you do will be in different environments, where different languages are in use.<br />
<br />
Here are the languages:<br />
<ul>
<li>C</li>
<li>C++</li>
<li>Java</li>
<li>Javascript</li>
<li>Go</li>
<li>Python</li>
<li>Bash</li>
<li>Pseudocode (not a real language, useful for sketching out designs)</li>
</ul>
<br />
On a daily basis in my job, I may work on some C or C++ for one of our embedded devices, some C++ for one of our backend servers, some Python to analyze frames in a video file or automate data retrieval from AWS, or some Bash to automate build steps or file management activities.<br />
<br />
I may have to read someone else's Javascript for another backend server or Java for a mobile app. I'm constantly scribbling stuff in pseudocode as I work through my thoughts.<br />
<br />
It doesn't stop there. I may have to update some Ruby to handle cloud deployment, or work on some Lisp for Emacs macros.<br />
<br />
My facility with each language varies. I'm barely functional with Ruby or Lisp. But a little Googling around and experimentation allows me to get the job done. My experience with other languages helps me out, despite significant differences in them. That's how versatility increases my ability to tackle different problems.<br />
<br />
In the process of teaching you, I'll also learn Java, Javascript, and Go myself.<br />
<br />
Hang on, didn't I just say I may have to read someone else's Java or Javascript code at work? Right, I can read it, based on the common characteristics it shares with the other languages, but I don't know it well enough to write it. Learning to do that means I'll be able to contribute more at work. And I want to learn Go because it's one of the hot technologies of the day, that I believe has a long life ahead of it.<br />
<br />
But if I don't know those languages, how can I teach them? That's another point from my welcome post: teaching is a great way to learn something, because I have to work it out myself well enough to explain it to others, augmented by my experience. See one, do one, teach one.<br />
<br />
How will I do all this? I'll jump right in and start showing you example code, then break those examples down and go through them. Realize that in some cases, the code is written in a particular way to illustrate a specific point. So some of it will be very simplistic. You have to walk before you can run.<br />
<br />
But I'll move on from there into more advanced topics, including:<br />
<ul>
<li><i>Socket programming</i></li>
<li><i>Synchronous</i> vs. <i>asynchronous programming</i></li>
<li><i>Multithreaded programming</i></li>
<li><i>Algorithms</i></li>
</ul>
<br />
I'll cover applying these in a number of real-world systems, ranging from personal computing to big iron to miniaturized devices:<br />
<ul>
<li><i>Desktop applications</i></li>
<li><i>Web apps</i></li>
<li><i>Mobile apps</i></li>
<li><i>Server systems</i></li>
<li><i>Embedded systems</i></li>
</ul>
<br />
I'll also touch on engineering process topics, including:<br />
<ul>
<li><i>Software development life cycle (SDLC)</i></li>
<li><i>Requirements</i></li>
<li><i>Architecture</i></li>
<li><i>Design</i></li>
<li><i>Testing</i> (see <a href="http://flinkandblink.blogspot.com/2017/11/testing-is-how-you-avoid-looking-stupid.html">Testing Is How You Avoid Looking Stupid</a> for a discussion of testing)</li>
<li><i>Source control</i></li>
</ul>
<br />
Why so much? Because I want to show you how to be a <i>Real Practical Software Developer (RPSD)</i>, someone who can really get a job done, not just a <i>Superficial Crappy Software Developer (SCSD)</i>.<br />
<br />
An RPSD has depth. An SCSD just scratches the surface. An RPSD seeks understanding. An SCSD just seeks a quick fix.<br />
<br />
These things are all important to know as an RPSD. Knowing how to code in the languages is just the first step. Making that code do real, useful things, working in a real software development environment, is where it really gets interesting.
<br />
<br />
I'll be introducing lots of terminology, indicated in italics as above. I may throw some terms out there before actually getting around to explaining them. Just remember that terminology can be slippery, having different meaning and usage to different people and in different contexts.
<br />
<br />
I'll also have some posts related to general programming knowledge and skills, like working with <i>binary</i> and <i>hexadecimal numbers</i>.
<br />
<br />
I'll use or refer to a number of online resources and books. The online resources are free, and include tutorials, programming tools, test platforms, and articles. Some of the books are also available as authorized downloadable PDF's that authors and publishers have generously made available for free (some PDF's are unauthorized pirated copies, so be careful what you download).<br />
<br />
And let's make something perfectly clear from the start: there's nothing wrong with using <a href="https://www.google.com/">Google</a> or <a href="https://stackoverflow.com/">StackOverflow</a> to find out how to do something, or as a reminder. As a student learning to code and later as a working RPSD, you'll get to know these as invaluable helpers, along with a few favorite tutorial sites.<br />
<br />
I do that all the time, because I can't possibly know or remember everything. Why should I? I have computers to help me do that. They have a vastly larger store of knowledge than any single human.<br />
<br />
When you look up something in a book or online, the key is not to just copy code blindly, but to take the time to understand what it's telling you, then adapt it to your situation. That's how you stand on the shoulders of others and increase your own store of knowledge.<br />
<br />
I've been doing this for 35 years, and I love it. Building things and seeing other people put them to use gives me the greatest satisfaction. I hope you'll find this series useful as I share what I've learned.<br />
<br />
<b>Comments and Questions</b><br />
<br />
I welcome comments here on the blog and emails at sdbranam at gmail dot com. Those might turn into good topics for further posts.<br />
<br />
Note that I actively filter out comment spam. If you put a link in a comment, the comment must be directly relevant to the post, and the link must either refer to your personal online resources or to something directly relevant to the post. Otherwise I'll delete it.<br />
<br />
(Continue to <a href="http://flinkandblink.blogspot.com/2017/12/first-code.html">First Code</a>)<br />
<ul>
</ul>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-16930221721178039052017-11-23T11:44:00.000-08:002018-07-23T18:52:00.574-07:00Testing Is How You Avoid Looking StupidThis is a presentation I gave at the <a href="http://iot.withthebest.com/">IOT With The Best</a> online conference on October 14, 2017: <a href="https://www.slideshare.net/stevebranam1/testing-is-how-you-avoid-looking-stupid-82190649">Testing Is How You Avoid Looking Stupid</a>.<br />
<br />
The SlideShare includes an embedded YouTube video recording of my original presentation (I typically watch things like this at 1.5 or 2x speed, selectable from the Settings menu in the YouTube window, which helps me maintain focus).<br />
<br />
The abstract:<br />
<blockquote class="tr_bq">
<i>As IOT products become more pervasive, they have an increasing ability to adversely affect the lives of their users and those around them. Testing is the due diligence that closes the engineering loop to verify proper behavior.</i> <i>Steve will present an introductory overview to testing for IOT products, covering the IOT triad: embedded IOT devices, backend servers, and frontend apps.</i> <i>He'll talk about the consequences of inadequate testing for companies and individual contributors, and levels and types of testing.</i></blockquote>
Testing is not an absolute guarantor of quality, and you need to have worked out requirements and design to test against, but without doing it, you'll look stupid.<br />
<br />
Skimping on testing also means you'll make life miserable for someone. Maybe even kill them.<br />
<br />
<b>Books</b><br />
<br />
Doing this presentation turned out to be a bit expensive, because it set me off on a book-buying binge. Fortunately, there's a robust online market in used books.<br />
<br />
This went down three paths. First, I wanted to reference the Toyota unintended acceleration problem as a case study. I was familiar with it from reading <a href="https://catless.ncl.ac.uk/Risks/">Risks Digest</a> (my source for all things safety, reliability, security, and usability).<br />
<br />
What I found was <a href="https://users.ece.cmu.edu/~koopman/">Professor Philip Koopman</a> at Carnegie Mellon University. He was a plaintiff's expert witness in one of the lawsuits, and had put together a <a href="https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf">nice presentation on the problem</a>.<br />
<br />
But it also turned out he had written a book on embedded systems entitled <a href="http://koopman.us/">Better Embedded System Software</a> (available from his site at half off). I ordered the book and read it immediately. It turned out to be a great overview of a broad range of topics on improving embedded system software.<br />
<br />
It also listed a number of other books as recommended reading at the end of each chapter. The thing I like about that is these are curated recommendations, helping select which books to read from the vast ocean of books available and raising awareness of obscure areas.<br />
<br />
Off to Amazon! And then of course those books had additional recommended reading as I started working my way through them, so more books...<br />
<br />
He also has some good videos at his company website, <a href="https://www.edge-case-research.com/videos/">Edge Case Research</a> (he uses Vimeo for his video; I use the <a href="https://chrome.google.com/webstore/detail/vimeo-repeat-speed/noonakfaafcdaagngpjehilgegefdima">Vimeo Repeat And Speed</a> Chrome extension for watching on Vimeo at 2x speed).<br />
<br />
Second, a name that leapt out at me on the speaker's list for the conference was <a href="https://en.wikipedia.org/wiki/Stephen_J._Mellor">Stephen Mellor</a>. Learning the Ward-Mellor method back in the late 80's was an absolute watershed moment for my career. I've applied parts of it informally ever since.<br />
<br />
Three minutes into watching his recorded presentation he mentions that he has a new book out on how to take models directly into code for embedded systems. Stop! Google! <a href="http://www.apress.com/us/book/9781484222164">Book ordered</a>!<br />
<br />
And of course as I started reading that one, it referenced others... These books cover Executable UML, which looks like an excellent follow-on to the Ward-Mellor method (unfortunately, I completely missed the boat on Schlaer-Mellor, but xUML also builds on that). One of the benefits I see in xUML is that it imposes rules and discipline on general UML that provide simplifying structure on what is already an extremely complex endeavor.<br />
<br />
Third, there were several titles in the many Amazon recommendations as I placed orders that looked interesting, especially having been sensitized to some of the topics by the other books.<br />
<br />
It'll take me a while to complete all these, but so far they've been well worth reading, an excellent addition to my bookshelf and another watershed for my career. There will probably be more.<br />
<br />
Here's the full list if you're interested in further reading, organized by reference source:<br />
<ul>
<li>Philip Koopman:</li>
<ul>
<li><a href="http://koopman.us/">Better Embedded System Software</a>, 2010, Philip Koopman.</li>
<ul>
<li><a href="https://www.amazon.com/gp/product/0470068523/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0470068523&linkCode=as2&tag=closegrain-20&linkId=d0c7e25d314e7b0569476a4212f504bc" target="_blank">Security Engineering: A Guide to Building Dependable Distributed Systems</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0470068523" style="border: none !important; margin: 0px !important;" width="1" />, 2008, Ross Anderson.</li>
<ul>
<li><a href="https://www.amazon.com/gp/product/0321356705/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321356705&linkCode=as2&tag=closegrain-20&linkId=d19041db1e63fcbb9d74a9c9a62520f6" target="_blank">Software Security: Building Security In</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0321356705" style="border: none !important; margin: 0px !important;" width="1" />, 2006, Gary McGraw.</li>
<li><a href="https://www.amazon.com/gp/product/0735617228/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0735617228&linkCode=as2&tag=closegrain-20&linkId=e400d7c0f7b88814421a2d6cad4001c1" target="_blank">Writing Secure Code: Practical Strategies and Proven Techniques for Building Secure Applications in a Networked World (Developer Best Practices)</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0735617228" style="border: none !important; margin: 0px !important;" width="1" />, 2004, Michael Howard, David LeBlanc.</li>
</ul>
<li><a href="https://www.amazon.com/gp/product/0131829572/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0131829572&linkCode=as2&tag=closegrain-20&linkId=b1a1cd73d28f9e4399caba85eb175df0" target="_blank">Software Architecture: Perspectives on an Emerging Discipline</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0131829572" style="border: none !important; margin: 0px !important;" width="1" />, 1996, Mary Shaw, David Garlan.</li>
<li><a href="https://www.amazon.com/gp/product/0138803455/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0138803455&linkCode=as2&tag=closegrain-20&linkId=a94377fffe0f09cbe0e2c990a0b89142" target="_blank">Systems Architecting: Creating & Building Complex Systems</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0138803455" style="border: none !important; margin: 0px !important;" width="1" />, 1991, Eberhardt Rechtin.</li>
</ul>
</ul>
<li>Stephen Mellor:</li>
<ul>
<li><a href="http://www.apress.com/us/book/9781484222164">Models To Code: With No Mysterious Gaps</a>, 2017, Leon Starr, George Mangogna, Stephen Mellor.</li>
<ul>
<li><a href="https://www.amazon.com/gp/product/0201748045/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0201748045&linkCode=as2&tag=closegrain-20&linkId=31de933d216aa548d7a8b130242d4649" target="_blank">Executable UML: A Foundation for Model-Driven Architecture</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0201748045" style="border: none !important; margin: 0px !important;" width="1" />, 2002, Stephen J. Mellor, Marc J. Balcer.</li>
<li><a href="https://www.amazon.com/gp/product/0130674796/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0130674796&linkCode=as2&tag=closegrain-20&linkId=22977aad8a4558ba21c7cf2b41af14d4" target="_blank">Executable UML How to Build Class Models</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0130674796" style="border: none !important; margin: 0px !important;" width="1" />, 2002, Leon Starr.</li>
</ul>
</ul>
<li>Amazon recommendations:</li>
<ul>
<li><a href="https://www.amazon.com/gp/product/1498726704/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1498726704&linkCode=as2&tag=closegrain-20&linkId=f13d178c87f628f92b4785cd52d2ef7e" target="_blank">Embedded Software Development for Safety-Critical Systems</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=1498726704" style="border: none !important; margin: 0px !important;" width="1" />, 2016, Chris Hobbs.</li>
<li><a href="https://www.amazon.com/gp/product/1107041090/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1107041090&linkCode=as2&tag=closegrain-20&linkId=0c7054d68729c99f4f7a219072ac6d39" target="_blank">Real-Time Software Design for Embedded Systems</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=1107041090" style="border: none !important; margin: 0px !important;" width="1" />, 2016, Hassan Gomaa.</li>
<li><a href="https://www.amazon.com/gp/product/0321822137/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321822137&linkCode=as2&tag=closegrain-20&linkId=5ca61bdb692be3ea57e2eac82357f49e" target="_blank">Secure Coding in C and C++ (2nd Edition) (SEI Series in Software Engineering)</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0321822137" style="border: none !important; margin: 0px !important;" width="1" />, 2013, Robert C. Seacord. This is particularly interesting because Seacord is the author of the CERT C Secure Coding Standard.</li>
<li><a href="https://www.amazon.com/gp/product/0201703696/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0201703696&linkCode=as2&tag=closegrain-20&linkId=ea638655eb4028c4a09423c2309a75a1" target="_blank">Software Fundamentals: Collected Papers by David L. Parnas</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0201703696" style="border: none !important; margin: 0px !important;" width="1" />, 2001, Daniel M. Hoffman, David M. Weiss, editors.</li>
<li><a href="https://www.amazon.com/gp/product/0321154959/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321154959&linkCode=as2&tag=closegrain-20&linkId=84bc43d01170a9c839a2c3f151d7d1df" target="_blank">Software Architecture in Practice (2nd Edition)</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0321154959" style="border: none !important; margin: 0px !important;" width="1" />, Len Bass, Paul Clements, Rick Kazman. I chose this over the 3rd edition due to the case studies listed.</li>
<li><a href="https://www.amazon.com/gp/product/0321552687/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321552687&linkCode=as2&tag=closegrain-20&linkId=07d6c9b9ccc72c533ac8022c3bb34718" target="_blank">Documenting Software Architectures: Views and Beyond (2nd Edition)</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0321552687" style="border: none !important; margin: 0px !important;" width="1" />, 2011, Paul Clements, et al.</li>
</ul>
<li>While I'm here, four other relevant books that I already had and highly recommend:</li>
<ul>
<li><a href="https://amzn.to/2mFPShO">Computer-Related Risks</a>, 1994, Peter G. Neumann. A compendium of people and companies looking stupid, from the first decade of Risks Digest.</li>
<li><a href="https://mitpress.mit.edu/books/engineering-safer-world">Engineering a Safer World: Systems Thinking Applied to Safety</a>, 2012, Nancy Leveson. The page includes a link to a free PDF download of the book under the "Open Access Title" heading, so there's no excuse for not reading it. This has some absolutely hair-raising case studies, and gives a pragmatic approach to understanding how and why systems fail. You'll never again blame it on "human error".</li>
<li><a href="https://www.amazon.com/gp/product/1578201241/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1578201241&linkCode=as2&tag=closegrain-20&linkId=18dfdfca12bc647b2f3e1f094e8731c0" target="_blank">Real-Time Concepts for Embedded Systems</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=1578201241" style="border: none !important; margin: 0px !important;" width="1" />, 2003, Qing Li, Caroline Yao. This is an excellent broad introduction for anyone new to embedded systems, as well as operating systems concepts for multithreaded systems.</li>
<li><a href="https://www.amazon.com/gp/product/0750687061/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0750687061&linkCode=as2&tag=closegrain-20&linkId=5ab60f9aad1bf5faa955951c26078f2d" target="_blank">Practical UML Statecharts in C/C++: Event-Driven Programming for Embedded Systems</a><img alt="" border="0" height="1" src="//ir-na.amazon-adsystem.com/e/ir?t=closegrain-20&l=am2&o=1&a=0750687061" style="border: none !important; margin: 0px !important;" width="1" />, 2008, Miro Samek. This book is just freakin' brilliant, applying the concepts of UML statecharts in the context of different classes of real-time systems, using the concepts outlined in Li and Yao's book. This could serve as the manual model compiler for xUML.</li>
</ul>
</ul>
<div>
The book of Parnas' writings deserves special mention because <a href="https://en.wikipedia.org/wiki/David_Parnas">Parnas</a> is one of the greats of the field, like <a href="https://en.wikipedia.org/wiki/Donald_Knuth">Knuth</a> and <a href="https://en.wikipedia.org/wiki/Edsger_W._Dijkstra">Dijkstra</a>. Many important concepts in software engineering can be traced to him; references to these papers pop up all over throughout decades of texts (such as in Gomaa's book!).</div>
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0tag:blogger.com,1999:blog-6470064853784686094.post-53721709475582469512017-11-23T07:59:00.001-08:002017-11-23T09:00:50.832-08:00Welcome!Welcome to <i>Flink And Blink</i>, my software engineering blog. I've been developing software since 1982, primarily with a background in networking, such as routers, video streaming servers, and now IOT (Internet Of Things). I work primarily in the embedded and backend spaces, with some dabbling in frontend apps.<br />
<br />
I'm mostly self-taught, which really means I've had many teachers, the many authors of books and articles, the many designers and developers who have built the things I've studied and worked on.<br />
<br />
The most important thing this has taught me is that learning is a never-ending experience. The most important skill you can develop is the ability to learn new skills.<br />
<br />
I love to learn new things, and I'm not afraid to make mistakes doing it. As long as there's no damage and no injury, it's all a learning experience. And a little blood on the deck isn't an injury.<br />
<br />
I've always found that if I have technical information and a system to play on, I can learn how to make it go. Experimentation, both the successes and the failures, is a great learning tool.<br />
<br />
Thomas Edison said, "Genius is one per cent inspiration, ninety-nine per cent perspiration." I temper that with Nikola Tesla's response: "...a little theory and calculation would have saved him ninety per cent of his labor." While it's Edison we all remember, it's Tesla's AC outlets that we plug everything into.<br />
<br />
I like to combine the two approaches into informed experimentation. Try it and see, but think about it first. The analytical and empirical methods make a powerful combination.<br />
<br />
Teaching is also a great way to learn. I have to be able to figure things out if I'm going to explain them.<br />
<br />
The discipline of writing things down and drawing up diagrams forces me to order my thoughts. That then leaves me with something I can pass on to others to share the knowledge. That's part of the see one, do one, teach one methodology.<br />
<br />
I've previously posted software-related things to my woodworking blog, <a href="http://www.closegrain.com/">CloseGrain</a>. I'll cross-post some of those here.<br />
<br />
Why "flink and blink"? Those who are wise in the ways of computer science will recognize these as the forward link and backward link of a <a href="http://en.wikipedia.org/wiki/Double_linked_list">double linked list</a>, one of the fundamental data structures.
Steve Branamhttp://www.blogger.com/profile/10526202082032043903noreply@blogger.com0