Showing posts with label LearnToCode. Show all posts
Showing posts with label LearnToCode. Show all posts

Saturday, September 22, 2018

So You Want To Be An Embedded Systems Developer

Then listen now to what I say.
Just get an electric guitar
and take some time and learn how to play.

Oh, wait, that's a song by the Byrds. But the strategy is the same. Get some information and tools and learn how to use them. No need to sell your soul to the company.

The items I've listed below are sufficient to get you started on a career as an embedded systems developer. There are of course many additional resources out there. But these will arm you with enough knowledge to begin.

I own or have watched every resource and piece of hardware listed on this page. I've either gone through them entirely, or am in the process of doing so. I can vouch for their usefulness in getting you up to speed. It's a firehose of learning.

My personal learning method is to bounce around between multiple books and videos in progress, while spending time hands-on with the hardware. This is similar to a college student juggling multiple classes with labs (without tests, term papers, or due dates!).

Your method may be different. Feel free to approach things in a different order. I offer this in the spirit of sodoto.

What's An Embedded System?

It's a computer that's embedded inside another product, like a car, a microwave, a robot, an aircraft, or a giant industrial machine in a factory; or an IoT device like an Amazon Echo, a Sonos speaker, or a SimpliSafe home security system. You think of the thing as the end product, not as a computer. The computer happens to be one of the things inside that makes it work.

The fascinating thing about embedded systems is that you get to have your hands in the guts of things. The code you write makes a physical object interact with the real world. It's a direct control feedback loop. Working on them is incredibly seductive.

Embedded systems are a multi-disciplinary endeavor. At a minimum they require a mix of electronics and software knowledge. Depending on the particular application (the end product you're building), they may also require some combination of mechanical, materials science, physics, chemical, biological, medical, or advanced mathematical knowledge.

Hobbyist vs. Professional Hardware

There's a wide range of hardware available to learn on, at very reasonable prices. Most of the microcontrollers and boards were originally aimed at the professional community, but advances in technology and falling prices have made them accessible to the hobbyist and educational communities.

Meanwhile, those same advances have enabled hardware aimed directly at the hobbyist and educational communities. Some of that hardware has advanced to the point that it is used in the professional community. So the lines have been blurred.

All of the boards covered here have a variety of hardware interfaces and connectors that allow you to connect up other hardware devices. These are the various sensors, indicators, and actuators that allow an embedded system to interact with the real world.

Two hobbyist/educational platforms are Arduino and Raspberry Pi. For a beginner, these offer a great way to start out. There's an enormous amount of information available on using them from the hobbyist, educational, and maker communities.

I've listed a few books on them below in the Primary Resources, and there are a great many more, as well as free videos and websites. These books tend to be written at a more beginner level than books aimed at professionals.

Arduino is a bare-metal platform, meaning it doesn't run an operating system. An IDE (Integrated Development Environment) is available for free, for writing and running programs on it. You program it with the C and C++ programming languages.

Many of the low-level details are taken care of for you. That's both the strength and the weakness of Arduino.

It's a strength because it offers a quick streamlined path to getting something running. That makes it a great platform for exploring new concepts and new hardware.

It's a weakness because it isolates you too much from the critical low-level details that you need to understand in order to progress beyond the level of beginner.

Those low-level details are the difference between success in the real world and dangerous mediocrity. Dangerous as in you can actually get people killed, so if you want to do this professionally, you need to understand the responsibility you're taking on.

My attitude is to take advantage of that streamlined path whenever needed, and use it to boost yourself into the more demanding work. There are always going to be new pieces of hardware to hook up to an Arduino. I'll always start out at the beginner level learning about them.

In that context, Arduino makes a great prototyping and experimentation platform, without having to worry so much about the low-level details. Then, every bit of knowledge I pick up that way can be carried over to more complex platforms. Meanwhile, Arduino is a perfectly capable platform in its own right.

Raspberry Pi is a Linux platform, meaning it is a single-board computer running the Linux operating system. In some ways it is similar to Arduino, in that many low-level details are taken care of for you.

But it is more capable due to more hardware interfaces and the Linux environment. It can operate as a full desktop computer in the palm of your hand. You program it with the Python, C, and C++ programming languages, as well as others. The Linux capability opens up lots of possibilities.

Many of the same arguments for and against Arduino apply to Raspberry Pi. It also offers a great way to learn Linux and its application to embedded systems. It can be used at the beginner level, but also offers greater range to go beyond that.

Professional hardware, aimed at commercial and industrial use, offers the classic embedded systems development experience. This is where you need to be able to dig down to the low levels. These platforms run both bare-metal and with operating systems.

The operating systems tend to be specialized, especially when the application requires true hard real-time behavior, but also include embedded Linux.

Hard real-time means the system must respond to real-world stimulus on short, fixed deadlines, reliably, every time, or the system fails. For instance, an aircraft flight control system that must respond to a sensor input within 100ms, or the plane crashes. Or a chemical plant process control system that must respond to a sensor within 100ms, or the plant blows up and spews a cloud of toxic chemicals over the neighboring city. Or a rocket nozzle control system that must respond to guidance computer input within 50ms or it goes off course and has to be destroyed, obliterating $800 million worth of satellite and launch vehicle.

Those are what system failure can mean, showing the responsibilities. There are hard real-time systems with less severe consequences of failure, as well as soft real-time systems with looser deadlines and allowable failure cases (such as a smart speaker losing its input stream after 200ms and failing to play music), but it's important to keep in mind what can be at stake.

If your goal is to work professionally as an embedded systems developer, you need to be able to work with the professional hardware. But don't hesitate to use the hobbyist hardware to give you a leg up learning new things. The broad range of experience from working with all of them will give you great versatility and adaptability.

The Primary Resources

The items listed below are all excellent resources that provide the minimum required knowledge for a beginner, progressing up to more advanced levels. If you already have some knowledge and experience, they'll fill in the gaps.

These are well-written, very practical guides. There's some overlap and duplication among them, but each author has a different perspective and presentation, helping to build a more complete picture.

They also have links and recommendations for further study. Once you've gone through them, you'll have the background knowledge to tackle more advanced resources.

The most important thing you can do is to practice the things covered. This material requires hands-on work to really get it down, understand it, and be able to put it to use, especially if you're using it to get a job.

Whether you practice as you read along or read through a whole book first, invest the time and effort to actually do what it says. That's how you build the skills and experience that will help you in the real world.

Expect to spend a few days to a few weeks on each of these resources, plus a few months additional. While they're mostly introductory, some assume more background knowledge than others, such as information on binary and hexadecimal numbers. You can find additional information on these topics online by searching on some of the keywords.

Some of the material can be very dense at first, so don't be afraid to go through it more than once. Also, coming back to something after having gone through other things helps break through difficulties.

Looking at this list, it may seem like a lot. Indeed, it is an investment in time and money, some items more than others. But if you think of each one as roughly equivalent to half a semester of a college course once you put in the time to practice the material, this adds up to about two years worth of focused college education.

That's on par with an Associate degree, or half of a Bachelor's degree. And it will leave you with practical skills that you can put to use on a real job.

These are in a roughly recommended order, but you can go through the software and electronics materials in parallel. You might also find it useful to jump around between different sections of different books based on your knowledge level at the time. Note that inexpensive hardware is listed in the next part of this post, including some of the boards these use.

If you find some of the material too difficult, never fear, back off to the beginner resources. If you find some too simple, never fear, it gets deep. Eventually, it all starts to coalesce, like a star forming deep in space, until it ignites and burns brightly in your mind.

The resources:
You can learn Arduino in 15 minutes.: This is a nice short video that talks about the basics of Arduino microcontroller systems. It helps to start breaking down the terminology and show some of the things involved. That makes it a good introduction to more involved topics. You can also dive down the rabbit hole of endless videos on Arduino, microcontrollers, and electronics from here. This guy's channel alone offers lots of good information.
Hacking Electronics: Learning Electronics with Arduino and Raspberry Pi, 2nd Edition, 2017, by Simon Monk. This is a great beginner-level hands-on book that covers just enough of a wide range of hardware and software topics to allow you to get things up and running, starting from zero knowledge.
Programming the Raspberry Pi: Getting Started with Python, 2nd Edition, 2016, by Simon Monk. This is a nice practical guide to Python on the Raspberry Pi, with much more detail on programming than his Hacking Electronics above. Meanwhile it has less beginner information on hardware. So the two books complement each other nicely.
Programming Arduino: Getting Started with Sketches, 2nd Edition, 2016, by Simon Monk. Similar to his book on Python, but for C on Arduino, also a nice complement to his Hacking Electronics.
Embedded Software Engineering 101: This is a fantastic blog series by Christopher Svec, Senior Principal Software Engineer at iRobot. What I really like about it is that he goes through things at very fine beginner steps, including a spectacular introduction to microcontroller assembly language.
Modern Embedded Systems Programming: This is a breathtakingly spectacular series of short videos by Miro Samek that take you from the ground up programming embedded systems. They're fast paced, covering lots of material at once, including the C programming language, but he does a great job of breaking things down. He uses an inexpensive microcontroller evaluation kit (see hardware below) and the free size-limited evaluation version of the IAR development software suite. He also has a page of additional resource notes. What I really like about this is that in addition to covering a comprehensive set of information with many subtle details, he shows exactly how the C code translates to data and assembly instructions in microcontroller memory and registers. In contrast to Arduino, this is all the low-level details. You will know how things work under the hood after this course (currently 27 videos). Along the way you'll pick up all kinds of practical design, coding, and debugging skills that would normally take years to acquire. Did I mention this course is freakin' awesome?
RoboGrok: This is an amazing complete online 2-semester college robotics video course by Angela Sodemann at Arizona State University, available to the public. Start with the preliminaries page. In addition to some of the basics of embedded systems, it covers kinematics and machine vision, doing hands-on motor and sensor control through a PSoC (Programmable System on a Chip) board. She sells a parts kit, listed below. This is a great example of applied embedded systems.
C Programming Language, 2nd Edition, 1988, by Brian W. Kernighan and Dennis M. Ritchie: C is the primary language used for embedded systems software, though C++ is starting to become common. This is the seminal book on C, extremely well-written, that influenced a generation of programming style and other programming books. The resources listed above all include some basics of C, and this will complete the coverage.
Embedded C Coding Standard, 2018 (BARR-C:2018), by Michael Barr: This will put you on the right track to writing clean, readable, maintainable code with fewer bugs. It's a free downloadable PDF, which you can also order as an inexpensive paperback. Coding standards are an important part of being a disciplined developer. When you see ugly, hard to read code, you'll appreciate this.
Programming Embedded Systems: in C and C++, 1999, by Michael Barr: Even though this is now 20 years old, it's a great technical introduction and remains very relevant. Similar in many respects to Samek's video series, it takes a beginner through the process of familiarizing yourself with the processor and its peripherals, and introduces embedded operating system concepts. There is a later edition available, but this one is available used at reasonable prices. 
Programming Arduino Next Steps: Going Further with Sketches, 2nd Edition, 2019, by Simon Monk. This goes deeper into Arduino, covering more advanced programming and interfacing topics. It also includes information on the wide array of third-party non-Arduino boards that you can program with the IDE. This starts to get past the argument that Arduino is just for beginners doing little toy projects.
Making Embedded Systems: Design Patterns for Great Software, 2011, by Elecia White. This is an excellent book on the software for small embedded systems that don't use operating systems (known as bare-metal, hard-loop, or superloop systems), introducing a broad range of topics essential to all types of embedded systems. And yes, the topic of design patterns is applicable to embedded systems in C. It's not just for non-embedded systems in object-oriented languages. The details of implementation are just different. 
Exploring Raspberry Pi: Interfacing to the Real World with Embedded Linux, 2016, by Derek Molloy. This goes into significantly more depth on the Raspberry Pi and embedded Linux. It's quite extensive, so is best approached by dividing it into beginner, intermediate, and advanced topics, based on your knowledge level at the moment. Spread out your reading accordingly. It has great information on hardware as well as software, including many details of the Linux environment. Two particularly fascinating areas are using other microcontrollers such as Arduino as slave real-time controllers, and creating Linux Kernel Modules (LKMs).
Make: Electronics: Learning Through Discovery, 2nd Edition, 2015, by Charles Platt. This is hands down the best book on introductory electronics I've ever seen. Platt focuses primarily on other components rather than microcontrollers, covering what all those other random parts on a board do. See Review: Make: Electronics and Make:More Electronics for more information on this and the next book, and Learning About Electronics And Microcontrollers for additional resources.
Make: More Electronics: Journey Deep Into the World of Logic Chips, Amplifiers, Sensors, and Randomicity, 2014, by Charles Platt. More components that appear in embedded systems.
Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems, 2006, David J. Agans. By now you've found many ways to get into trouble with code and hardware. This is a fantastic book for learning how to get out of trouble. It's a simple read that outlines a set of very practical rules that are universally applicable to many situations, then elaborates on them with real-life examples. 
Real-Time Concepts for Embedded Systems, 2003, by Qing Li and Caroline Yao. This is an introduction to the general concurrency control mechanisms in embedded operating systems (and larger-scale systems).
Reusable Firmware Development: A Practical Approach to APIs, HALs, and Drivers, 2017, by Jacob Beningo. This covers how to write well-structured low-level device driver code in a way that you can use on multiple projects. Embedded systems are notorious for having non-reusable low-level code, written in a way that's very specific to a specific hardware design, which can often ripple up to higher levels. That means you have to rewrite everything every time for every project. Good Hardware Abstraction Layers (HALs) and Application Programming Interfaces (APIs) provide a disciplined, coherent approach that allows you to reuse code across projects, saving you enormous amounts of time in development and testing. This also helps you become a better designer, because it encourages you to think in a modular way, starting to think in terms of broader architecture in a strategic manner, not just how to deal with the immediate problem at hand in a tactical manner.
Embedded Systems Architecture, 2018, by Daniele Lacamera. This is a very up-to-date book that uses the popular ARM Cortex-M microcontroller family as its reference platform. That makes it a great complement to Samek's video series, since the TI TIVA C that he uses is an ARM Cortex-M processor. This also goes into more detail on areas  such as the toolchain (including debugging with OpenOCD), bootloading, and memory management. It briefly uses the ST STM32F746 Discovery board as an example.
Embedded Systems Fundamentals with Arm Cortex-M based Microcontrollers: A Practical Approach, 2017, by Alexander G. Dean. As the name indicates, this is another detailed book on ARM Cortex-M, intended as a college-level textbook. Among other good practical details, it includes a nice chapter on analog interfacing. It uses the inexpensive NXP FRDM-KL25Z development board for hands-on examples.
TI Tiva ARM Programming For Embedded Systems: Programming ARM Cortex-M4 TM4C123G with C, 2016, by Muhammad Ali Mazidi, Shujen Chen, Sarmad Naimi, and Sepehr Naimi. This is a detailed book that uses the exact same Tiva C board as Samek's video series. 
Designing Embedded Hardware: Create New Computers and Devices, 2nd Edition, 2005, by John Catsoulis. This covers the hardware side of things, an excellent complement to White's book. It provides the microcontroller information to complement Platt's books.
Test Driven Development for Embedded C, 2011, by James Grenning. This is a spectacular book on designing and writing high quality code for embedded systems. See Review: Test Driven Development for Embedded C, James W. Grenning for full details. Just as White's book applies concepts from the OO world to embedded systems, Grenning applies Robert C. Martin's "Clean Code" concepts that are typically associated with OO to embedded systems. We'll all be better off for it.
Modern C++ Programming with Test-Driven Development: Code Better, Sleep Better, 2013, by Jeff Langr. This is an equally spectacular book on software development. It reinforces and goes into additional detail on the topics covered in Grenning's book, so the two complement each other well. Even if you don't know C++, it's generally easy enough to follow and the material still applies.
Taming Embedded C (part 1), 2016, by Joe Drzewiecki. This YouTube video is part of the Microchip MASTERs conference series. It covers some of the things that can be risky in embedded code and some methods for avoiding them. This gets into the characteristics that make embedded systems more challenging. I like to watch videos like this at 2X speed initially. Then I go back through sections at normal speed if I need to watch them more carefully.
Interrupt and Task Scheduling - No RTOS Required, 2016, by Chris Tucker. Another MASTERs video, this covers a critical set of topics for working in embedded systems.
Some Advanced Resources

Ready to dig in further and deeper?
MC/OS the Real-Time Kernel, 1992, by Jean Labrosse. Labrosse decided to write his own real-time operating system when he had trouble getting support for a commercial one he was using. The rest is history. You can hear some of that history in this podcast interview with him, "How Hard Could It Be?". This not only explains how things work under the hood, it gives you the source code.
MC/OS III, The Real-Time Kernel for the Texas Instruments Stellaris MCUs, 2010, by Jean Labrosse. This covers the 3rd generation of MC/OS, as well as details on the Stellaris microcontroller covered in Samek's video series. You can also download a free PDF version of this, as well as companion software. The MC/OS II and other books are also available there. The value in getting multiple versions is to see how the software evolved over time.
Software Engineering for Embedded Systems: Methods, Practical Techniques, and Applications, 2nd edition, 2019, edited by Robert Oshana and Mark Kraeling. This is a broad survey of topics by various authors (Labrosse wrote the chapter on real-time operating systems).
Some Hardware

The items listed below include some of the inexpensive boards and evaluation kits used in the resources above. There are a bazillion microcontroller boards out there that are useful for learning how to work on embedded systems. It's worth getting some from different vendors so you can learn their different microcontrollers, different capabilities, and different toolchains.

That also helps you appreciate the importance of abstracting low-level hardware differences in writing your code. Each vendor provides a range of support tools as part of the package.

Note that large vendor websites can be a pain, because they want you to create an account with profile, asking questions like your company name (call it "Independent"), what your application is, how many zillion parts you expect to order, when you expect to ship your product, etc. They're setup for industrial use, not hobbyist/individual use. They also may work through distributors like Mouser or Digi-Key for shipping and orders. Just roll with it!

The hardware:
Arduino Uno - R3, $22.95. This is the board used in the Arduino video listed above. There's also a wide array of "shields" available, external devices that connect directly to the board. Exploring these is one of the great educational values that Arduino offers. Remember that because Arduino takes care of many of the details for you, you can be up and learning about new devices faster. Then you can take that knowledge and apply it to other boards. You can also download the Arduino IDE there.
Raspberry Pi 3 - Model B+, $35. This is an updated version of the boards used in Simon Monk's books above. You will also need the 5V 2.5A Switching Power Supply with 20AWG MicroUSB Cable, $7.50, and the 8GB Card With full PIXEL desktop NOOBS -  v2.8. You may also want the Mini HDMI to HDMI Cable - 5 Feet, $5.95, and the Ethernet Hub and USB Hub w/ Micro USB OTG Connector, $14.95. These are sufficient to connect it to a monitor, keyboard, and mouse, and use it as a desktop Linux computer.
Texas Instruments MSP430F5529 USB LaunchPad Evaluation Kit, $12.99 (16-bit microcontroller). An evaluation kit is a complete ready-to-use microcontroller board for general experimentation. Christopher Svec uses this kit in his blog series above, where he also covers using the free downloadable development software. If you buy directly from the TI site, register as an "Independent Designer".
Texas Instruments Stellaris LaunchPad Evaluation Kit, was $12.99 (32-bit microcontroller). This is the kit Miro Samek started out with in lesson 0 of his video series above. However, as he points out at the start of lesson 10, TI no longer sells it, and has replaced it with the Tiva C LaunchPad, which is an acceptable replacement (see next item below). 
You might be able to find the Stellaris offered by third-party suppliers. But you have to be careful that you're actually going to get that, and not the Tiva C kit, even though they list it as Stellaris. I now have two Tiva C boards because of that, one that I order directly from TI, and one that was shipped when I specifically ordered a Stellaris from another vendor.
Fortunately, that doesn't matter for this course, but it highlights one of the problems you run into with embedded systems, that vendors change their product lines and substitute products (sometimes it's just rebranding existing products with new names, which appears to be what TI did here). That can be confusing and annoying at the least, and panic-inducing at the worst, if something you did in your project absolutely depends on a hardware feature of the original product that's not available on the replacement.
One of the design lessons you should learn is to future-proof your projects and try to isolate hardware-specific features so that you can adapt to the newer product when necessary.
Texas Instruments Tiva C TM4C123G LaunchPad Evaluation Kit, $12.99 (32-bit microcontroller). This is TI's replacement for the Stellaris LaunchPad, that you can use with Miro Samek's video series. Samek addresses the replacement issue at the beginning of lesson 10. The good news is that he says the Tiva C is equivalent to the Stellaris (apparently all TI did was rename the product), so it's usable for the course. You'll notice that some parts of the toolchain (the software you use to develop the software for the board, in this case the IAR EWARM size-limited evaluation version) still refer to it as TI Stellaris. 
The specific TI device on the board is the TM4C123GH6PM, so when you set the EWARM Project->Options->General Options->Device, you can select TexasInstruments->TM4C->TexasInstruments TM4C123GH6PM, not theLM4F120H5QR that's on the Stellaris board. However, Samek shows that you can continue to use the toolchain configured for Stellaris.
That's one of those details that can be maddening when vendors swap parts around on you. Getting it wrong can produce subtle problems, because some things may work fine (you selected a device variant very similar to the one you need), but others won't. Welcome to the world of embedded development! Small details matter. The alphabet soup and sea of numbers in the product names can also drive you batty and be a source of mistakes. PAY CLOSE ATTENTION!
A related detail: the file lm4f120h5qr.h that Samek supplies in his projects for working with the Stellaris board's processor also works with the Tiva C board's processor. However, there is also a TM4C123GH6PM.h file for the Tiva processor. Both files are in the directory C:\Program Files (x86)\IAR Systems\Embedded Workbench 8.2\arm\inc\TexasInstruments (or whichever version of EWARM you have).
You can copy them to your project directory, or have the compiler use that directory as an additional include directory by selecting Project->Options->C/C++ Compiler and clicking the ... button next to the "Additional include directories:" box.
STMicroelectronics STM32F746 Discovery Board, $54 (ARM Cortex-M7 microcontroller). This is used briefly in Daniele Lacamera's book above. It's relatively expensive compared to the other evaluation kits here, but includes a 4.3" LCD capacitive touch screen and other hardware elements, making it a much more capable platform, and still an outstanding value.
NXP Semiconductor FRDM-KL25Z Freedom Development Board, $15 (ARM Cortex-M0+ microcontroller). This is the board that Alexander Dean uses in his book above.
uC32: Arduino-programmable PIC32 Microcontroller Board, $34 (Microchip PIC32 32-bit processor). This isn't covered specifically by any of the resources above, but the PIC32 microcontroller is a popular family that offers a different hardware environment. This is programmable using the Arduino IDE, and can also be programmed using Microchip's MPLAB IDE.
Adafruit Parts Pal, $19.95. This is a small general parts kit for working with the various boards above. It includes LEDs, switches, resistors, capacitors, simple sensors, a small breadboard, and jumper wires for interconnecting things, plus a few other interesting items.
RoboGrok parts kit, $395. This is the parts kit for Angela Sodemann's course above. While you can gather the parts yourself for less, she saves you all the work of doing that, and buying her kit is a nice way of compensating her. 
Extech EX330 Autoranging Mini Multimeter, $58. There are also a bazillion multimeters out there. This is one reasonable mid-range model. The multimeter is a vital tool for checking things on boards.
One of the following logic analyzers. A logic analyzer is an incredibly valuable tool that allows you to see complex signals in action on a board. They used to cost thousands of dollars and need a cart to roll them around. These are miraculously miniaturized versions that fit in your pocket, at a price that makes them a practical, must-have personal tool. They plug into your USB port and are controlled via free downloadable software you run on your computer:
Saleae Logic 8 Logic Analyzer, 8 D/A Inputs, 100 MS/s, $199 with awesome "enthusiast/student" discount of $200, using a discount code that you can request and apply to your cart when checking out, thanks guys! This is also covered briefly in Svec's blog series. You can play with the software in simulation mode if you don't have an analyzer yet.
Digilent Analog Discovery 2, 100MS/s USB Oscilloscope, Logic Analyzer and Variable Power Supply, Pro Bundle, $299. As amazing as the Saleae is, this one adds oscilloscope, power supply, and signal generator functions, combining a number of pieces of equipment into one tiny package. They also have academic discounts for those who qualify (36% discount on the base unit).
For a full shopping list to equip a personal electronics lab, see the Shopping List heading at Limor Fried Is My New Hero. That page also has many links to resources on how to use the tools.

Glossaries

It can be a bit maddening as you learn the vocabulary, with lots of terms, jargon, and acronyms being thrown around as if you completely understood them. As you get through the resources, the accumulation of knowledge starts to clarify things. Sometimes you'll need to go back and reread something once you get a little more information.
Other Links

These sites have articles and links useful for beginners through advanced developers.
Final Thought

Our society is becoming more and more dependent on embedded systems and the various backend and support systems they interact with. It's our responsibility as developers to build security in to make sure that we're not creating a house of cards ready to collapse at any moment. Because people's lives can depend on it.

If you think I'm overstating that, see Bruce Schneier's new book. We are the ones on the front lines.

Learning About Electronics And Microcontrollers


Tutorial books: to read beginning to end. "Make: Electronics" is the best introduction to electronics I've ever seen in any form, a must read!


Reference books: to jump around and get more details as necessary.

If you're new to electronics and microcontrollers and want to learn about them, here's my recommended reading list, in order (Amazon links)
There's been an evolutionary leap forward in the affordability and ease of use of microcontrollers as a result of inexpensive open-source hardware and free open-source software.

These have removed many of the traditional roadblocks that made learning to use microcontrollers daunting. Working with them is now within reach of everyone from elementary school children to adults.

Microcontrollers are then excellent platforms for learning to code, because they allow you to interact directly with the physical world. Making the hardware do what your code told it to do is very satisfying.

Tutorial Books

First Two Books

These cover basic electronics, and make an outstanding starting point. You can read my review of them to see why I like them so much (as well as information on components kits for the experiments in the first book).

They don't focus on microcontrollers. Instead, they focus on the other parts that surround microcontrollers, as well as projects that don't need microcontrollers.

If you're impatient to get on to the microcontroller information, save Make: More Electronics until later. But you should definitely start with Make: Electronics no matter what.

Third Book

This briefly covers some of the same basics as the first two, then covers some other useful basics. The first two and this one complement each other very well.

It then gets into working with Arduino and Raspberry Pi microcontrollers. It includes a number of simple projects for working with external modules and sensors.

Arduino is programmed in C on "bare metal", i.e. without an operating system, and Raspberry Pi is programmed in Python on embedded Linux, so the two illustrate the variety of programming and runtime environments for microcontrollers.

I found this to be a nice gentle introduction to the practicalities of working with microcontrollers and the huge array of third-party modules available. While it only skims the surface of a vast topic, it makes an excellent jumping off point for learning about embedded systems.

Reference Books

First Three Books

These gather in one place information from a wide array of resources on how to use a wide array of electronic components. They focus on practical concerns rather than theory, and are illustrated with the same excellent color diagrams and photos as Make: Electronics.

For each component, they contain the following sections: What It Does, How It Works, Variants, Values, How To Use It, and What Can Go Wrong.

Fourth Book

This is like multiple smaller books bound into one. It starts with an extensive chapter on theory and related math. The authors point out that much of the math throughout the book is simply to prove the theory, so if you're not interested in that level of detail, you can skip over it.

The remainder of the book covers a broad range of devices, providing both theory and practical material. It has a chapter on microcontrollers that makes a good follow-up to Hacking Electronics.

Using The Books

The tutorial books are meant to be read from beginning to end as you tinker with their projects. They are easy reading with hands-on experiments, where each chapter builds on previous material.

The reference books are meant to read here and there, jumping around as you need more details on a specific topic.

Even though some of the topics are duplicated between all the books, each author and each book has a different perspective. Each has a different emphasis and presentation.

They complement each other to give a more complete picture because one author may delve deeper into details that another glosses over. You may prefer one author's explanation over another's. No single resource is ever able to give the whole story, so it helps to have multiple perspectives.

These books will give you a good foundation so that you'll be able to understand other books and resources.

If you're interested in doing embedded systems software development, see So You Want To Be An Embedded Systems Developer.

Electronics Suppliers

There are two outstanding suppliers of discrete electronics, microcontrollers, tools, modules, sensors, and breakout boards that cater to the small-scale needs of hobbyists, students, and experimenters:
You can read my paean to Adafruit at Limor Fried Is My New Hero,which includes the shopping list for setting up my small-scale electronics lab. For a simple example of using this equipment, see First Use Of New Tools.

There are several suppliers for industrial scale, but who also supply at small scale (do you need 10 pieces, or 10 million?):
Supplier Learning Resources

All these suppliers have extensive online learning resources. However, the industrial suppliers don't have much for the absolute beginner; they're good once you've built up some background knowledge.

Adafruit Learn and SparkFun Learn resources are in both written and video form. You can scan through videos for a quick overview pass by setting the speed in the YouTube window settings (the gear icon) to 2x, then come back and watch at normal speed for a second pass.

There's a lot of duplication between them (and between these resources and the books above), but it's useful to see how different people approach the same topics. Just like reading books by different authors, they provide additional perspectives to help fill in the gaps.

Both sites can be a bit overwhelming to dig through, so I've selected a number of beginner resources below, organized by supplier and then type of resource. Many of them have links to additional information.

These Adafruit videos by Collin Cunningham cover basic electronics lab skills:
  • Soldering and Desoldering: how to solder components together properly, and how to pull them apart for salvage and rework.
  • Surface Mount Soldering: how to solder surface-mount components.
  • Multimeters: how to use a meter for basic measurements.
  • Oscilloscopes: how to use an oscilloscope for advanced measurements and waveforms.
  • Hand Tools: the basic hand tools used for assembling and disassembling electronics.
  • Schematics: how to read schematics (no, they're not Greek!).
  • Breadboards and Perfboards: how to combine the parts on a schematic into a functioning circuit.
  • Ohm's Law: understanding the relationship between voltage, current, and resistance.
He also has these videos on the basics of various components:
These are Adafruit written guides by various contributors:
These SparkFun videos by Shawn Hymel cover basic electronics lab skills:
He also has these videos on electronics basics:
These are SparkFun written guides by various contributors:
Another great YouTube resource is Dave Jones' EEVblog.

Saturday, April 21, 2018

Sodoto: See One, Do One, Teach One

Here's a useful strategy on this learning path: see one, do one, teach one. Sodoto.

Sodoto is a learning method and a teaching method rolled into one. It's the cycle of knowledge.

I'm familiar with it from medicine. My wife is a surgical nurse, and this is the traditional method of teaching in surgery. Obviously, safety concerns mean that you don't just watch a brain surgeon at work and then go try it yourself.

But this forms a useful pattern of mentoring and learning and passing knowledge along. It applies to any kind of knowledge- or skill-based activity.

It works with a single student at a time, or a whole group. You don't have to be a formal teacher.

See one: watch someone do a procedure.

Do one: do what you saw.

Teach one: show it to someone else.

Once you learn a procedure, you're primed to teach it. That's how knowledge spreads.

Here's the real kicker: the teaching step is actually a powerful learning step for you as the teacher. It locks the knowledge into your brain.

You have to have sorted out what you're talking about in order to teach it. You can't just vaguely know it and wave your hands in the air glossing over details. Your students will be annoyed and you'll feel stupid.

The process of getting ready to teach and then doing the teaching forces you to organize your thoughts and chase down details, because you don't want to look stupid, and you want to be prepared for questions.

That motivates you to dig deeper. As a result, you end up learning more yourself.

There are two keys to making this work: background knowledge, and the experience of doing it.

Background Knowledge

Background knowledge applies at each stage of see, do, and teach. Note that "see" can mean live and in person, or on video.

Whatever the subject, medicine, coding, climbing, sailing, scuba diving, physical fitness, martial arts, building anything from woodworking to electronics, any knowledge you have before seeing the procedure will help you understand it.

You can bet that surgeon learning how to do brain surgery brought a huge amount of background knowledge.

Some things take minimal background, just the random skills and knowledge you already have from life. But more difficult subjects benefit from whatever time you can invest beforehand. Videos, books, blogs, and articles, in print and online, are all good resources, as well as online forums.

That establishes the background knowledge you'll bring to seeing the procedure.

Once you've seen the procedure, as you prepare to do it yourself, it's useful to go back to your resources. Now that you know better what to look for, you can get more details. You can reinforce what you saw.

That expands the background knowledge you'll bring to doing the procedure.

Once you've done the procedure, as you prepare to teach it, go back to your resources again. As a result of doing, there will be details you want to fill in, and you may understand the material better. You may have run into some things that you wished you knew more about. You may anticipate additional questions from your students.

That further expands the background knowledge you'll bring to teaching the procedure.

Experience Of Doing

The experience of doing the procedure is critical. That's where you have the opportunity to work through mistakes and see what works and doesn't work for you. That's where you start to lock it into your brain.

Don't be afraid to make mistakes! Mistakes are great learning opportunities. As long as there's no injury and no damage, there's no harm done. And a little blood on the deck isn't an injury.

This is also where you can work out your own changes to the procedure. Just because you saw it done one way doesn't mean that's the only way to do it. That was one way. You can use it as your starting point, and add your own tweaks.

Or maybe you'll realize what you saw really was a good way and you shouldn't mess with it.

You might need to do the procedure more than once before teaching it. Some procedures take practice before you feel confident teaching them to someone else.

The experience of teaching the procedure will be different from the experience of doing it for yourself. Your students may have questions or difficulties that force you to think about things in different ways.

Plus there's the pressure of performing for an audience. But as you gain experience teaching, that will get easier. It's just a different kind of doing.

The experience of teaching is where you finish locking it into your brain.

Teamwork

Sodoto is a great method for dividing up a project. Whether at work, at school, or with your friends, you can divide up the project and have each person take on a part.

They go off and see how it's done, do it themselves until they feel ready, then bring it back to the team to teach everyone else.

What if you can't agree how to divide it up because multiple people want to do the same thing? Fine! Let them!

Each person will have their own take on the experience and teach it slightly differently. That helps explore all the possibilities in the procedure.

Tuesday, April 17, 2018

More C+-

In The Case For C+-, I talked about writing quick tools in a simple C style, but taking advantage of the C++ standard library, primarily the dynamic data structures. It ends up being C++ without any (or just a few) user-defined classes, so is something of a lightweight object-oriented approach (yes, yes, I'm sure OO purists are barfing at the thought). The main benefit is fast coding.

There I showed as an example the msgresolve tool, which I used to resolve messages logged by an IOT device (the client) and its server. This is a lot of string processing and cross-indexing, with logs containing potentially thousands or tens of thousands of messages.

Shortly after I had completed msgresolve, I needed to have a tool to help me sift through large text files of server logs, logging the TCP connections made by clients and their subsequent activity. I was chasing down a problem where some of the connections were shutting down sooner than expected.

I wasn't sure what was causing the early shutdowns, and wasn't even sure initially which connections had experienced it, so I wanted to be able to gather all the lines for a given connection and list them out for tracing through, for each connection.

That would help me identify the ones that were live at the end of the log sample vs. the ones that had ended early. The log entries for hundreds of connections were all intermixed.

Armed with the methods I had used in msgresolve.cpp, conceptual design was easy. I wanted an ordered list of connections, and associated with each one, the sequential list of log entries associated with the connection.

There were also connections with some internal addresses I wanted to ignore. I could have done this filtering with grep, but it was easy enough to build the capability into the program so that it could stand alone. That also helped me explore some additional string processing functions.

Given that architecture, the data structure I needed was a std::map that mapped a string (the connection identification) to a std::list of strings (the log lines for the connection).

I had the program working in less than an hour. Then I spent at least another hour screwing around with the timestamps in the log entries, figuring out how to process them and deciding what to do with them. Then a little more time on refactoring and cleanup.

Throughout, I used a sample log file that had entries for several connections, including addresses I wanted to skip. I used that as a simple unit test to exercise the code paths.

The resulting code provided the impetus for a simple generalized string processing module, which I'll cover in another post. But you can see some clear patterns emerging in this code.

Doing quick tools like this is fun and very satisfying. It makes your serotonin flow. You have a problem you need to deal with, so you sit down and spew a bunch of code in a short time, refine it, and use the results.

This is actually quite different from long-term product development. That kind of work has its intense coding phases, but once the initial version of the product is out, a lot of the work is much smaller surgical changes.

Even fitting a major new feature in often involves many small bits of code scattered throughout the larger code base, integrating the tendrils. Getting that to work has a different kind of satisfaction.

Design Choices

These tools also give you a chance to think about different approaches. You can balance the variables of memory consumption, CPU consumption, I/O consumption, time, and code complexity (that is, ease of writing and maintaining the code, and compiled code space consumption, not algorithmic complexity) for a given situation.

For instance, the log files I was dealing with had over a million lines of data, some 200MB worth covering hundreds of connections.

That meant I had several choices:
  1. I could load all the data into memory and then print it out in an orderly manner. This is a single-pass solution, that consumes large amounts of memory.
  2. I could scan the file once, identifying all the individual connections, then for each connection, scan the file from beginning to end to read and print their lines. This a multi-pass solution that requires little memory but significant file I/O.
  3. I could scan the file once, and for each identified connection, track the file position of the first and last line, then for each connection, just scan that range of the file. This is a multi-pass solution that reduces the total file I/O for a negligible increase in memory.
  4. I could do the same thing, but instead of tracking just the first and last line file positions, build a list of the file position and length of each line, then on each pass, just skip directly to the locations of the lines. This is still multi-pass, but significantly reduces the total file I/O because it only visits each file position twice, requiring a bit more complexity and a bit more memory.
The decision on which choice to use is system-dependent. If memory is cheap and plentiful, and file I/O is relatively expensive, either in terms of time or charges to transfer data over a data link (maybe the data is remote, accessed over a cellular link), then the single-pass solution is best, choice #1.

On a small-memory system, a multi-pass solution is better, and you just have to live with the extra I/O. In that case, choice #4, which is the most complicated code, has the best compromise of low memory and low I/O consumption.

Although if you're really pressed for code space, the simplest multipass solution that iterates over the entire file for each connection is the better choice, #2.

Realistically though, you don't run tools such as this locally on small systems. Where possible, you offload the data to a larger system and run the tools offline.

In this case, I'm running on a Mac with 16GB of memory. Slurping up a 200MB text file and holding everything in memory is nothing. So the single-pass solution is the way to go.

Loading The Data

The log file may just be a chronological sample of all the activity logged by the server, so some connections will already exist at the start of the log, and some will remain at the end. Meanwhile, connection starts and terminations will appear in the log.

The program parses the lines from the log to find connection lines (i.e. some activity for a given connection). It identifies them by looking for a connection ID, which consists of an IPv4 address/port pair (a remote socket ID), and may optionally include a hexadecimal client ID.

It uses just the socket ID as the connection ID, which is the key to the connection map. When it finds a new connection ID, it adds it to the map with an empty list. For each connection line, it finds the connection map entry and appends the line to the list of lines for that connection.

As it loads connection lines, it filters them against the set of IP addresses to skip (these skip address are due to logging of other types of connections besides the client connections). That helps reduce the noise from a large log file.

Printing The Data

The program prints the connections by first iterating through the map and printing out a summary of each one: the connection ID, the number of lines, the duration of the connection data found in the log, and how its lifetime relates to the overall log. Since the map is an ordered structure, printing connections is always in sorted order by connection ID (though string sorted, not numeric sorted).

Then it makes a second pass through the connection map, iterating through each line for each connection and printing it out, with separators between connections. It prints a header and summary line before and after the activity lines for each connection, showing where the connection starts and ends relative to the start and end of the log.

As a last quick change, I added a threshold time value to clearly identify connections that ended at least 60 seconds before the end of the log. This would be a good candidate for a command-line parameter to override the default threshold.

All the output lines have unique greppable features or keywords so you can use other tools for additional postprocessing or extraction. For instance, I could grep out the end summary line of each connection, and maybe the last couple of activity lines before it, to see how each connection ended up. I could use the "threshold exceeded" indication to identify the ones that had ended early.

Some Design Evolution

This program adds the isMember() function, which determines whether a string is a member of a set of strings. Since my usage here was intended to deal with a small set and I had other functions that had similar iterative structure, in the heat of battle I quickly coded it as a linear search of a vector of strings.

That worked fine here, but as I pulled a bunch of this code out into a general string processing module, I realized that was a bad choice, because it's an O(N) search.

That became especially bad when I wanted an overload that took a vector of strings and determined if they were all members. That meant an O(M) repetition of O(N) searches: an O(M*N) or effectively O(N^2) algorithm.

That gets out of hand fast as M and N get larger. Meanwhile, the std::unordered_set is perfect for this, an O(1) algorithm for single searches, and an O(M) algorithm when repeated for M items.

I've left the original isMember() implementation here as an example of the evolution of a concept as you generalize it for other uses.

I also threw in a few overloads that I didn't end up using, but that set the stage for running with the concept in the string processing module. More discussion of that in the post containing the module.

The Result

This program turned out to be another fun exercise in string processing once I had built the basic support functions and could see the problem in those terms. It felt more like working in Python, and in fact just as the dict structure from msgresolve.cpp was inspired by Python, so are the split() and join() functions here.

The funny thing is that it took doing stuff in Python to make me see this approach. That points out one of the advantages of working in multiple different languages: you start seeing opportunities to apply some of the common idioms from one language in another language.

Here's logsplit.cpp:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
// Usage: logsplit <serverLog>
//
// Splits a server log file by IPv4 connection. Prints a
// summary list of the connections, then the log lines for
// each separate connection.
//
// This is an example of a C++ program that is written mostly
// in plain C style, but that makes use of the container and
// composition classes in the C++ standard library. It is a
// lightweight use of C++ with no user-defined classes.
//
// 2018 Steve Branam <sdbranam@gmail.com> learntocode

#include <iostream>
#include <iomanip>
#include <sstream>
#include <vector>
#include <list>
#include <map>

enum ARGS
{
    ARGS_PROGNAME,
    ARGS_SERVER_LOG,
    ARGS_REQUIRED,
    ARGS_SKIP = ARGS_REQUIRED
};

enum SERVER
{
    SERVER_DATE,
    SERVER_TIME,
    SERVER_THREAD,
    SERVER_SEVERITY,
    SERVER_FUNC,
    SERVER_CONN,
    SERVER_TIME_LEN = 16,
    SERVER_TIMESTAMP_LEN = 28
};

enum CONN
{
    CONN_IP,
    CONN_PORT,
    CONN_CLIENT_ID
};

enum
{
    END_TIME_THRESHOLD = 60
};

typedef std::string String;
typedef std::vector<String> StringVec;
typedef std::list<String> StringList;
typedef std::map<String, StringList> ConnMap;
typedef std::pair<String, StringList> ConnMapEntry;

const char *timeFormat = "%Y-%m-%d %H:%M:%S";
StringVec skipIps;
size_t lines = 0;
size_t skipped = 0;
String firstTimestamp;
String lastTimestamp;
ConnMap connections;

StringVec split(const String& str, const char* delim)
{
    char buffer[str.size() + 1];
    StringVec strings;

    strcpy(buffer, str.c_str());

    char *token = std::strtok(buffer, delim);
    while (token != NULL) {
        strings.push_back(token);
        token = std::strtok(NULL, delim);
    }
    
    return strings;
}

String join(const StringVec& strings, const String& sep,
            size_t start = 0, size_t end = 0)
{
    String str;

    if (!end) {
        end = strings.size();
    }
    for (size_t i = start; i < end; ++i) {
        str.append(strings[i]);
        if (i + 1 < end) {
            str.append(sep);
        }
    }
    return str;
}

bool isMember(const String&str, const StringVec& set)
{
    for (size_t i = 0; i < set.size(); ++i) {
        if (str == set[i]) {
            return true;
        }
    }

    return false;
}

typedef int (*CharMatch)(int c);

bool isToken(const String& token, CharMatch isMatch)
{
    if (token.empty()) {
        return false;
    }
    else {
        for (size_t i = 0; i < token.size(); ++i)
        {
            if (!isMatch(token[i])) {
                return false;
            }
        }
    }
    return true;
}

bool isToken(const StringVec& tokens, CharMatch isMatch,
             size_t start = 0, size_t end = 0)
{
    if (!end) {
        end = tokens.size();
    }
    for (size_t i = start; i < end; ++i) {
        if (!isToken(tokens[i], isMatch)) {
            return false;
        }
    }
    return true;
}

bool isNumeric(const String& token)
{
    return isToken(token, isdigit);
}

bool isHex(const String& token)
{
    return isToken(token, isxdigit);
}

bool isNumeric(const StringVec& tokens,
               size_t start = 0, size_t end = 0)
{
    return isToken(tokens, isdigit, start, end);
}

bool isIpv4Address(const String& str)
{
    StringVec tokens(split(str, "."));

    return ((tokens.size() == 4) &&
            isNumeric(tokens));
}

bool isIpv4Port(const String& str)
{
    return ((str.size() <= 5) &&
            isNumeric(str));
}

bool isIpv4Socket(const StringVec& strings)
{
    return ((strings.size() >= 2) &&
            isIpv4Address(strings[0]) &&
            isIpv4Port(strings[1]));
}

time_t getTime(const String& strTime, const char* format)
{
    std::tm t = {};
    std::istringstream ss(strTime);
    ss >> std::get_time(&t, format);
    return mktime(&t);
}

time_t getTime(const String& field)
{
    // Skip opening and closing brackets.
    return getTime(field.substr(1, SERVER_TIMESTAMP_LEN - 2),
                   timeFormat);
}

size_t getDuration(const time_t& start, const time_t& stop)
{
    size_t seconds(difftime(stop, start));
    return seconds;
}

size_t getDuration(const String& start, const String& stop)
{
    return getDuration(getTime(start), getTime(stop));
}

size_t getDuration(const time_t& start, const String& stop)
{
    return getDuration(start, getTime(stop));
}

size_t getDuration(const String& start, const time_t& stop)
{
    return getDuration(getTime(start), stop);
}

bool isServerTime(const String& str)
{
    if (str.size() == SERVER_TIME_LEN) {
        for (size_t i = 0; i < str.size(); ++i)
        {
            if (!isdigit(str[i]) &&
                (str[i] != ':') &&
                (str[i] != '.') &&
                (str[i] != ']')) {
                return false;
            }
        }
        return true;
    }
    return false;
}

bool isConnId(const String& str)
{
    StringVec fields(split(str, ":"));

    return (isIpv4Socket(fields) &&
            (fields.size() < CONN_CLIENT_ID + 1 ||
             isHex(fields[CONN_CLIENT_ID])));
}

bool isServerConn(const StringVec& fields)
{
    return ((fields.size() > SERVER_CONN) &&
            isServerTime(fields[SERVER_TIME]) &&
            isConnId(fields[SERVER_CONN]));
}

bool loadServer(const char* fileName)
{
    FILE* file = std::fopen(fileName, "r");
    
    if (file) {
        char buffer[1000];
        while (std::fgets(buffer, sizeof(buffer), file) != NULL) {
            String line(buffer);
            StringVec fields = split(buffer, " \t");

            if (isServerConn(fields)) {
                ++lines;
                lastTimestamp = line.substr(0, SERVER_TIMESTAMP_LEN);
                if (firstTimestamp.empty()) {
                    firstTimestamp = lastTimestamp;
                }
                
                strncpy(buffer, fields[SERVER_CONN].c_str(),
                        sizeof(buffer));
                StringVec conn = split(buffer, ":");

                if (isMember(conn[CONN_IP], skipIps)) {
                    ++skipped;
                }
                else {
                    String key(conn[CONN_IP]);
                    key.append(":");
                    key.append(conn[CONN_PORT]);

                    ConnMap::iterator match;
                    match = connections.find(key);
                    if (match == connections.end()) {
                        connections.insert(ConnMapEntry(key,
                                           StringList()));
                        match = connections.find(key);
                    }
                    match->second.push_back(line);
                }
            }
        }
        std::fclose(file);
        if (connections.empty()) {
            std::cout << "No connections found" << std::endl;
            return false;
        }
        return true;
    }
    std::cout << "Failed to open server file"
              << fileName << std::endl;
    return false;
}

void printSeparator()
{
    std::cout << std::endl
              << "=-=-=-=-" << std::endl
              << std::endl;
}

void listConnections()
{
    std::cout << connections.size() << " connections "
              << firstTimestamp << "-" << lastTimestamp << " "
              << lines << " lines, "
              << getDuration(firstTimestamp, lastTimestamp) << " sec:"
              << std::endl;
    if (skipIps.size()) {
        std::cout << "(skipped " << skipped << " connections with "
                  << join(skipIps, ", ") << ")" << std::endl;
    }
    std::cout << std::endl;
    for (ConnMap::iterator curConn = connections.begin();
         curConn != connections.end();
         curConn++) {        
        String conn(curConn->first);
        StringList connLogs(curConn->second);
        std::cout << conn << "\t"
                  << connLogs.front().substr(0, SERVER_TIMESTAMP_LEN)
                  << "-" << connLogs.back().substr(0, SERVER_TIMESTAMP_LEN)
                  << " " << connLogs.size() << " lines, "
                  << getDuration(connLogs.front(), connLogs.back())
                  << " sec" << std::endl;
    }
    printSeparator();
}

void logConnections()
{
    time_t timeFirst = getTime(firstTimestamp);
    time_t timeLast = getTime(lastTimestamp);
    
    for (ConnMap::iterator curConn = connections.begin();
         curConn != connections.end();
         curConn++) {        
        String conn(curConn->first);
        StringList connLogs(curConn->second);
        size_t duration(getDuration(connLogs.front(), connLogs.back()));
        
        std::cout << "Connection " << conn
                  << " " << connLogs.front().substr(0, SERVER_TIMESTAMP_LEN)
                  << "-" << connLogs.back().substr(0, SERVER_TIMESTAMP_LEN)
                  << " " << connLogs.size() << " lines, "
                  << duration << " sec:" << std::endl << std::endl;

        size_t seconds = getDuration(timeFirst, connLogs.front());
        std::cout << firstTimestamp << " Starts " << seconds
                  << " sec after start of log." << std::endl;

        for (StringList::iterator curLog = connLogs.begin();
             curLog != connLogs.end();
             curLog++) {        
            std::cout << *curLog;
        }

        seconds = getDuration(connLogs.back(), timeLast);
        std::cout << lastTimestamp
                  << " " << connLogs.size() << " lines, "
                  << duration << " sec. Ends "
                  << seconds << " sec before end of log.";
        if (seconds > END_TIME_THRESHOLD) {
            std::cout << " Exceeds threshold.";
        }
        std::cout << std::endl;
        printSeparator();
    }
}

int main(int argc, char* argv[])
{
    if (argc < ARGS_REQUIRED ||
        String(argv[1]) == "-h") {
        std::cout << "Usage: " << argv[ARGS_PROGNAME]
                  << " <serverLog> [<skipIps>]" << std::endl;
        std::cout << "Where <skipIps> is comma-separated list "
                  << "of IP addresses to skip." << std::endl;
        return EXIT_FAILURE;
    }
    else {
        if (argc > ARGS_REQUIRED) {
            skipIps = split(argv[ARGS_SKIP], ",");
        }
        
        if (loadServer(argv[ARGS_SERVER_LOG])) {
            listConnections();
            logConnections();
        } else {
            return EXIT_FAILURE;
        }
    }
    return EXIT_SUCCESS;
}

The sample log file, conns.log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[2018-04-03 13:16:29.469659] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket receive_buffer_size=117708
[2018-04-03 13:16:29.469678] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket send_buffer_size=43520
[2018-04-03 13:16:29.867381] [0x00007fb3ff129700] [debug]   set_idle_send_timeout() 10.1.180.206:30450 set idle send timeout to 60 seconds
[2018-04-03 13:16:29.867394] [0x00007fb3ff129700] [debug]   set_idle_receive_timeout() 10.1.180.206:30450 set idle receive timeout to 120 seconds
[2018-04-03 13:16:29.867450] [0x00007fb3ff92a700] [info]    handle_connected() 10.1.180.206:30450 remote connected [2423/8951]
[2018-04-03 13:16:29.959877] [0x00007fb3ff129700] [debug]   install_connection() 10.1.180.206:30450:00003bd4 linkType 0
[2018-04-03 13:16:29.966599] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x4a2137a6, 231 bytes
[2018-04-03 13:16:29.966935] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xa11f878a, 35 bytes, msg type 3
[2018-04-03 13:17:29.967117] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x1c35386e, 29 bytes, msg type 1
[2018-04-03 13:17:29.967228] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:17:30.086722] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:30.086813] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:17:40.086722] [0x00007fb3ff129700] [debug]   receive() 10.2.80.206:3050:00000bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:40.086813] [0x00007fb3ff129700] [debug]   send() 10.2.80.206:3050:00000bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:17:30.139377] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xa0f8a8e1, 78 bytes
[2018-04-03 13:18:29.867494] [0x00007fb3ff129700] [debug]   handle_idle_receive_timeout() 127.0.0.1:32450:00003bd4 60 seconds
[2018-04-03 13:18:29.967315] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:18:30.086988] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x8fcf53f6, 29 bytes, msg type 1
[2018-04-03 13:18:30.087101] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:18:30.197029] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x515f1d4e, 29 bytes
[2018-04-03 13:18:30.197120] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x6827d190, 29 bytes, msg type 17
[2018-04-03 13:18:30.249027] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xda66c5c7, 78 bytes
[2018-04-03 13:19:30.087189] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:19:30.139486] [0x00007fb3ff129700] [debug]   handle_idle_receive_timeout() 10.1.180.206:30450:00003bd4 60 seconds
[2018-04-03 13:19:30.197322] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x812afb16, 29 bytes, msg type 1

Sample run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
$ g++ logsplit.cpp -o logsplit
$ ./logsplit conns.log 127.0.0.1,3.3.3.3
2 connections [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 25 lines, 181 sec:
(skipped 1 connections with 127.0.0.1, 3.3.3.3)

10.1.180.206:30450 [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 22 lines, 181 sec
10.2.80.206:3050 [2018-04-03 13:17:40.086722]-[2018-04-03 13:17:40.086813] 2 lines, 0 sec

=-=-=-=-

Connection 10.1.180.206:30450 [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 22 lines, 181 sec:

[2018-04-03 13:16:29.469659] Starts 0 sec after start of log.
[2018-04-03 13:16:29.469659] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket receive_buffer_size=117708
[2018-04-03 13:16:29.469678] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket send_buffer_size=43520
[2018-04-03 13:16:29.867381] [0x00007fb3ff129700] [debug]   set_idle_send_timeout() 10.1.180.206:30450 set idle send timeout to 60 seconds
[2018-04-03 13:16:29.867394] [0x00007fb3ff129700] [debug]   set_idle_receive_timeout() 10.1.180.206:30450 set idle receive timeout to 120 seconds
[2018-04-03 13:16:29.867450] [0x00007fb3ff92a700] [info]    handle_connected() 10.1.180.206:30450 remote connected [2423/8951]
[2018-04-03 13:16:29.959877] [0x00007fb3ff129700] [debug]   install_connection() 10.1.180.206:30450:00003bd4 linkType 0
[2018-04-03 13:16:29.966599] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x4a2137a6, 231 bytes
[2018-04-03 13:16:29.966935] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xa11f878a, 35 bytes, msg type 3
[2018-04-03 13:17:29.967117] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x1c35386e, 29 bytes, msg type 1
[2018-04-03 13:17:29.967228] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:17:30.086722] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:30.086813] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:17:30.139377] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xa0f8a8e1, 78 bytes
[2018-04-03 13:18:29.967315] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:18:30.086988] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x8fcf53f6, 29 bytes, msg type 1
[2018-04-03 13:18:30.087101] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:18:30.197029] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x515f1d4e, 29 bytes
[2018-04-03 13:18:30.197120] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x6827d190, 29 bytes, msg type 17
[2018-04-03 13:18:30.249027] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xda66c5c7, 78 bytes
[2018-04-03 13:19:30.087189] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:19:30.139486] [0x00007fb3ff129700] [debug]   handle_idle_receive_timeout() 10.1.180.206:30450:00003bd4 60 seconds
[2018-04-03 13:19:30.197322] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x812afb16, 29 bytes, msg type 1
[2018-04-03 13:19:30.197322] 22 lines, 181 sec. Ends 0 sec before end of log.

=-=-=-=-

Connection 10.2.80.206:3050 [2018-04-03 13:17:40.086722]-[2018-04-03 13:17:40.086813] 2 lines, 0 sec:

[2018-04-03 13:16:29.469659] Starts 71 sec after start of log.
[2018-04-03 13:17:40.086722] [0x00007fb3ff129700] [debug]   receive() 10.2.80.206:3050:00000bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:40.086813] [0x00007fb3ff129700] [debug]   send() 10.2.80.206:3050:00000bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:19:30.197322] 2 lines, 0 sec. Ends 110 sec before end of log. Exceeds threshold.

=-=-=-=-