Sunday, April 29, 2018

How To Ace Calculus

This is the method I used to ace 3 semesters of calculus. Also linear algebra, differential equations, and a semester of physics. It should work for any math or science class, at earlier or later level.

When I say ace, I mean getting a grade of 100 on most homework, quizzes, tests, midterms, and finals. In all cases, the top grade in the class. Yes, I was the one breaking the curve.

Sounds arrogant? Well, I didn't start off in that lofty position. So before I give you the recipe for success, let me give you the recipe for failure.

The Recipe For Failure

In 1978, I entered Northwestern University, in Evanston, IL, as a mechanical engineering major. My dream was to work for NASA.

This was a major in the Technical Institute, requiring calculus, physics, and mechanics (statics and dynamics).

The recipe:
  1. Show up for all classes and pay attention.
  2. Complete all reading assignments on time.
  3. Complete all assigned homework on time.
  4. Study for tests, reviewing homework.
This was the recipe that had gotten me through high school, where I was usually able to do most of the homework in the last 5 minutes of class allocated for that purpose. Then just a few minutes in "study hall" period or at home to complete it, and another 10 or 15 to read the next section.

Sounds like a pretty good plan, right? Sounds like a good student, right?

The problem was that the material in college was more difficult and faster paced. Doing just the assigned problems was barely enough to keep your head above water, sometimes not even enough for that. It didn't give you enough practice thinking through and performing the work.

Back in algebra, problems were simple, they had one procedure to follow to the solution. Calculus wasn't like that. There were multiple procedures depending on the style of the equation. A good portion of the battle was classifying the equation to determine what approach to bring to it.

Physics was similar. Both topics required a more analytical approach. That meant building up a problem database in your mind so you could pick the approach. That meant experience doing lots of problems.

The result of following that recipe? C's, D's, and finally, an F in physics. Where I had prided myself on my math and science abilities, my favorite subjects, I had failed. Distraught, I dropped out of Northwestern.

The Recipe For Success

About 5 years later, I started part-time classes at Richland College, part of the Dallas County Community College District.

Oh sure, you may say, community college. That's easy, it's not a real college.

Negative. Richland used exactly the same textbooks as Northwestern, just the next editions. So it was exactly the same material. And I had Ralph Esparza as instructor for calculus I and III. Ralph was feared among students as a tough math teacher, who cares if it's community college or Ivy League.

I was determined to repeat all those classes in my favorite subjects, and do well in them. Somewhere in hindsight, I had realized the need to do more than the minimum.

The recipe:
  1. Show up for all classes and pay attention.
  2. Complete all reading assignments on time.
  3. Complete all assigned homework on time.
  4. Complete all remaining odd-numbered problems in the section and check against answers in the back.
  5. Complete all remaining even-numbered problems in the section.
  6. Study for tests (regular tests, midterms, and finals), redoing all problems that the tests cover.
This is a great way to work in study groups, too. When someone in the group has difficulty, everyone can contribute to helping them understand it. Or maybe the one person in the group who understands it is able to help everyone else.
This boils down to doing every problem in every section of the textbook at least twice, more like three or four times.

You may say, that's a lot of work. Yes, it is.

In Nike ads, athletes show how tough they are. Just do it. Be tough.

The result? Redemption.

Saturday, April 21, 2018

Sodoto: See One, Do One, Teach One

Here's a useful strategy on this learning path: see one, do one, teach one. Sodoto.

Sodoto is a learning method and a teaching method rolled into one. It's the cycle of knowledge.

I'm familiar with it from medicine. My wife is a surgical nurse, and this is the traditional method of teaching in surgery. Obviously, safety concerns mean that you don't just watch a brain surgeon at work and then go try it yourself.

But this forms a useful pattern of mentoring and learning and passing knowledge along. It applies to any kind of knowledge- or skill-based activity.

It works with a single student at a time, or a whole group. You don't have to be a formal teacher.

See one: watch someone do a procedure.

Do one: do what you saw.

Teach one: show it to someone else.

Once you learn a procedure, you're primed to teach it. That's how knowledge spreads.

Here's the real kicker: the teaching step is actually a powerful learning step for you as the teacher. It locks the knowledge into your brain.

You have to have sorted out what you're talking about in order to teach it. You can't just vaguely know it and wave your hands in the air glossing over details. Your students will be annoyed and you'll feel stupid.

The process of getting ready to teach and then doing the teaching forces you to organize your thoughts and chase down details, because you don't want to look stupid, and you want to be prepared for questions.

That motivates you to dig deeper. As a result, you end up learning more yourself.

There are two keys to making this work: background knowledge, and the experience of doing it.

Background Knowledge

Background knowledge applies at each stage of see, do, and teach. Note that "see" can mean live and in person, or on video.

Whatever the subject, medicine, coding, climbing, sailing, scuba diving, physical fitness, martial arts, building anything from woodworking to electronics, any knowledge you have before seeing the procedure will help you understand it.

You can bet that surgeon learning how to do brain surgery brought a huge amount of background knowledge.

Some things take minimal background, just the random skills and knowledge you already have from life. But more difficult subjects benefit from whatever time you can invest beforehand. Videos, books, blogs, and articles, in print and online, are all good resources, as well as online forums.

That establishes the background knowledge you'll bring to seeing the procedure.

Once you've seen the procedure, as you prepare to do it yourself, it's useful to go back to your resources. Now that you know better what to look for, you can get more details. You can reinforce what you saw.

That expands the background knowledge you'll bring to doing the procedure.

Once you've done the procedure, as you prepare to teach it, go back to your resources again. As a result of doing, there will be details you want to fill in, and you may understand the material better. You may have run into some things that you wished you knew more about. You may anticipate additional questions from your students.

That further expands the background knowledge you'll bring to teaching the procedure.

Experience Of Doing

The experience of doing the procedure is critical. That's where you have the opportunity to work through mistakes and see what works and doesn't work for you. That's where you start to lock it into your brain.

Don't be afraid to make mistakes! Mistakes are great learning opportunities. As long as there's no injury and no damage, there's no harm done. And a little blood on the deck isn't an injury.

This is also where you can work out your own changes to the procedure. Just because you saw it done one way doesn't mean that's the only way to do it. That was one way. You can use it as your starting point, and add your own tweaks.

Or maybe you'll realize what you saw really was a good way and you shouldn't mess with it.

You might need to do the procedure more than once before teaching it. Some procedures take practice before you feel confident teaching them to someone else.

The experience of teaching the procedure will be different from the experience of doing it for yourself. Your students may have questions or difficulties that force you to think about things in different ways.

Plus there's the pressure of performing for an audience. But as you gain experience teaching, that will get easier. It's just a different kind of doing.

The experience of teaching is where you finish locking it into your brain.

Teamwork

Sodoto is a great method for dividing up a project. Whether at work, at school, or with your friends, you can divide up the project and have each person take on a part.

They go off and see how it's done, do it themselves until they feel ready, then bring it back to the team to teach everyone else.

What if you can't agree how to divide it up because multiple people want to do the same thing? Fine! Let them!

Each person will have their own take on the experience and teach it slightly differently. That helps explore all the possibilities in the procedure.

Tuesday, April 17, 2018

More C+-

In The Case For C+-, I talked about writing quick tools in a simple C style, but taking advantage of the C++ standard library, primarily the dynamic data structures. It ends up being C++ without any (or just a few) user-defined classes, so is something of a lightweight object-oriented approach (yes, yes, I'm sure OO purists are barfing at the thought). The main benefit is fast coding.

There I showed as an example the msgresolve tool, which I used to resolve messages logged by an IOT device (the client) and its server. This is a lot of string processing and cross-indexing, with logs containing potentially thousands or tens of thousands of messages.

Shortly after I had completed msgresolve, I needed to have a tool to help me sift through large text files of server logs, logging the TCP connections made by clients and their subsequent activity. I was chasing down a problem where some of the connections were shutting down sooner than expected.

I wasn't sure what was causing the early shutdowns, and wasn't even sure initially which connections had experienced it, so I wanted to be able to gather all the lines for a given connection and list them out for tracing through, for each connection.

That would help me identify the ones that were live at the end of the log sample vs. the ones that had ended early. The log entries for hundreds of connections were all intermixed.

Armed with the methods I had used in msgresolve.cpp, conceptual design was easy. I wanted an ordered list of connections, and associated with each one, the sequential list of log entries associated with the connection.

There were also connections with some internal addresses I wanted to ignore. I could have done this filtering with grep, but it was easy enough to build the capability into the program so that it could stand alone. That also helped me explore some additional string processing functions.

Given that architecture, the data structure I needed was a std::map that mapped a string (the connection identification) to a std::list of strings (the log lines for the connection).

I had the program working in less than an hour. Then I spent at least another hour screwing around with the timestamps in the log entries, figuring out how to process them and deciding what to do with them. Then a little more time on refactoring and cleanup.

Throughout, I used a sample log file that had entries for several connections, including addresses I wanted to skip. I used that as a simple unit test to exercise the code paths.

The resulting code provided the impetus for a simple generalized string processing module, which I'll cover in another post. But you can see some clear patterns emerging in this code.

Doing quick tools like this is fun and very satisfying. It makes your serotonin flow. You have a problem you need to deal with, so you sit down and spew a bunch of code in a short time, refine it, and use the results.

This is actually quite different from long-term product development. That kind of work has its intense coding phases, but once the initial version of the product is out, a lot of the work is much smaller surgical changes.

Even fitting a major new feature in often involves many small bits of code scattered throughout the larger code base, integrating the tendrils. Getting that to work has a different kind of satisfaction.

Design Choices

These tools also give you a chance to think about different approaches. You can balance the variables of memory consumption, CPU consumption, I/O consumption, time, and code complexity (that is, ease of writing and maintaining the code, and compiled code space consumption, not algorithmic complexity) for a given situation.

For instance, the log files I was dealing with had over a million lines of data, some 200MB worth covering hundreds of connections.

That meant I had several choices:
  1. I could load all the data into memory and then print it out in an orderly manner. This is a single-pass solution, that consumes large amounts of memory.
  2. I could scan the file once, identifying all the individual connections, then for each connection, scan the file from beginning to end to read and print their lines. This a multi-pass solution that requires little memory but significant file I/O.
  3. I could scan the file once, and for each identified connection, track the file position of the first and last line, then for each connection, just scan that range of the file. This is a multi-pass solution that reduces the total file I/O for a negligible increase in memory.
  4. I could do the same thing, but instead of tracking just the first and last line file positions, build a list of the file position and length of each line, then on each pass, just skip directly to the locations of the lines. This is still multi-pass, but significantly reduces the total file I/O because it only visits each file position twice, requiring a bit more complexity and a bit more memory.
The decision on which choice to use is system-dependent. If memory is cheap and plentiful, and file I/O is relatively expensive, either in terms of time or charges to transfer data over a data link (maybe the data is remote, accessed over a cellular link), then the single-pass solution is best, choice #1.

On a small-memory system, a multi-pass solution is better, and you just have to live with the extra I/O. In that case, choice #4, which is the most complicated code, has the best compromise of low memory and low I/O consumption.

Although if you're really pressed for code space, the simplest multipass solution that iterates over the entire file for each connection is the better choice, #2.

Realistically though, you don't run tools such as this locally on small systems. Where possible, you offload the data to a larger system and run the tools offline.

In this case, I'm running on a Mac with 16GB of memory. Slurping up a 200MB text file and holding everything in memory is nothing. So the single-pass solution is the way to go.

Loading The Data

The log file may just be a chronological sample of all the activity logged by the server, so some connections will already exist at the start of the log, and some will remain at the end. Meanwhile, connection starts and terminations will appear in the log.

The program parses the lines from the log to find connection lines (i.e. some activity for a given connection). It identifies them by looking for a connection ID, which consists of an IPv4 address/port pair (a remote socket ID), and may optionally include a hexadecimal client ID.

It uses just the socket ID as the connection ID, which is the key to the connection map. When it finds a new connection ID, it adds it to the map with an empty list. For each connection line, it finds the connection map entry and appends the line to the list of lines for that connection.

As it loads connection lines, it filters them against the set of IP addresses to skip (these skip address are due to logging of other types of connections besides the client connections). That helps reduce the noise from a large log file.

Printing The Data

The program prints the connections by first iterating through the map and printing out a summary of each one: the connection ID, the number of lines, the duration of the connection data found in the log, and how its lifetime relates to the overall log. Since the map is an ordered structure, printing connections is always in sorted order by connection ID (though string sorted, not numeric sorted).

Then it makes a second pass through the connection map, iterating through each line for each connection and printing it out, with separators between connections. It prints a header and summary line before and after the activity lines for each connection, showing where the connection starts and ends relative to the start and end of the log.

As a last quick change, I added a threshold time value to clearly identify connections that ended at least 60 seconds before the end of the log. This would be a good candidate for a command-line parameter to override the default threshold.

All the output lines have unique greppable features or keywords so you can use other tools for additional postprocessing or extraction. For instance, I could grep out the end summary line of each connection, and maybe the last couple of activity lines before it, to see how each connection ended up. I could use the "threshold exceeded" indication to identify the ones that had ended early.

Some Design Evolution

This program adds the isMember() function, which determines whether a string is a member of a set of strings. Since my usage here was intended to deal with a small set and I had other functions that had similar iterative structure, in the heat of battle I quickly coded it as a linear search of a vector of strings.

That worked fine here, but as I pulled a bunch of this code out into a general string processing module, I realized that was a bad choice, because it's an O(N) search.

That became especially bad when I wanted an overload that took a vector of strings and determined if they were all members. That meant an O(M) repetition of O(N) searches: an O(M*N) or effectively O(N^2) algorithm.

That gets out of hand fast as M and N get larger. Meanwhile, the std::unordered_set is perfect for this, an O(1) algorithm for single searches, and an O(M) algorithm when repeated for M items.

I've left the original isMember() implementation here as an example of the evolution of a concept as you generalize it for other uses.

I also threw in a few overloads that I didn't end up using, but that set the stage for running with the concept in the string processing module. More discussion of that in the post containing the module.

The Result

This program turned out to be another fun exercise in string processing once I had built the basic support functions and could see the problem in those terms. It felt more like working in Python, and in fact just as the dict structure from msgresolve.cpp was inspired by Python, so are the split() and join() functions here.

The funny thing is that it took doing stuff in Python to make me see this approach. That points out one of the advantages of working in multiple different languages: you start seeing opportunities to apply some of the common idioms from one language in another language.

Here's logsplit.cpp:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
// Usage: logsplit <serverLog>
//
// Splits a server log file by IPv4 connection. Prints a
// summary list of the connections, then the log lines for
// each separate connection.
//
// This is an example of a C++ program that is written mostly
// in plain C style, but that makes use of the container and
// composition classes in the C++ standard library. It is a
// lightweight use of C++ with no user-defined classes.
//
// 2018 Steve Branam <sdbranam@gmail.com> learntocode

#include <iostream>
#include <iomanip>
#include <sstream>
#include <vector>
#include <list>
#include <map>

enum ARGS
{
    ARGS_PROGNAME,
    ARGS_SERVER_LOG,
    ARGS_REQUIRED,
    ARGS_SKIP = ARGS_REQUIRED
};

enum SERVER
{
    SERVER_DATE,
    SERVER_TIME,
    SERVER_THREAD,
    SERVER_SEVERITY,
    SERVER_FUNC,
    SERVER_CONN,
    SERVER_TIME_LEN = 16,
    SERVER_TIMESTAMP_LEN = 28
};

enum CONN
{
    CONN_IP,
    CONN_PORT,
    CONN_CLIENT_ID
};

enum
{
    END_TIME_THRESHOLD = 60
};

typedef std::string String;
typedef std::vector<String> StringVec;
typedef std::list<String> StringList;
typedef std::map<String, StringList> ConnMap;
typedef std::pair<String, StringList> ConnMapEntry;

const char *timeFormat = "%Y-%m-%d %H:%M:%S";
StringVec skipIps;
size_t lines = 0;
size_t skipped = 0;
String firstTimestamp;
String lastTimestamp;
ConnMap connections;

StringVec split(const String& str, const char* delim)
{
    char buffer[str.size() + 1];
    StringVec strings;

    strcpy(buffer, str.c_str());

    char *token = std::strtok(buffer, delim);
    while (token != NULL) {
        strings.push_back(token);
        token = std::strtok(NULL, delim);
    }
    
    return strings;
}

String join(const StringVec& strings, const String& sep,
            size_t start = 0, size_t end = 0)
{
    String str;

    if (!end) {
        end = strings.size();
    }
    for (size_t i = start; i < end; ++i) {
        str.append(strings[i]);
        if (i + 1 < end) {
            str.append(sep);
        }
    }
    return str;
}

bool isMember(const String&str, const StringVec& set)
{
    for (size_t i = 0; i < set.size(); ++i) {
        if (str == set[i]) {
            return true;
        }
    }

    return false;
}

typedef int (*CharMatch)(int c);

bool isToken(const String& token, CharMatch isMatch)
{
    if (token.empty()) {
        return false;
    }
    else {
        for (size_t i = 0; i < token.size(); ++i)
        {
            if (!isMatch(token[i])) {
                return false;
            }
        }
    }
    return true;
}

bool isToken(const StringVec& tokens, CharMatch isMatch,
             size_t start = 0, size_t end = 0)
{
    if (!end) {
        end = tokens.size();
    }
    for (size_t i = start; i < end; ++i) {
        if (!isToken(tokens[i], isMatch)) {
            return false;
        }
    }
    return true;
}

bool isNumeric(const String& token)
{
    return isToken(token, isdigit);
}

bool isHex(const String& token)
{
    return isToken(token, isxdigit);
}

bool isNumeric(const StringVec& tokens,
               size_t start = 0, size_t end = 0)
{
    return isToken(tokens, isdigit, start, end);
}

bool isIpv4Address(const String& str)
{
    StringVec tokens(split(str, "."));

    return ((tokens.size() == 4) &&
            isNumeric(tokens));
}

bool isIpv4Port(const String& str)
{
    return ((str.size() <= 5) &&
            isNumeric(str));
}

bool isIpv4Socket(const StringVec& strings)
{
    return ((strings.size() >= 2) &&
            isIpv4Address(strings[0]) &&
            isIpv4Port(strings[1]));
}

time_t getTime(const String& strTime, const char* format)
{
    std::tm t = {};
    std::istringstream ss(strTime);
    ss >> std::get_time(&t, format);
    return mktime(&t);
}

time_t getTime(const String& field)
{
    // Skip opening and closing brackets.
    return getTime(field.substr(1, SERVER_TIMESTAMP_LEN - 2),
                   timeFormat);
}

size_t getDuration(const time_t& start, const time_t& stop)
{
    size_t seconds(difftime(stop, start));
    return seconds;
}

size_t getDuration(const String& start, const String& stop)
{
    return getDuration(getTime(start), getTime(stop));
}

size_t getDuration(const time_t& start, const String& stop)
{
    return getDuration(start, getTime(stop));
}

size_t getDuration(const String& start, const time_t& stop)
{
    return getDuration(getTime(start), stop);
}

bool isServerTime(const String& str)
{
    if (str.size() == SERVER_TIME_LEN) {
        for (size_t i = 0; i < str.size(); ++i)
        {
            if (!isdigit(str[i]) &&
                (str[i] != ':') &&
                (str[i] != '.') &&
                (str[i] != ']')) {
                return false;
            }
        }
        return true;
    }
    return false;
}

bool isConnId(const String& str)
{
    StringVec fields(split(str, ":"));

    return (isIpv4Socket(fields) &&
            (fields.size() < CONN_CLIENT_ID + 1 ||
             isHex(fields[CONN_CLIENT_ID])));
}

bool isServerConn(const StringVec& fields)
{
    return ((fields.size() > SERVER_CONN) &&
            isServerTime(fields[SERVER_TIME]) &&
            isConnId(fields[SERVER_CONN]));
}

bool loadServer(const char* fileName)
{
    FILE* file = std::fopen(fileName, "r");
    
    if (file) {
        char buffer[1000];
        while (std::fgets(buffer, sizeof(buffer), file) != NULL) {
            String line(buffer);
            StringVec fields = split(buffer, " \t");

            if (isServerConn(fields)) {
                ++lines;
                lastTimestamp = line.substr(0, SERVER_TIMESTAMP_LEN);
                if (firstTimestamp.empty()) {
                    firstTimestamp = lastTimestamp;
                }
                
                strncpy(buffer, fields[SERVER_CONN].c_str(),
                        sizeof(buffer));
                StringVec conn = split(buffer, ":");

                if (isMember(conn[CONN_IP], skipIps)) {
                    ++skipped;
                }
                else {
                    String key(conn[CONN_IP]);
                    key.append(":");
                    key.append(conn[CONN_PORT]);

                    ConnMap::iterator match;
                    match = connections.find(key);
                    if (match == connections.end()) {
                        connections.insert(ConnMapEntry(key,
                                           StringList()));
                        match = connections.find(key);
                    }
                    match->second.push_back(line);
                }
            }
        }
        std::fclose(file);
        if (connections.empty()) {
            std::cout << "No connections found" << std::endl;
            return false;
        }
        return true;
    }
    std::cout << "Failed to open server file"
              << fileName << std::endl;
    return false;
}

void printSeparator()
{
    std::cout << std::endl
              << "=-=-=-=-" << std::endl
              << std::endl;
}

void listConnections()
{
    std::cout << connections.size() << " connections "
              << firstTimestamp << "-" << lastTimestamp << " "
              << lines << " lines, "
              << getDuration(firstTimestamp, lastTimestamp) << " sec:"
              << std::endl;
    if (skipIps.size()) {
        std::cout << "(skipped " << skipped << " connections with "
                  << join(skipIps, ", ") << ")" << std::endl;
    }
    std::cout << std::endl;
    for (ConnMap::iterator curConn = connections.begin();
         curConn != connections.end();
         curConn++) {        
        String conn(curConn->first);
        StringList connLogs(curConn->second);
        std::cout << conn << "\t"
                  << connLogs.front().substr(0, SERVER_TIMESTAMP_LEN)
                  << "-" << connLogs.back().substr(0, SERVER_TIMESTAMP_LEN)
                  << " " << connLogs.size() << " lines, "
                  << getDuration(connLogs.front(), connLogs.back())
                  << " sec" << std::endl;
    }
    printSeparator();
}

void logConnections()
{
    time_t timeFirst = getTime(firstTimestamp);
    time_t timeLast = getTime(lastTimestamp);
    
    for (ConnMap::iterator curConn = connections.begin();
         curConn != connections.end();
         curConn++) {        
        String conn(curConn->first);
        StringList connLogs(curConn->second);
        size_t duration(getDuration(connLogs.front(), connLogs.back()));
        
        std::cout << "Connection " << conn
                  << " " << connLogs.front().substr(0, SERVER_TIMESTAMP_LEN)
                  << "-" << connLogs.back().substr(0, SERVER_TIMESTAMP_LEN)
                  << " " << connLogs.size() << " lines, "
                  << duration << " sec:" << std::endl << std::endl;

        size_t seconds = getDuration(timeFirst, connLogs.front());
        std::cout << firstTimestamp << " Starts " << seconds
                  << " sec after start of log." << std::endl;

        for (StringList::iterator curLog = connLogs.begin();
             curLog != connLogs.end();
             curLog++) {        
            std::cout << *curLog;
        }

        seconds = getDuration(connLogs.back(), timeLast);
        std::cout << lastTimestamp
                  << " " << connLogs.size() << " lines, "
                  << duration << " sec. Ends "
                  << seconds << " sec before end of log.";
        if (seconds > END_TIME_THRESHOLD) {
            std::cout << " Exceeds threshold.";
        }
        std::cout << std::endl;
        printSeparator();
    }
}

int main(int argc, char* argv[])
{
    if (argc < ARGS_REQUIRED ||
        String(argv[1]) == "-h") {
        std::cout << "Usage: " << argv[ARGS_PROGNAME]
                  << " <serverLog> [<skipIps>]" << std::endl;
        std::cout << "Where <skipIps> is comma-separated list "
                  << "of IP addresses to skip." << std::endl;
        return EXIT_FAILURE;
    }
    else {
        if (argc > ARGS_REQUIRED) {
            skipIps = split(argv[ARGS_SKIP], ",");
        }
        
        if (loadServer(argv[ARGS_SERVER_LOG])) {
            listConnections();
            logConnections();
        } else {
            return EXIT_FAILURE;
        }
    }
    return EXIT_SUCCESS;
}

The sample log file, conns.log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[2018-04-03 13:16:29.469659] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket receive_buffer_size=117708
[2018-04-03 13:16:29.469678] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket send_buffer_size=43520
[2018-04-03 13:16:29.867381] [0x00007fb3ff129700] [debug]   set_idle_send_timeout() 10.1.180.206:30450 set idle send timeout to 60 seconds
[2018-04-03 13:16:29.867394] [0x00007fb3ff129700] [debug]   set_idle_receive_timeout() 10.1.180.206:30450 set idle receive timeout to 120 seconds
[2018-04-03 13:16:29.867450] [0x00007fb3ff92a700] [info]    handle_connected() 10.1.180.206:30450 remote connected [2423/8951]
[2018-04-03 13:16:29.959877] [0x00007fb3ff129700] [debug]   install_connection() 10.1.180.206:30450:00003bd4 linkType 0
[2018-04-03 13:16:29.966599] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x4a2137a6, 231 bytes
[2018-04-03 13:16:29.966935] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xa11f878a, 35 bytes, msg type 3
[2018-04-03 13:17:29.967117] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x1c35386e, 29 bytes, msg type 1
[2018-04-03 13:17:29.967228] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:17:30.086722] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:30.086813] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:17:40.086722] [0x00007fb3ff129700] [debug]   receive() 10.2.80.206:3050:00000bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:40.086813] [0x00007fb3ff129700] [debug]   send() 10.2.80.206:3050:00000bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:17:30.139377] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xa0f8a8e1, 78 bytes
[2018-04-03 13:18:29.867494] [0x00007fb3ff129700] [debug]   handle_idle_receive_timeout() 127.0.0.1:32450:00003bd4 60 seconds
[2018-04-03 13:18:29.967315] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:18:30.086988] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x8fcf53f6, 29 bytes, msg type 1
[2018-04-03 13:18:30.087101] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:18:30.197029] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x515f1d4e, 29 bytes
[2018-04-03 13:18:30.197120] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x6827d190, 29 bytes, msg type 17
[2018-04-03 13:18:30.249027] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xda66c5c7, 78 bytes
[2018-04-03 13:19:30.087189] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:19:30.139486] [0x00007fb3ff129700] [debug]   handle_idle_receive_timeout() 10.1.180.206:30450:00003bd4 60 seconds
[2018-04-03 13:19:30.197322] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x812afb16, 29 bytes, msg type 1

Sample run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
$ g++ logsplit.cpp -o logsplit
$ ./logsplit conns.log 127.0.0.1,3.3.3.3
2 connections [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 25 lines, 181 sec:
(skipped 1 connections with 127.0.0.1, 3.3.3.3)

10.1.180.206:30450 [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 22 lines, 181 sec
10.2.80.206:3050 [2018-04-03 13:17:40.086722]-[2018-04-03 13:17:40.086813] 2 lines, 0 sec

=-=-=-=-

Connection 10.1.180.206:30450 [2018-04-03 13:16:29.469659]-[2018-04-03 13:19:30.197322] 22 lines, 181 sec:

[2018-04-03 13:16:29.469659] Starts 0 sec after start of log.
[2018-04-03 13:16:29.469659] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket receive_buffer_size=117708
[2018-04-03 13:16:29.469678] [0x00007fb3ff129700] [debug]   start() 10.1.180.206:30450 TCP socket send_buffer_size=43520
[2018-04-03 13:16:29.867381] [0x00007fb3ff129700] [debug]   set_idle_send_timeout() 10.1.180.206:30450 set idle send timeout to 60 seconds
[2018-04-03 13:16:29.867394] [0x00007fb3ff129700] [debug]   set_idle_receive_timeout() 10.1.180.206:30450 set idle receive timeout to 120 seconds
[2018-04-03 13:16:29.867450] [0x00007fb3ff92a700] [info]    handle_connected() 10.1.180.206:30450 remote connected [2423/8951]
[2018-04-03 13:16:29.959877] [0x00007fb3ff129700] [debug]   install_connection() 10.1.180.206:30450:00003bd4 linkType 0
[2018-04-03 13:16:29.966599] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x4a2137a6, 231 bytes
[2018-04-03 13:16:29.966935] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xa11f878a, 35 bytes, msg type 3
[2018-04-03 13:17:29.967117] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x1c35386e, 29 bytes, msg type 1
[2018-04-03 13:17:29.967228] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:17:30.086722] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:30.086813] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:17:30.139377] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xa0f8a8e1, 78 bytes
[2018-04-03 13:18:29.967315] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:18:30.086988] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x8fcf53f6, 29 bytes, msg type 1
[2018-04-03 13:18:30.087101] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 59 seconds
[2018-04-03 13:18:30.197029] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0x515f1d4e, 29 bytes
[2018-04-03 13:18:30.197120] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x6827d190, 29 bytes, msg type 17
[2018-04-03 13:18:30.249027] [0x00007fb3ff129700] [debug]   receive() 10.1.180.206:30450:00003bd4 RX sum 0xda66c5c7, 78 bytes
[2018-04-03 13:19:30.087189] [0x00007fb3ff129700] [debug]   handle_idle_send_timeout() 10.1.180.206:30450:00003bd4 0 seconds
[2018-04-03 13:19:30.139486] [0x00007fb3ff129700] [debug]   handle_idle_receive_timeout() 10.1.180.206:30450:00003bd4 60 seconds
[2018-04-03 13:19:30.197322] [0x00007fb3ff129700] [debug]   send() 10.1.180.206:30450:00003bd4 TX sum 0x812afb16, 29 bytes, msg type 1
[2018-04-03 13:19:30.197322] 22 lines, 181 sec. Ends 0 sec before end of log.

=-=-=-=-

Connection 10.2.80.206:3050 [2018-04-03 13:17:40.086722]-[2018-04-03 13:17:40.086813] 2 lines, 0 sec:

[2018-04-03 13:16:29.469659] Starts 71 sec after start of log.
[2018-04-03 13:17:40.086722] [0x00007fb3ff129700] [debug]   receive() 10.2.80.206:3050:00000bd4 RX sum 0xf6e3f3bf, 29 bytes
[2018-04-03 13:17:40.086813] [0x00007fb3ff129700] [debug]   send() 10.2.80.206:3050:00000bd4 TX sum 0xfb208d9f, 29 bytes, msg type 17
[2018-04-03 13:19:30.197322] 2 lines, 0 sec. Ends 110 sec before end of log. Exceeds threshold.

=-=-=-=-

Saturday, April 14, 2018

First Use Of New Tools


Using my new tools to fix my Droid MAXX, lying open front and center.

Sometimes life just amazes me. Three days after receiving my Adafruit tools, I needed them. Twice.

This isn't the first time that I've learned something completely new out of the blue, and almost immediately needed it. It's just incredibly serendipitous.

First Problem

Yesterday, Friday, I worked from home. The project I'm working on is embedded system firmware to control a daughter board. The main smart chip on the board was shutting down unexpectedly, which I could tell initially from the timeouts reading responses from it, and then from its status output LED going off.

There's a power enable line for the chip in the cable pigtail that I control via software, and scouring through my code assured me that no code path should be shutting it off during the scenario I was exercising.

Yet the LED was going off and the chip was unresponsive. Maybe it was a bug in the chip, and my commands to it were triggering the shutdown.

Or maybe something besides my code was affecting the power enable. So I pulled out my new meter, opened up the cable to the daughter card, and clipped the meter into the ground and enable lines, switched to VDC.

With the code in the state where power was enabled, the LED was on, the chip was responsive, and the meter showed 3.2V. With the code in the state where power was disabled, the LED was off, the chip was unresponsive, and the meter showed 0.0V. Ok, good, correct setup.

Running my tests, with additional debug logging verifying the code was NOT running through the parts that shutoff power, I saw the LED go off and the meter drop from 3.2 to 0. So something unrelated was indeed dropping the power enable.

After more testing, I noticed a pattern in the logging, unrelated to my code, that correlated with the shutoff. Now, correlation does not imply causation. But it's a hint about where to look.

So running a couple more debug commands that didn't exercise my code path at all, I was able to confirm that the correlated activity did indeed result in disabling power to the chip.

Yes! Phew, it's not the chip crashing on me, and it's not a problem in the code I just wrote.

Monday, I'll get together with the hardware guys and chase the problem down. First, I'll try it with another unit.

For all I know, my unit is damaged, since it's all pulled apart on my desk to connect the daughter board. Open frame hardware is always vulnerable.

That's an example of diagnosing a problem and ruling out two large potential root cause areas.
Monday update: Another unit showed the same problem. So it wasn't damage in my unit. 
I discussed the evidence in the logging with the hardware engineer. We identified some register accesses used to control GPIO (General Purpose Input/Output) pins on the processor that were affecting more pins than intended. These were causing the GPIO pin controlling the power enable to be cleared.
This shows one of the challenges inherent in embedded systems. You're working at a much lower, more direct level in the hardware, with lots of opportunities to create strange conflicts between different parts of the system. 
Second Problem

Today, Saturday, I was down in the basement, and as I put my phone in my pocket, it slipped and fell flat on its face on the concrete floor with a sharp thud. DOHH!

It's a Droid Maxx in a case that goes in a holster. The case protected the screen and phone from overall damage, but the power had shut off (yeah, another power problem). I held the power button in, but it wouldn't power on, even after several attempts.

I plugged the charger cord in. The display came on, showing 0% power. It tried to boot, then quickly shutoff. It just repeated this cycle continuously. So there was some level of system function, but no power.

The shock of impact had apparently knocked something loose inside, which I figured was most probably the battery connection. Dead phone. Stupid reason. Words were said.

And this is a sealed phone, not a removable battery like older models. More words were said.

I emailed my wife who was visiting some friends that my phone was dead. Ugh. I look forward to going to the cell phone store the way I look forward to going to a used car lot (although I'm being unfair, my last several experiences with them have been excellent).

But Internet to the rescue, I Googled "access battery in droid maxx" and found a page showing how to do it, plus someone in the comments had pointed to a YouTube video they liked better.

I read the page and watched the video. Yep, I can do that. Worst case, the phone is already dead, I can't make it any worse.

Since I didn't have a spudger or plastic pry tool as shown in the tutorials, I improvised one from a piece of 1/8" thick padauk, which is a very hard tropical wood. I split off a strip about 1/2" wide and beveled the end of it with a block plane.

Between that and a small flat screwdriver, I was able to get the case open. The metal driver tip did break a couple bits of plastic off the edge of the back, but no major damage.

This also shows why I ordered multiple tool sets (I've added an iFixit set to the shopping list on my Adafruit post, because it has pry tools). I used the smallest flat screwdriver from one set for lifting tabs, the T5 from another set to remove most of the screws, and the even tinier flat tip from a third set to get into the points of a T4 screw.

I used the compartments in the base of my Panavise to hold all the loose parts.

With all the screws removed, I separated the board from the display, and sure enough, I could see immediately that the tiny connector on the flat battery cable was loose on the back of the board. This wasn't the first time I had dropped it, so it probably loosened up progressively. Everything inside the phone is secured in place pretty tightly.

I pressed on it and it clicked positively into place. I pulled it back off to look at it, then clicked it back in. So I was confident that it was secure.

As I was handling the phone with everything folded together loosely, I accidentally held the power button in. I noticed the display come on, so I let it continue. It booted up (SIM removed), so power appeared to be restored.

I shut it down and reassembled the phone. The tweezers were a big help dropping the tiny screws back into their holes.

Popped the case back together, reinserted the SIM, and held the power button in. Voila! It powered on, connected to the network, and I was back in business, with a couple of quick tests to verify that things were working right and it was staying on. I texted my wife that I had fixed it.

That's an example of digging in and not being afraid to play around and see what's going on. It's all a learning experience!

Sunday, April 8, 2018

Limor Fried Is My New Hero

 
Meet Limor Fried, founder of Adafruit.

I'm cross-posting this to both my woodworking blog www.CloseGrain.com and my software engineering blog FlinkAndBlink.blogspot.com (under the LearnToCode label), because even though there's no woodworking in it, this is all about building stuff, so it bridges the worlds. It's the maker ethos.

If you're interested in learning to code, and actually building the stuff that you're coding on, this is for you. This is all about working on embedded systems, from the hobby level to the professional.

In Which I Find Out About Limor Fried

Allow me a moment of self-indulgent gushing admiration here. Or you can skip down to the real information that starts at the Electronics Learning Resources heading.

I admit to instant and total nerd-crush. Limor Fried, who goes by the name Ladyada online (for Lady Ada Lovelace, The First Programmer) is the founder of Adafruit.

Adafruit is a small electronics manufacturing company in Manhattan, NY, that focuses on teaching electronics to makers of all ages. You can read about them here.

Electronics is another of those hobbies that I wanted to pursue as a teenager, but never could due to lack of funds. Fortunately I've advanced beyond that impecunious stage of life, and seeing this has fired instant obsession (hence the shopping list below!).

I'm familiar with that feeling of obsession settling on my shoulders. It propelled me into hand tool woodworking, turning into a book. It propelled me into violinmaking. It propelled me into boatbuilding.

Each time, the pattern is the same. I buy a bunch of books, watch a bunch of videos, dig through a bunch of blogs and forums, then buy a bunch of tools and start playing. Last year it propelled me into small engine repair and oxy-acetylene welding after I found Taryl Dactyl (yes, blog posts will be forthcoming).

Now, in my copious free time (that's a joke, son), I'll finally be realizing that dream to get my hands dirty with electronics.

I owe this to Matt Pandina, whom we recently hired at work. It quickly turned out that Matt is a maker and likes sharing information. He has some nice stuff on Google Groups under the moniker artcfox (in fact, one of his articles was coincidentally the answer to the embedded systems programming problem I use when interviewing candidates!).

He made a comment about how Adafruit is doing manufacturing in Manhattan, and I asked, "Who's Adafruit?". That was all it took. Thanks, Matt!

I was tickled to read Fried's favorite quote in the Entrepreneur Magazine article about her:
“We are what we celebrate.” —entrepreneur and inventor Dean Kamen.
Kamen is one of my other heroes. She whose hero is my hero is my hero!

I managed to score his autograph at the 2015 MassMEDIC conference. I was at the 2015 Embedded Systems Conference (ESC Boston), which was being held concurrently at the Boston Convention Center.


When I saw Kamen listed as keynote speaker, I scooted down early and got a chance to talk to him and tell him I wanted to work for him (he probably gets a lot of stalker geeks like that!). Came close the following year, but logistics didn't work out.

Electronics Learning Resources

On the business side, Adafruit sells kits, parts, tools, and books. That's pretty cool (along with being able to pull off a manufacturing operation in Manhattan). But what's truly spectacular about them is their online learning resources.

Fried is a big proponent of open source, sharing the knowledge. So the Adafruit website is chock full of information. There's also an extensive YouTube channel.

You'll also finds lots of cross-pollination with others in the maker community. There are magazines, blogs, and videos by the score, by independent makers like Matt, and by larger organizations.

I've just barely begun to scratch the surface. This is great, because I know how to program embedded systems, but I don't know much about the components that go into them and connect to them. It's the combination of hardware and software that really makes something work.

Pretty much everything I know about digital electronics I owe to Forrest M. Mims and George Young 35 years ago. Now, after that brief hiatus, I can take the next step.

Basic Electronics Lab Skills


Step into Collin's lab!

Among the resources is a series of very accessible quick guides and videos by Collin Cunningham. Of particular interest to the electronics beginner such as myself is this set of basic electronics lab skills (you can scan through all these for quick grok of the big picture by setting the speed in the YouTube window settings (the gear icon) to 2x, then come back and watch at normal speed for a second pass):
  • Soldering and Desoldering: how to solder components together properly, and how to pull them apart for salvage and rework.
  • Surface Mount Soldering: how to solder surface-mount components.
  • Multimeters: how to use a meter for basic measurements.
  • Oscilloscopes: how to use an oscilloscope for advanced measurements and waveforms.
  • Hand Tools: the basic hand tools used for assembling and disassembling electronics.
  • Schematics: how to read schematics (no, they're not Greek!).
  • Breadboards and Perfboards: how to combine the parts on a schematic into a functioning circuit.
  • Ohm's Law: understanding the relationship between voltage, current, and resistance.
Once you have these skills, you are unleashed. Just like hand tool woodworking, it takes a little investment in tools and equipment, and a little time practicing with them.

These form the basis of the shopping list below. And of course they lead to lots of other interesting videos, like Collin's videos on the basics of various components:
There are also a number of other introductory Adafruit written guides by various contributors (as well as oceans of more specialized and advanced guides, check them out!):
Shopping List

These are the tools, equipment, supplies, and books to do the work. With the exception of the oscilloscope and logic analyzer, these are all links to the Adafruit shopping pages. Prices as of April 8, 2018.

Tools and equipment:
Consumable supplies:
Books:
Finally, here are some additional random useful items that they don't carry, all via Amazon:
Total cost: $1567 for everything (I ordered 2 spools leaded solder and 1 leaded Chip Quik, no lead-free items, 10 DC barrel jacks, and all the screwdriver/tool sets, since you never know which tips and shanks will fit, and some cases need special access tools to open), with free shipping from both Adafruit and Amazon, $10 for Saleae. Plus Adafruit threw in a free half-size breadboard and a Circuit Playground Express.

Back in my teenage days, $10 was a major expenditure, and $100 was simply inconcievable. This is starting to add up to some real money, but it will leave you armed with the tools, knowledge, and skills sufficient to launch a career.

The really nice thing is that Adafruit provides a curated list of things to choose from, so you're getting the benefit of their experience and recommendations, all guided by that maker ethos. That was a big plus for me.


Bridging three centuries of maker technology in my workshop.

You can read about my first use of these tools, since I needed them almost immediately.

For a review of the outstanding Charles Platt books listed above, see Review: Make: Electronics and Make:More Electronics.

For a useful set of resources to help you learn electronics, see Learning About Electronics And Microcontrollers.