Welcome back to Tech Tuesday. We are still in the process of wrapping up the initial series on programming. In the interim review I had identified a couple of high level questions that had not yet been answered dealing with code that has been written. Just before the end of the year we looked at bugs and debugging and I mentioned that there are two types of testing: unit testing and black box testing. More on these today.

One key purpose of testing is to identify bugs before they are exposed to endusers. Software systems are often extremely complex and even seemingly small changes can have large impacts (future posts will deal with techniques for reducing complexity and dependencies). As a result we cannot rely on the idea that something was working previously and so should still be working now following a “small change.” Instead after any change we need to test again to check for new bugs and also for regression – features that stop working or previously “fixed” bugs that are now reappearing.

For a long time most of the testing that was done was black box testing. That means testing the entire system from an enduser perspective and treating the code itself as a hermetic black box (hence the name). Often this black box testing was done by separate Quality Assurance (QA) departments and carried out manually. I once worked on a large DoD project that involved hundreds of pages of manual test procedures that would be carried out by a separate contractor and literally reported on paper. There were many problems with this approach including infrequent testing (in combination with a waterfall model of development), human testing errors and most importantly an adversarial relationship between the QA and development teams. Over the years tool makers jumped into this as an opportunity and today if you are not running mostly automated black box testing you are doing it wrong.

Unit testing emerged much more recently and refers to writing automated tests for small units of code. They are made part of an automated tool chain so that they can be invoked whenever a change has occurred. It is somewhat strange that unit testing took a long time to emerge because in manufacturing automated checks of the quality of intermediate steps has been a key component of achieving overall quality for a long time. Part of the reason is was the separation of QA and development in the waterfall model but it was also difficult to implement this with bare bones lower level languages such as C.

In some approaches to programming the unit tests are written before the code for the unit. It is similarly possible to create black box test script based on a mockup of the user interface. There are many advantages of thinking about testing up front this way. Among other things it forces smaller units of code and more precise use cases. The tests can also serve as documentation. By making both product (black box) and development (unit) responsible for testing it removes the organizational friction of a separate QA department and dramatically speeds up the feedback cycle.

Bringing this all back to the original analogy of programming being similar to telling a human what to do, testing is thinking up front about how you will know if the person is doing what you asked them to do. That can be done by examining the output of their work (black box testing) and by looking at the intermediate steps they are taking (unit testing). Both clearly play a role in every day life!

With that we have almost reached the end of this initial series on programming. I am planning to write one more post next Tuesday on an important topic which is the source of much of the difficulty in debugging and testing: concurrency – the computer trying to do many things at the same time. This is similar to one person receiving instructions for what to do from multiple bosses which we all know to be difficult!

Welcome back to Tech Tuesday. We are still in the process of wrapping up the initial series on programming. In the interim review I had identified a couple of high level questions that had not yet been answered dealing with code that has been written. Just before the end of the year we looked at bugs and debugging and I mentioned that there are two types of testing: unit testing and black box testing. More on these today.

One key purpose of testing is to identify bugs before they are exposed to endusers. Software systems are often extremely complex and even seemingly small changes can have large impacts (future posts will deal with techniques for reducing complexity and dependencies). As a result we cannot rely on the idea that something was working previously and so should still be working now following a “small change.” Instead after any change we need to test again to check for new bugs and also for regression – features that stop working or previously “fixed” bugs that are now reappearing.

For a long time most of the testing that was done was black box testing. That means testing the entire system from an enduser perspective and treating the code itself as a hermetic black box (hence the name). Often this black box testing was done by separate Quality Assurance (QA) departments and carried out manually. I once worked on a large DoD project that involved hundreds of pages of manual test procedures that would be carried out by a separate contractor and literally reported on paper. There were many problems with this approach including infrequent testing (in combination with a waterfall model of development), human testing errors and most importantly an adversarial relationship between the QA and development teams. Over the years tool makers jumped into this as an opportunity and today if you are not running mostly automated black box testing you are doing it wrong.

Unit testing emerged much more recently and refers to writing automated tests for small units of code. They are made part of an automated tool chain so that they can be invoked whenever a change has occurred. It is somewhat strange that unit testing took a long time to emerge because in manufacturing automated checks of the quality of intermediate steps has been a key component of achieving overall quality for a long time. Part of the reason is was the separation of QA and development in the waterfall model but it was also difficult to implement this with bare bones lower level languages such as C.

In some approaches to programming the unit tests are written before the code for the unit. It is similarly possible to create black box test script based on a mockup of the user interface. There are many advantages of thinking about testing up front this way. Among other things it forces smaller units of code and more precise use cases. The tests can also serve as documentation. By making both product (black box) and development (unit) responsible for testing it removes the organizational friction of a separate QA department and dramatically speeds up the feedback cycle.

Bringing this all back to the original analogy of programming being similar to telling a human what to do, testing is thinking up front about how you will know if the person is doing what you asked them to do. That can be done by examining the output of their work (black box testing) and by looking at the intermediate steps they are taking (unit testing). Both clearly play a role in every day life!

With that we have almost reached the end of this initial series on programming. I am planning to write one more post next Tuesday on an important topic which is the source of much of the difficulty in debugging and testing: concurrency – the computer trying to do many things at the same time. This is similar to one person receiving instructions for what to do from multiple bosses which we all know to be difficult!