Writing Tests For Your Code

What kind of code are you trying to test? Some things are much more amenable to testing than others. Some things, like graphics and UI, shouldn't be automatically tested as systems (you can unit test components where it makes sense, but know when to draw the line).

If you have problems coming up with ways to test code, it's probably a strong sign that the API for that code is shitty and needs to be redesigned. Clean APIs practically beg you to test them.

What kind of code are you trying to test? Some things are much more amenable to testing than others. Some things, like graphics and UI, shouldn't be automatically tested as systems (you can unit test components where it makes sense, but know when to draw the line).

If you have problems coming up with ways to test code, it's probably a strong sign that the API for that code is shitty and needs to be redesigned. Clean APIs practically beg you to test them.

A while back, when I was writing a 64bit integer class, testing every function was essential; I caught a lot of bugs. However, I was frequently getting burned out with tests that were no brainers, and the tests were fairly arbitrary. I'm sure you can imagine how boring it is to write 400 or so tests to ensure that a plain int is working. Same concept.

Then, I was writing a huge battery of tests for my string class. Several hundred tests as well, to make sure that every method worked. Again, all necessary, because some of them didn't until I did tests for them. It was hugely demoralizing to pick an almost random way to test if functionality works with trivial test vectors just drawn off the top of your head.

Now, I am writing tests for my stream class, to test all of its functionality, with the intention that when I finish porting the backends from the old C interface to the new interface, I'd be able to reuse most of the tests. Creating test files containing purely arbitrary data and magic numbers to test if I can read them, and then writing data out to be examined is a tiring thing, and the severe lack of motivation makes it take forever. It doesn't help that I am 99% sure that it all works, but I need to prove it regardless.

So, I guess I should revise my statement. It is incredibly easy to implement a test for these interfaces, but the problem is more on the human side, where I need to keep coming up with unique tests to challenge the interface and prove that all of the code is working properly, which totals in several hundred per module on things you would expect to work.

So, for these cases, from where do you devise your sufficiently unique data to be used for tests, since these classes take data in, manipulate it, and give data out, and how do you overcome the apathy of testing hundreds of conditions that should obviously work?

I advocate the test-first approach.

Write the test, then write the code that satisfies the test. This forces you to think about how the code is used, how it's tested, exactly what it's effects and responsibilities are and whether or not it has too many of them. Ugly tests indicate the code under test is ugly. The solution is to clean the code, and make the tests just as beautiful. If you can't think of a test, then you don't adequately know what the thing your trying to write is supposed to be doing, what it is responsible for.

You may not be able to test certain aspects of the lowest levels of your code automatically (I disagree with this notion, but accept it may be more effort than it's worth in many situations), but anything using those levels can be tested by mocking those lowest levels. This keeps responsibilities clean and this forces you to keep the low level at the low level, and separate your game logic from your inputs and outputs. And that's the main advantage of writing tests. If you do it well, commit to it and importantly if you accept the benefits given to your code by taking the time to refactor, the code practically keeps itself clean. The time invested in tests and refactoring is regained in the reduced number of bugs, the less time spent in the debugger and the ease of making changes to clean code.

And if you need to change legacy code that lacks tests, wrapping it in tests before making the changes also accomplishes this by highlighting the issues and providing a safety net as you refactor.

Things that violate the single responsibility principle and inversion of control become so cumbersome to test that you need to clean them up to maintain productivity. Which is an important thing to remember: When the tests become ugly, it's not the fault of testing, but the code under test.

I advocate the test-first approach.

Yes, TDD has its benefits. I like to use it, and it has done wonders for my quality of code. I've been trying hard to maintain my tests and my documentation. Unfortunately, this comes back to bite me, because while I'm motivated to write some more features, I won't move on until I document and test the just written features, and that blocks me if I lack the motivation to write tedious tests. However, right now, I'm refactoring legacy code, and writing tests for untested code. This will run in the thousands.

Which is an important thing to remember: When the tests become ugly, it's not the fault of testing, but the code under test.

I wouldn't say the tests are ugly. Would having test text in a file, reading portions of it using input methods, and comparing it against what the correct answer should be count as ugly? Unfortunately, the problem is that there are a lot of input methods. And output methods, and combinations of both. I need to make sure they all work. Testing one would be great, testing one hundred of them, and making sure that they set the right flags and have the correct side-effects exacerbates the number of things I need to check.

Should I not be checking to see if each method on an I/O stream works, even though if I test a higher level, any one of them could be the reason that a broader test fails?

So this is actually a hard problem. Everyone who works on low-level library or heavily-used code will encounter this.

My personal approach (I'm writing a compiler) is to not test exhaustively. I write a test in one of three situations:

• I need to make sure the functionality basically works (smoke test)[/*]

I think of a potential issue that I want to verify - either I catch a bug (lucky) or I prevent a future regression (also good)[/*]

I find a bug and want to write a regression check for it[/*]

Occasionally, and by this I mean almost never, I will find the motivation to write a test just for the hell of it. I almost never test every single permutation of every factor I can think of.

Part of this is understanding your own code deeply enough to have an intuition for how much bang-for-buck your tests can get. For some things, testing every permutation is a total waste, because you can test, say, 5% of the permutations and get 99% coverage. 100% coverage is a silly goal that only ignorant ivory-tower types seriously pursue. Everyone in the business of getting stuff done knows that you aspire to maximal coverage, but don't become a slave to that magic 100%. The dirty truth is that even if you reach 100% coverage, the next line of code you add or change will break your perfect number. And you probably just think you have 100% coverage anyways - chances are you don't, especially in complex languages like C++ where control flow can do bizarre things due to chasing wild vtable pointers, or exception handling, or whatever else.

From the Wikipedia article on Smoke Testing, under Software Development, the key line that stood out to me is: "The purpose is to determine whether the application is so badly broken that further testing is unnecessary."

If I have this correct, the term "smoke testing" in programming means to test the most important features and most used code. It doesn't quite strike me as being the same as using smoke to find leaks in pipes or vents, being that the latter two would comprehensively fill out the entire system and find almost every crack where there's adequate gas pressure.

However, I might not be looking at it the right way. Perhaps the two are analogous in that it is just a dry run, to ensure the pipes can at least hold water, before making sure that they get to their destination, and not simply counting the number of places that are implicitly checked for leaks.

However, I'm not sure how to do this without being exhaustive. It seems fairly important to test if my file reading functions actually read from the file, before doing anything else.

For some things, testing every permutation is a total waste, because you can test, say, 5% of the permutations and get 99% coverage.

So, am I to take this to mean that I wait until a bug occurs in the 1%, and fix it faster than it would take to test it all (and repeat until no bugs surface over time)? One of the reasons I started testing was to be confident that it all worked (something I lacked before I started the massive refactor). I understand the logic behind testing the most used code, to get the most value for the least effort, but how do I find the middle ground between saving time and peace of mind? I'm thinking that there is none.

To the downvoter, some explanation would be nice, because I was simply asking for help. By all means, if you find anything within that post that was factually inaccurate, I would appreciate you letting me know so that I can correct it.

I am in agreement with ApochPiQ here. He is talking about what I tend to refer to as "strategic unit testing". I describe it as testing the basic functionality and testing the things that you intuitively feel are most likely have bugs, either now or later. What this usually amounts to is testing the happy "garden path" case and testing of the hairy edge cases that you were hoping to handle, this is where off-by-one errors, exceptions and infinite loops usually lurk.

Other than that I like to write tests when bugs are found to prevent regressions and even to help track bugs; if you think a certain scenario is playing out that results in something bad happening then write a test for it. That way you'll have proof one way or another and an easy way to rapidly re-test as you go about fixing it.

Testing, to me, feels like tightening screws and unless your requirements are fixed and long-term (mine are never like that) then you don't really want to be over-tightening any one screw in particular. You'll come in the next day and need to change stuff to cope with new requirements anyway.

When are you done testing something? When you get bored of it. At that point pat yourself on the back that you wrote any tests whatsoever and that you have strategically prioritised writing the most important tests first, then move onto doing the next thing.

When are you done testing something? When you get bored of it. At that point pat yourself on the back that you wrote any tests whatsoever and that you have strategically prioritised writing the most important tests first, then move onto doing the next thing.

I guess that makes sense. When not using TDD, it seems that having tests for 100% coverage is by no means necessary, and it isn't a blemish or relapse in trying to maintain a discipline of having tested and proven code. Knowing that hardly anyone else tests every method is reassuring. Thank you, all that posted, for your help. I think I'll move on, now that I've covered the most common functionality, then.