Software testing

Oh the joy of testing, first you have to accept the fact that everyone is going to hate you. The devs will hate you for bringing them more bugs, management will hate you for holding up the process and the end users will hate you for not finding all the bugs. But with that being said good software can't be made without a good tester.

Step one: Testing plan. You need to get a hold of the project design document and start to work on your testing plan. Look at each feature and see how it is suppose to work and try to figure out ways to break it. Same goes with the UI design, look at every control and combinations of controls and see what might not work together. Let's say you have a input field for number of hours worked; what happens if you leave it blank, how about a negative number, how about 165,653,465,394,285,184,866. You can never predict what a "looser", oh I mean
"user" is going to do so you have to come up with things that just don't make sense.

Developers are not very good at testing because they normally just test it as it is suppose to work. They also are suppose to do "unit" testing, but seem to let it slide. By the way I am a developer.

There are many differnt "styles" of testing, but from what I have seen what matters most if orginization and a methodalogy. You need to have a plan. You will be doing system test, which is testing how the whole thing works, but just as important is regression testing (a lot of folks seem to get a little slack here). Regression testing is going back and making sure that everything that was tested before still works. I believe that this is where scripted testing shines.

There are lots of good books on testing and software desisgn out there. A good software design has a testing plan in place. There are also lots of articles on the web.

The other thing that you will need for what you are doing is a knowledge of accounting. Think like an accountant and you will find a lot of bugs before your users do.

theTroll

Testing & Bugfixing in our company: About 20% of actual development time; more if there are surprisingly many bugs for some reason.

You should google "Acceptance Testing," as it sounds like that more specifically describes what you're going to be doing.

I've found that developers will naturally gravitate towards testing practices that are useful. If you're writing code, check out "unit tests" and "automated testing." There's lots of stuff in the Xp literature about it under Test-Driven Development.

Unit testing really needs to be done by the developers. As an independent tester you need to focus on system-level test cases.

Ideally you want to be able to run everything in some sort of simulated environment with lots of logging features and the ability to capture and replay what you did. That way when you find a problem you can repeat it and track down what went wrong. We see lot's of "bench testing" where the testers observe a problem, but can not reproduce it nor describe a root cause to the developers (read wasting time).

The most basic type of system checks the state of the output, runs a test, and checks the new state matches the expected resultant state. That works good for non-time-critical applications. For time-critical applications you really need a logging mechanism that tracks the state of the system over time and ensures everything happens when it is supposed to.

A single system test will test 45%-65% of the system. Use system-level test and measure code-coverage. Keep adding system level test until you're testing all the requirements. Then you can turn a critical eye to any uncovered code. Do NOT just write test-cases to hit this code - it will be incredibly tempting to do so. Resist this. You need to determine why that code was not called. Did you forget to test a requirement (go find it!)? Is there a missing requirement (go add it!)? Should this code be removed (tell the developers they have dead-code)?
If everyone is doing their job well, the code that is left untested will be "this should never happen" code (e.g. default case statements that throw an exception etc...).

If you write unit test you can guarantee 100% coverage, but generally you are required to do acceptance (system-level) testing anyway. If you measure the low-level coverage while doing this testing you can demonstrate that you do not need to unit-testing on 50% to 90% of your code since it is already tested by the system-level test-cases. (I want to stress that we routinely measure 45%-65% code coverage from 1 system-level test. Compare the work of creating 1 system-level test-case to the work of unit-testing 45%-65% of the code.)

The key to making this work is automated system-level regression testing.

I've read a complaint a couple of times about unit-testing that it feels like it's a waste of time, and the above numbers are analytically why it is. You only have so much man-power & time to test, and I am now thoroughly convinced you are best off focusing that effort on the system not classes or methods.
Maybe that's what the XP guys meant all along, but it's not what we find organizations doing in practice.


For a video games, I think you should record movies of what is happening and save a timeline of all the entity information such as location and various properties of the objects (color? speed? health?) every frame (every "physics" frame if it's different than the graphics frame-rate). Then you need a report that tells you how different the movies & waveform data are on subsequent regressions. Then you go down the list and look the differences and determine if the new results are acceptable or not. Unacceptable results become issues in the bug tracking software and acceptable results are then stored for future regressions.

Shannon, could you elaborate some more on why you don't think unit testing is wortwhile? I've been writing automated unit tests for close to two years and I can't stand it when I have to work in code that doesn't allow me the luxury. It's just so... unsafe. The primary benefit I see to unit tests is that, because they are so focused, you have a much better idea of what went wrong during a feature addition\change or refactoring when a test fails.