automated GUI testing

so how do you test your games automatically ?

Hi,

In order to test automatically, you need to define the list of tests needed. These tests should be about wanted and unwanted behaviours.

Example with a Button:
- wanted behaviour: when the user clicks on a button and the button is focusable and visible, it should depress, emit a click sound and signal its status change to the program.
- unwanted behaviour 1: when the user clicks on a button and the button is NOT focusable, nothing should happen.
- unwanted behaviour 2: when the user clicks on a button and the button is NOT visible, nothing should happen.

Once all these tests are listed, you script these tests by simulating mouse move and clicks (sending yourself timed events instead of reading the real mouse states and position) and log all test results. Note that scripted tests are never definitive: user testing will always show new wanted and unwanted behaviours. You will have to add new test scripts to your testing unit to take care of new user behaviours.

Ghostly yours,
Red.

I am using the the following procedure to test my GUI stuff:
I try to separate the logic from the GUI ( for instance by using a pattern
like MVC ). Now I can test my logic against my wished bahaviour by using a test-script.
And I am simulating the user GUI by simulating user actions. Here we are able to verify the result by a screenshot diff for each widget.
If both works fine I am starting integration tests. An integration test replays a defined user story with a well-defined result at the end, where a lot classes of the logic and the GUI are involved. If the wished result is there the test passed. The hard part is to automate the integration tests and the verification because you have to verify GUI and logic interactions and screens. Maybe something like a written log could be useful. If a codepath was passed a log mechanism writes this into a protocoll file. At the end you can diff the protocoll against an older protocoll where the result was fine.
But this is just one approach :-).

Kimmi

hmm this sounds pretty realistic and doable. I guess I could hook up pretty fast something using Perl with Win32::GuiTest and Win32::Screenshot and Text::Diffsome which would work the way you describe.

I think Win32::GuiTest will allow me to send keyboard and mouse events. I hope it can allow me to do drag & drop.

I'm afraid however it can't be generalized to a game.
For example if you have some particle system it will have a degree of randomness in it and the screenshots won't match.
But basically for a GUI which has nothing unpredictable going on it should work
fine.

Are you using some testing framework for this , or did you write it ? If you wrote it , what language ? What language do you write the test-scripts in ?

You also suggest to make diffs of screens and also diffs of protocol files, isn't that somehow redundant or am I missing some point ?

Quote:

Original post by Kimmi I am using the the following procedure to test my GUI stuff: I try to separate the logic from the GUI ( for instance by using a pattern like MVC ). Now I can test my logic against my wished bahaviour by using a test-script. And I am simulating the user GUI by simulating user actions. Here we are able to verify the result by a screenshot diff for each widget. If both works fine I am starting integration tests. An integration test replays a defined user story with a well-defined result at the end, where a lot classes of the logic and the GUI are involved. If the wished result is there the test passed. The hard part is to automate the integration tests and the verification because you have to verify GUI and logic interactions and screens. Maybe something like a written log could be useful. If a codepath was passed a log mechanism writes this into a protocoll file. At the end you can diff the protocoll against an older protocoll where the result was fine. But this is just one approach :-). Kimmi

I also made a GUI in SDL (with OpenGL) once, and also made a unit test for the GUI, and it works like this:

There's an input class that gets mouse location and button presses and such from SDL. Well, for the unit test I just use a mock implementation of this input class instead, where I can set the coordinates of the mouse cursor myself. Then I do tests like setting the mouse coordinates to something over the button, simulating a left mouse button press, and checking if the GUI button then correctly responds to this. I did this too for things like dragging a window, pressing at the location of the button while a window is over that button blocking it, and so on...

With a framework in SDL this is possible, I have no idea how something similar could be done with a win32, Qt or GTK gui...

Quote:

With a framework in SDL this is possible, I have no idea how something similar could be done with a win32, Qt or GTK gui...

At the very least you can use the OS functions to create a separate program that will send events to the target window using, for example, Xlib's XSendEvent or Win32 SendMessage/PostMessage. There are a few ways to get the window handle/identification/whatever, and once you have it, you can simulate whatever behaviour you want by sending the right messages to the application.

I used different approaches for the scripts. I tried out python and xml-configurations. But I prefer python. I wrote a simple bindtool to generate a scripting interface, but of course you also can use a third-party-library like boost::python.
To simulate the user input we wrote a tool with creates a journal-input-hook ( the main reason for this was just our wish to do it ). But of course you also can use the perl- and the screenshot-package. For the unittests I used cppunit.
The interesting part for all these stuff is your architecture: you should have different layers where each single layer should be able to test on its own ( for instance by using mock-objects for the used interfaces, dependency injection is a important thing, too ). So you should be able to generate the log output in the mock-objects. A test-script can call this layer-tests one after another in a batch-script.
The hardest part is to test the visual feedback. If you are using randomized particles it is really hard to automate the testings. But of cource you can test the separate steps like create one particle, move it arout etc. . For the integration test ( test the complete subsystem ) an approach could be to record a small movie with the generated particles in each testrun. And every morning / week / whatever you have to watch those generated movies if everything looks as you expected.

You can spend a lot of time for this, so maybe there is a point where a tester who is validating some integartion test is cheaper than a completly automated test enviroment ( which IMHO is impossible to build, you cannot avoid the tester completly ). That is the reason why we have a crew of integrators, which are validating new stuff once a week.

Kimmi

Quote:

Original post by Red Ghost move and clicks (sending yourself timed events instead of reading the real mouse states and position) and log all test results

Quote:

Original post by Kimmi Maybe something like a written log could be useful. If a codepath was passed a log mechanism writes this into a protocoll file. At the end you can diff the protocoll against an older protocoll where the result was fine.

Quote:

Original post by Kimmi So you should be able to generate the log output in the mock-objects.

ok, can you recommend me a good cross-platform opensource logging library ? I could write my own but I'd prefer to use something that someone thought-out better because I want to use it but it's not my main focus. my main focus is testing.

I've written a quick script in Perl to test my app:

package Mole::Test;use strict;use warnings;use feature ":5.10"; # always use this for given/whenuse Win32::GuiTest qw(:ALL);use Win32::Clipboard;use Imager::Screenshot;use Data::Dumper;use Digest::MD5;use Test::More;use Moose;use GD;use Carp;# the HWND of the wfsdl Windowhas wfsdl => (    isa => 'Int',    is  => 'rw',    default => undef,);has image => (    isa     => 'Any',    is      => 'rw',    default => undef,);# focus on the window with the HWND given as parametersub focus {    my ($self) = @_;    SetForegroundWindow($self->wfsdl);    MouseMoveAbsPix((GetWindowRect($self->wfsdl))[0,1]);}# find the window needed, if not fire up wfsdl.exe and then find its HWNDsub BUILD {    my ($self) = @_;    FindWindowLike(0, "readme|README", "Notepad");    $self->wfsdl(   (FindWindowLike(0, "", "SDL_app"))[0]   );    unless($self->wfsdl) {        system("start wfsdl.exe");        sleep(1);        $self->wfsdl(   (FindWindowLike(0, "", "SDL_app"))[0]   );    }    $self->focus;    print "HWND of wfsdl application is ".$self->wfsdl."\n";}sub click {    my ($self,$x,$y,$type) = @_;    SendMouse ( "{REL$x,$y}" );    given($type) {        when(/^LUP$/)   { SendLButtonUp();      }        when(/^LDOWN$/) { SendLButtonDown();    }        when(/^RUP$/)   { SendRButtonUp();      }        when(/^RDOWN$/) { SendRButtonDown();    }        default         {                       }    };}sub drag_drop {    my ($self,$x1,$y1,             ,$x2,$y2,             ,$code    ) = @_;    $self->click($x1,$y1,"LDOWN");    $self->click($x2,$y2,"LUP");    $self->focus;}sub test_button_move {    my ($self) = @_;    $self->drag_drop(51,71,200,30,        sub{}    );    $self->get_picture();}# read pixel in Uint32 format from last image taken from get_picturesub read_pixel {    my ($self,$x,$y) = @_;    croak 'no last image' unless $self->image;    my $index = $self->image->getPixel($x,$y);    my ($r,$g,$b) = $self->image->rgb($index);    return $r<<16 + $g<<8 + $b;}# TODO: add bmp2png utilitary to subversion code## NOTE:# this basically selects the window and does PRTSCR and pastes# in a bmp file the clipboard and then converts to png and saves the png# object in the image attributesub get_picture {    my ($self) = @_;    my $clip = Win32::Clipboard->new();    $self->focus;    SendKeys("%{PRTSCR}");    my $fname = "$ARGV[ 0 ]-@{[ time() ]}";    if( my $bitmap = $clip->GetBitmap() ) {        local $\;        my $bmpfile = $fname . '.bmp';        open my $bmp, '>', $bmpfile or die $!;        binmode $bmp;        print $bmp $bitmap;        close $bmp;        print "Bitmap written to $bmpfile\n";        system "bmp2png $bmpfile";        print "Converted to $fname.png";        $self->image(GD::Image->newFromPng("$fname.png"));    }}sub run_tests {    my ($self) = @_;    $self->test_button_move;}package main;my $mole = Mole::Test->new;$mole->run_tests;done_testing();