Moving a camera

So far I've used a static camera which simply sits at Vec3(0.0f) and stares down negative Z axis. I've got geometry like cubes to play correctly when translating, scaling, rotating etc, but now I want to try get the camera move correctly.

I am using glm, and to get a view matrix I am using the glm::lookAt method with the Vec3 data provided from my camera struct

Mat4 CreateViewMatrix(const Camera& camera)
    {
        return glm::lookAt(camera.CameraPosition, camera.TargetVector, camera.UpVector);
    }

Here is my camera definition:

/* Camera definition */
    struct Camera
    {
        Vec3 CameraPosition;
        Vec3 TargetVector;
        Vec3 UpVector;

        Camera();
    };


    /* Camera inlines */
    inline Camera::Camera() : CameraPosition(0.0f), TargetVector(0.0f, 0.0f, -1.0f), UpVector(0.0f, 1.0f, 0.0f)
    {
    }

I basically try to move it like this between the rendering:

 // move camera
            if (evnt.KeySymbol == LEFT)
                activeScene->GetSceneCamera().CameraPosition -= Vec3(0.1f, 0.0f, 0.0f);

            if (evnt.KeySymbol == RIGHT)
                activeScene->GetSceneCamera().CameraPosition += Vec3(0.1f, 0.0f, 0.0f);

            if (evnt.KeySymbol == UP)
                activeScene->GetSceneCamera().CameraPosition -= Vec3(0.0f, 0.0f, 0.1f);

            if (evnt.KeySymbol == DOWN)
                activeScene->GetSceneCamera().CameraPosition += Vec3(0.0f, 0.0f, 0.1f);

but this creates some strange results. For example, instead of the 'strafing' effect I would expect when moving the camera on the x-axis, it is as if I am rotating it, standing still. And when I try to move it in Z-axis, I really cant quite explain how it looks like, kinda like 'zooming' down the Z axis or something..

So I guess it's not as easy as I think?