Animation-transitioning

You don't mention what API you're using (Animation, AnimationTrack, and KeyFrame are unknown object types at this point), so it's impossible to tell what the expected results are for AnimationTrack.add( KeyFrame ) and Animation.add( AnimationTrack ).

However, just making a guess from only what you've posted, the equation for valueToModify looks suspicious. I.e., if lastKey.value and nextKey.value are matrices, how those matrices are composed being unknown, the purpose of multiplying each matrix by a scalar, and then adding those two results, is unclear.

A more common approach is with keyframes to be comprised of separate scaling, rotation and translation values. Interpolation between two keyframes is then done by interpolating the component values and then, if a matrix result is required, composing a matrix from the results. E.g., in pseudo-code:

scale = Interpolate( lastKey.scale, nextKey.scale, interpolationValue);
rotation = SLERP( lastKey.rotation, nextKey.rotation, interpolationValue);
translation = Interpolate( lastKey.translation, nextKey.translation, interpolationValue);
matrix = ComposeMatrix( scale, rotation, translation );

Commonly, in each keyframe, scale and translation are stored as vectors, and rotation is a quaternion.

Alternatively, depending on how the rest of your code is implemented, you could interpolate vertex positions. E.g., given an input vertex V,

Vlast = V * lastKey.value;
Vnext = V * nextKey.value;
Vresult = Vlast * (1.0f - interpolation) + Vnext * interpolation;

This article, which discusses blending animations together, may give you a better idea how the above processes can be implemented.

I'm sorry, I can see the code beeing confusing. I tried to keep it simple, but that doesn't seem to be a good idea.

I am not storing the translation + rotation and then composing matrices from it each frame, I am simply using the finished matrix for each bone from the start.

Therefore, I am animating the 12 numbers of the matrix that change (rightmost numbers appear to be irrelevant).

Those 12 numbers are the ones I am interpolating between. Is this ok to do, or am I screwing something up in doing so?

I am not storing the translation + rotation and then composing matrices from it each frame, I am simply using the finished matrix for each bone from the start.

This will simply not work, if you want more advanced stuff like animation blending. The interpolation/blending between bones is always done locally, therefor if you do it globally(=working with the final matrix), bones will suddenly start to travel in really funny ways (especially extremities like hands/feets etc.).The global interpolation could result in temporary disconnected bones which looks like shrinking/stretching of mesh parts.

In addition to hierarchical transformations, if you do not enforce things like orthonormality in the matrix you will get strange stretching/shearing effects if you try to interpolate large rotations.

It seems that you are using bone transformation matrices in world or modelling space as Ashaman73 mentioned. All of the professional animation systems store the keyframe matrices relative to binding pose or maybe reference pose. During animation blending the keyframe matrices are being interpolated relative to ref or binding pose and the final pose will be calculated in the modeling or world space to be passed to the skinned mesh renderer.

Maybe you are calculating the blended pose in the world or modelling space and add it to the animation track and the animation system you are using is assuming that your blended transformation is calculated in local space and then it adds that local transformation to the final world space matrix.

For a more accurate blending you can interpolate position rotation and scale tracks separately just like what Buckeye suggested. However the way you are interpolating is not going to produce bad results on scale and translation if you are interpolating between two correct matrices. It can cause non-linear interpolation on rotations because you are using LERP instead of SLERP.