Problem with normal mapping.

I'm trying to implement normal mapping in deferred rendering.

The tangents are generated with Assimp. It returns HasTangentsAndBitangents to true.

In this specific scene I have a bunch of plants, the statue and the rock loaded with Assimp. (.obj file)

And other simple generated geometry like spheres, boxes and planes.

I have 2 problems:

1. The Loaded models don't have correct tangents as opposed to the generated geometry. Should I be calculating the tangents manually without using Assimp?

2. Even if the tangents are correct in the generated geometry I'm getting really noisy specular with the normal maps. I tested the albedo + normal map in Unity to see if the problem was in the normal map, but it was woking as expected, nothing was wrong with it. It just gives really sharp results in my engine, really noisy. Could it be the lighting calculation that is wrong? But judging from the debug normals it seems like it's working.

Here are the normals, with and without normal map.

Without normal map, just surface normal.

With normal map.

Notice how the normals in some models are completely wrong.

My shader calculation for the final normal output is:

In HLSL

float3 CalcNormal(float3 normalMap, float3 surfaceNormalW, float3 tangentW)
{
	float3 normalT = 2.0f * normalMap - 1.0f;

	float3 N = surfaceNormalW;
	float3 T = normalize(tangentW - dot(tangentW, N) * N);
	//This is why we don't need provide binormals, we can calculate it.
	float3 B = cross(N, T);

        //In case of mirroring
	if (dot(cross(N, T), B) < 0.0)
		T = T * -1.0;

	float3x3 TBN = float3x3(T, B, N);

	float3 result = mul(normalT, TBN);
	
	return result;
}