Skip to main content
GameDev.net gamedev.net

RefCastObject dereferences a null context when the request-context callback returns null

Started by Jeremi247 Today at 9:52 AM 0 replies
Original Post
Jeremi247
Jeremi247

In asCScriptEngine::RefCastObject, the void opCast(?&out) fallback calls ctx->Prepare(universalCastFunc) on the result of RequestContext() without checking it for null:

// One last chance if the object has a void opCast(?&out) behaviour
if( universalCastFunc )
{
	// TODO: Add proper error handling
	asIScriptContext *ctx = RequestContext();
	ctx->Prepare(universalCastFunc);

RequestContext() returns whatever the application's callback from SetContextCallbacks gives it, and that can be null (for example a pool that is exhausted). The library's other callers of RequestContext() check for null (as_module.cpp, as_scriptobject.cpp, as_scriptengine.cpp), but this one doesn't. Besides the application calling RefCastObject directly, the ref add-on's cast also goes through it.

Repro

AngelScript 2.39.0 WIP (master at c9e74722), MSVC 19.44 (VS 2022), x64, static library from projects/cmake with default options.

#include <angelscript.h>
#include <cstdio>

struct A {};
struct B {};

// void A::opCast(?&out). Sets the output handle to null; never reached in the failing case.
void A_opCast(asIScriptGeneric *gen)
{
	*static_cast<void**>(gen->GetArgAddress(0)) = 0;
}

// A context pool that has nothing to give.
asIScriptContext *RequestContext(asIScriptEngine *engine, void *)
{
	// Workaround: never return null, fall back to a new context instead
	// return engine->CreateContext();
	return 0;
}
void ReturnContext(asIScriptEngine *, asIScriptContext *ctx, void *) { ctx->Release(); }

int main()
{
	asIScriptEngine *engine = asCreateScriptEngine();

	engine->RegisterObjectType("A", 0, asOBJ_REF | asOBJ_NOCOUNT);
	engine->RegisterObjectType("B", 0, asOBJ_REF | asOBJ_NOCOUNT);
	engine->RegisterObjectMethod("A", "void opCast(?&out)", asFUNCTION(A_opCast), asCALL_GENERIC);

	engine->SetContextCallbacks(RequestContext, ReturnContext, 0);

	A a;
	void *result = 0;
	printf("calling RefCastObject\n");
	fflush(stdout);
	int r = engine->RefCastObject(&a, engine->GetTypeInfoByName("A"), engine->GetTypeInfoByName("B"), &result);
	printf("RefCastObject returned %d, result = %p\n", r, result);

	engine->ShutDownAndRelease();
	return 0;
}

Expected

RefCastObject returns a negative error code and leaves result null.

Actual

a log of "calling RefCastObject" and then an access violation (exit code 0xC0000005) at ctx->Prepare(universalCastFunc), in both Release and Debug.

Sign in to reply to this topic.