[Lua] Runtime compiling from C/C++

Recently I've been interested in the network applications of Lua for prototyping out ideas in a server/client environment. I was thinking something along the lines of custom logic generated on a server being sent to a client for execution. For simple security reasons, it'd be best if this logic were not sent in a text format that could easily be modified by anyone, but rather in a compiled form. In order to do that though, I needed the ability to generate compiled Lua bytecode at runtime (from C/C++) without having to use the Luac program. I didn't want to have to write the logic to a file and then read in the compiled results to use. I definitely didn't want it to be bound by disk i/o either. However, I've not really been able to find any existing code that actually does this or any topics of the potential uses of it. So, I spent a little time hacking away at Luac's code to come up with a solution. This is what I ended up with, it seems to work fine, but I've not gone through any intensive testing yet. It is really just a basic PoC project that could be extended in many ways. Source:


extern "C"
{
	#include "zlib/zlib.h"

	#include "lua/lua.h"
	#include "lua/lualib.h"
	#include "lua/lauxlib.h"

	#include "lua/ldo.h"
	#include "lua/lundump.h"
}

#include <string>
#include <sstream>

#include <stdlib.h>
#include <memory.h>
#include <string.h>

// Formats a stream of data into a hex/ascii dump
std::string DumpToString(void * data_, size_t count);

typedef struct TStreamBuffer
{
	const char * src;
	size_t sze;
	size_t ind;
	char buff[LUAL_BUFFERSIZE];
} TStreamBuffer;

typedef struct TCompiledBuffer
{
	char * ptr;
	size_t sze;
	int result;
} TCompiledBuffer;

int MyCustomWriter(lua_State * L, const void * src, size_t sze, void * userdata)
{
	TCompiledBuffer * cb = (TCompiledBuffer *)userdata;
	UNUSED(L);
	cb->ptr = (char *)realloc(cb->ptr, cb->sze + sze);
	if(cb->ptr == NULL)
	{
		return 1;
	}
	memcpy(cb->ptr + cb->sze, src, sze);
	cb->sze += sze;
	return 0;
}

const char * MyBufferStreamer(lua_State * L, void * userdata, size_t * sze)
{
	TStreamBuffer * sb = (TStreamBuffer *)userdata;
	*sze = sizeof(sb->buff);
	UNUSED(L);
	memset(sb->buff, 0, sizeof(sb->buff));
	if(sb->ind >= sb->sze)
	{
		*sze = 0;
		return NULL;
	}
	if(sb->sze - sb->ind < *sze)
	{
		*sze = sb->sze - sb->ind;
	}
	memcpy(sb->buff, sb->src + sb->ind, *sze);
	sb->ind += *sze;
	return sb->buff;
}

TCompiledBuffer CompileString(lua_State * L, const char * input)
{
	const Proto * proto = 0;
	TCompiledBuffer cb = {0};
	TStreamBuffer sb = {0};
	sb.sze = strlen(input);
	sb.src = input;
	cb.result = lua_load(L, MyBufferStreamer, &sb, "");
	if(cb.result != 0)
	{
		return cb;
	}
	lua_lock(L);
	proto = clvalue(L->top+(-1))->l.p;
	cb.result = luaU_dump(L, proto, MyCustomWriter, &cb, 1);
	lua_unlock(L);
	if(cb.result != 0)
	{
		cb.sze = 0;
		if(cb.ptr)
		{
			free(cb.ptr);
			cb.ptr = 0;
		}
	}
	return cb;
}

int main(int argc, char * argv[])
{
	int res;

	lua_State * L = 0;
	lua_State * M = 0;

	TCompiledBuffer cb1;
	TCompiledBuffer cb3;

	L = lua_open();
	luaL_openlibs(L);

	cb1 = CompileString(L, "print(\"Hello World!\")\n");

	cb3 = CompileString(L, 
"-- create private index""\n"
"local index = {}""\n"
"""\n"
"-- create metatable""\n"
"local mt =""\n"
"{""\n"
"	__index = function (t,k)""\n"
"		print(\"*access to element \\\"\" .. tostring(k) .. \"\\\" in table \\\"\" .. t[index][\"name\"] .. \"\\\"\")""\n"
"		return t[index][k]   -- access the original table""\n"
"	end,""\n"
"""\n"
"	__newindex = function (t,k,v)""\n"
"		print(\"*update of element \\\"\" .. tostring(k) .. \"\\\" to \\\"\" .. tostring(v) .. \"\\\" in table \\\"\" .. t[index][\"name\"] .. \"\\\"\")""\n"
"		t[index][k] = v   -- update original table""\n"
"	end""\n"
"}""\n"
"""\n"
"function track (t)""\n"
"  local proxy = {}""\n"
"  proxy[index] = t""\n"
"  setmetatable(proxy, mt)""\n"
"  return proxy""\n"
"end""\n"
"""\n"
"tbl1 = {}""\n"
"tbl1[\"name\"]=\"tbl1\"""\n"
"tbl1 = track(tbl1)""\n"
"""\n"
"tbl1[\"id\"] = 0""\n"
);

	lua_close(L);
	L = 0;

	M = lua_open();
	luaL_openlibs(M);
	if(cb1.result == 0)
	{
		uLongf destLen;
		Bytef * dest;

		res = luaL_loadbuffer(M, cb1.ptr, cb1.sze, "");
		res = lua_pcall(M, 0, LUA_MULTRET, 0);

		destLen = cb1.sze * 2 < 1024 ? 1024 : cb1.sze * 2;
		dest = (Bytef *)malloc(destLen);
		memset(dest, 0, destLen);
		res = compress2(dest, &destLen, (Bytef *)cb1.ptr, cb1.sze, Z_BEST_COMPRESSION);
		if(res != Z_OK)
		{
			printf("compress2 failed with error: %i\n", res);
		}
		printf("Original (%i):\n", cb1.sze);
		printf("%s\n", DumpToString(cb1.ptr, cb1.sze).c_str());
		printf("Compressed (%i):\n", destLen);
		printf("%s\n", DumpToString(dest, destLen).c_str());
		free(cb1.ptr);
		free(dest);
	}
	else
	{
		printf("There was an error compiling the input string.\n");
	}

	if(cb3.result == 0)
	{
		uLongf destLen;
		Bytef * dest;

		res = luaL_loadbuffer(M, cb3.ptr, cb3.sze, "test");
		res = lua_pcall(M, 0, LUA_MULTRET, 0);

		destLen = cb3.sze * 2 < 1024 ? 1024 : cb3.sze * 2;
		dest = (Bytef *)malloc(destLen);
		memset(dest, 0, destLen);
		res = compress2(dest, &destLen, (Bytef *)cb3.ptr, cb3.sze, Z_BEST_COMPRESSION);
		if(res != Z_OK)
		{
			printf("compress2 failed with error: %i\n", res);
		}
		printf("Original (%i):\n", cb3.sze);
		printf("%s\n", DumpToString(cb3.ptr, cb3.sze).c_str());
		printf("Compressed (%i):\n", destLen);
		printf("%s\n", DumpToString(dest, destLen).c_str());
		free(cb3.ptr);
		free(dest);
	}
	else
	{
		printf("There was an error compiling the input string.\n");
	}
	lua_close(M);
	M = 0;

	return 0;
}

// Formats a stream of data into a hex/ascii dump
std::string DumpToString(void * data_, size_t count)
{
	char msg[256] = {0};
	std::stringstream smsg;
	int sze = count;
	int outputsize = sze;
	if(outputsize % 16 != 0) outputsize += (16 - outputsize % 16);
	if(outputsize == 0) outputsize = 16;
	int ctr = 0;
	char ch[17] = {0};
	int x1 = 0;
	unsigned char * data = reinterpret_cast<unsigned char *>(data_);
	for(int x = 0; x < outputsize; ++x)
	{
		if(x < sze)
		{
			unsigned char b = data[x];
			sprintf_s(msg + ctr, sizeof(msg) - ctr, "%.2X ", b);
			ch[x1] = (isprint(b) && !isspace(b) ? b : '.');
		}
		else
		{
			sprintf_s(msg + ctr, sizeof(msg) - ctr, "   ");
			ch[x1] = '.';
		}
		x1++;
		ctr += 3;
		if((x+1) % 16 == 0)
		{
			x1 = 0;
			sprintf_s(msg + ctr, sizeof(msg) - ctr, "  %s", ch);
			ctr += 18;
			sprintf_s(msg + ctr, sizeof(msg) - ctr, "\n");
			try
			{
				smsg << msg;
			}
			catch (std::exception & e)
			{
				printf("[%s] %s\n", __FUNCTION__, e.what());
				return "";
			}
			ctr = 0;
			memset(msg, 0, sizeof(msg));
		}
	}
	return smsg.str();
}

Example Output:


Hello World!
Original (97):
1B 4C 75 61 51 00 01 04 04 04 08 00 00 00 00 00   .LuaQ...........
00 00 00 00 00 00 00 00 00 00 02 02 04 00 00 00   ................
05 00 00 00 41 40 00 00 1C 40 00 01 1E 00 80 00   ....A@...@......
02 00 00 00 04 06 00 00 00 70 72 69 6E 74 00 04   .........print..
0D 00 00 00 48 65 6C 6C 6F 20 57 6F 72 6C 64 21   ....Hello.World!
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00                                                ................

Compressed (71):
78 DA 93 F6 29 4D 0C 64 60 64 61 61 E1 60 40 05   x...)M.d`daa.`@.
4C 4C 2C 40 92 15 88 1D 1D 18 18 64 1C 18 18 E5   LL,@.......d....
18 1A 18 98 80 7C 16 36 20 51 50 94 99 57 C2 C0   .....|.6.QP..W..
C2 0B 64 7A A4 E6 E4 E4 2B 84 E7 17 E5 A4 28 32   ..dz....+.....(2
A0 03 00 AC 3F 09 B4                              ....?...........

*update of element "id" to "0" in table "tbl1"
Original (727):
1B 4C 75 61 51 00 01 04 04 04 08 00 00 00 00 00   .LuaQ...........
00 00 00 00 00 00 00 00 00 00 02 04 17 00 00 00   ................
0A 00 00 00 4A 80 00 00 A4 00 00 00 00 00 00 00   ....J...........
49 80 00 80 A4 40 00 00 00 00 00 00 49 80 80 80   I....@......I...
A4 80 00 00 00 00 00 00 00 00 80 00 87 80 00 00   ................
8A 00 00 00 87 C0 00 00 85 C0 00 00 89 C0 40 82   ..............@.
85 80 00 00 C5 C0 00 00 9C 80 00 01 87 C0 00 00   ................
85 C0 00 00 89 80 C1 82 1E 00 80 00 07 00 00 00   ................
04 08 00 00 00 5F 5F 69 6E 64 65 78 00 04 0B 00   .....__index....
00 00 5F 5F 6E 65 77 69 6E 64 65 78 00 04 06 00   ..__newindex....
00 00 74 72 61 63 6B 00 04 05 00 00 00 74 62 6C   ..track......tbl
31 00 04 05 00 00 00 6E 61 6D 65 00 04 03 00 00   1......name.....
00 69 64 00 03 00 00 00 00 00 00 00 00 03 00 00   .id.............
00 00 00 00 00 07 00 00 00 0A 00 00 00 01 02 00   ................
08 11 00 00 00 85 00 00 00 C1 40 00 00 05 81 00   ..........@.....
00 40 01 80 00 1C 81 00 01 41 C1 00 00 84 01 00   .@.......A......
00 86 81 01 00 86 01 41 03 C1 41 01 00 D5 C0 81   .......A..A.....
01 9C 40 00 01 84 00 00 00 86 80 00 00 86 40 00   ..@...........@.
01 9E 00 00 01 1E 00 80 00 06 00 00 00 04 06 00   ................
00 00 70 72 69 6E 74 00 04 15 00 00 00 2A 61 63   ..print......*ac
63 65 73 73 20 74 6F 20 65 6C 65 6D 65 6E 74 20   cess.to.element.
22 00 04 09 00 00 00 74 6F 73 74 72 69 6E 67 00   "......tostring.
04 0D 00 00 00 22 20 69 6E 20 74 61 62 6C 65 20   .....".in.table.
22 00 04 05 00 00 00 6E 61 6D 65 00 04 02 00 00   "......name.....
00 22 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ."..............
00 00 00 00 00 00 00 0C 00 00 00 0F 00 00 00 01   ................
03 00 0B 14 00 00 00 C5 00 00 00 01 41 00 00 45   ............A..E
81 00 00 80 01 80 00 5C 81 00 01 81 C1 00 00 C5   .......\........
81 00 00 00 02 00 01 DC 81 00 01 01 02 01 00 44   ...............D
02 00 00 46 42 02 00 46 42 C1 04 81 82 01 00 15   ...FB..FB.......
81 02 02 DC 40 00 01 C4 00 00 00 C6 C0 00 00 C9   ....@...........
80 80 00 1E 00 80 00 07 00 00 00 04 06 00 00 00   ................
70 72 69 6E 74 00 04 15 00 00 00 2A 75 70 64 61   print......*upda
74 65 20 6F 66 20 65 6C 65 6D 65 6E 74 20 22 00   te.of.element.".
04 09 00 00 00 74 6F 73 74 72 69 6E 67 00 04 07   .....tostring...
00 00 00 22 20 74 6F 20 22 00 04 0D 00 00 00 22   ...".to."......"
20 69 6E 20 74 61 62 6C 65 20 22 00 04 05 00 00   .in.table.".....
00 6E 61 6D 65 00 04 02 00 00 00 22 00 00 00 00   .name......"....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00 12 00 00 00 17 00 00 00 02 01 00 05 09 00 00   ................
00 4A 00 00 00 84 00 00 00 49 00 00 01 85 00 00   .J.......I......
00 C0 00 80 00 04 01 80 00 9C 40 80 01 5E 00 00   ..........@..^..
01 1E 00 80 00 01 00 00 00 04 0D 00 00 00 73 65   ..............se
74 6D 65 74 61 74 61 62 6C 65 00 00 00 00 00 00   tmetatable......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00 00 00 00 00 00 00                              ................

Compressed (395):
78 DA 9D 52 41 4A 03 41 10 AC DE 4C 4C 34 8A 28   x..RAJ.A...LL4.(
06 0F 8A 2C 39 7A F3 07 1B 51 41 F1 E2 5D 94 35   ...,9z...QA..].5
19 25 98 6C 82 19 D1 E3 6C 30 11 7D 43 F0 94 87   .%.l....l0.}C...
64 C1 08 3E C0 9B 8F B1 66 4D 14 45 3D D8 50 3B   d..>....fM.E=.P;
D5 33 CD 74 55 CF AE EC 5F 86 07 10 A5 54 1E 5F   .3.tU..._....T._
C3 53 CB FC CE 10 7B 16 18 8C 77 77 2D EC 20 98   .S....{...ww-...
70 6B 07 76 52 6F 71 4B 7E 4F 76 3B 04 BA C4 DD   pk.vRoqK~Ov;....
30 E8 74 B9 37 22 EF 5B C8 C7 BE 4D 3A 6B AC CF   0.t.7".[...M:k..
B1 36 6D 7B 7C 5C 8B AA FA 1A AA 90 26 91 BE 1A   .6m{|\......&...
E7 53 CC CD 45 58 39 87 CA 3A 7A 52 DF 78 67 51   .S..EX9..:zR.xgQ
D8 D0 50 19 B2 5A 15 99 89 84 09 C9 8D 85 8B 87   ..P..Z..........
FC 02 D7 2E 91 50 74 36 06 02 B1 58 8D 21 E5 04   .....Pt6...X.!..
B8 11 A0 17 0B 7A 52 CE 24 65 C1 CB 30 96 7E 00   .....zR.$e..0.~.
B9 61 7D 8F C2 7B E4 0F BC C6 89 75 52 52 3D AD   .a}..{.....uRR=.
8B 5A 64 A0 8A A4 EB 61 A5 A2 DB 6D DF 34 7D 5D   .Zd....a...m.4}]
D7 0D 1D 19 BF 04 35 ED 94 36 DB 86 75 67 50 73   ......5..6..ugPs
CC 4A 7E 2D F2 4D 78 52 D7 EE FC 53 BF E7 CE F0   .J~-.MxR...S....
53 CC 12 F3 CE 41 06 85 25 AE 23 C7 CB C0 36 1D   S....A..%.#...6.
58 3A 38 A4 83 98 0E 46 B1 7B 29 C8 2B 73 F1 04   X:8....F.{).+s..
5B BC 72 67 D3 23 12 15 77 04 C5 D8 F3 5E E9 E2   [.rg.#..w....^..
91 65 4F 1C FD B3 B5 F8 18 FD 77 37 97 AD 6A 68   .eO.......w7..jh
B4 DF 3C FD DD 4D 2E 75 43 BF A5 FF 3A 5B 24 DC   ..<..M.uC...:[$.
8F 45 B1 59 77 F7 1E E1 E6 BD 4B 83 EE 9D 86 54   .E.Yw.....K....T
A7 E8 B0 1F 58 39 1A CF 5E 9C 5A D7 AE AD 4D 43   ....X9..^.Z...MC
9B 30 6D 88 BF E2 0D AC 13 75 FB                  .0m......u......

Press any key to continue . . .

Download: luaRuntimeCompile (complete project) Basically, I want to explore sending logic to a client to execute rather than sticking to the traditional approach of sending data. I want to cut out the client's low level data processing and aim for something higher level now. I don't actually need the ability to compile Lua scripts at run time for that, since sending strings and calling DoString is good enough, but having it makes it all the more practical for a real demo. Just though I'd share in case anyone else had a need like this. Please feel free to leave any questions or comments!

Quote:
Original post by Drew_Benton
Recently I've been interested in the network applications of Lua for prototyping out ideas in a server/client environment. I was thinking something along the lines of custom logic generated on a server being sent to a client for execution. For simple security reasons, it'd be best if this logic were not sent in a text format that could easily be modified by anyone, but rather in a compiled form.

Sending lua bytecode really isn't any more secure than sending source. Lua bytecode retains all variable names (including locals) and even line numbers, so decompilation is trivial. Mind you, I have not heard of any such decompiler, but I would guess that such a thing does exist, and if not, wouldn't take much to write.

Even if you could obfuscate the code in such a way as to prevent decompilation, a properly inclined user could run arbitrary code anyways. I don't know much about security but I think you just have to assume that the user has full control over their machine, and work from there. Besides, it is rather degrading to assume or enforce that the user's machine is not soveriegn. I think the most robust strategy is to do validation and capability checking server-side, which theoretically can be secured.

I hope this helps.

EDIT: Sending bytecode might still be a good idea if you can compress it smaller. But again, don't rely on that for security.

Topic Locked