Skip to main content
GameDev.net gamedev.net
🔒 Locked

Run-time compiled C in python scripts. Works.

Started by smr Dec 1, 2009 at 1:33 PM 19 replies 4.6k views
Original Post
smr
smr
I've written a module that uses ctypes and Tiny C to compile C source code from within a python program and makes it callable within python. Is this something the community would be interested in having? If so I can put up a page and post the code on my gamedev.net space for all to have. The application I'm currently working on has some performance critical sections where python just wasn't cutting it. It's easy to extend python with C using ctypes, but it's not very convenient when you only need to write very small amounts of performance-critical code in C. I found a few libraries that would suit this purpose one way or another, but I felt that they'd be too cumbersome for what I am doing. I did run across a very old, seemingly abandoned project called "C in Python" that used the Tiny C compiler and ctypes to compile C code from strings at run-time. It didn't work, but I thought it was a great idea, so I've rewritten it so that it works with the latest TCC and python 2.6. For now I've only tested it on linux 32 bit. It would need some modifications to work on Windows, but it shouldn't be difficult to do. It would probably work on Mac OS X also, but I couldn't get Tiny C to build on my mac. Here's a working example:

import ctypes
import compiler

# Defining functions that will be exported to python from the C source.
# compile expects a list of two-tuples containing first the name
# of the exported function in the C source code and the ctypes prototype
exports = (('fib', ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)),
           ('test_callback', ctypes.CFUNCTYPE(None)))

# defining a function to call back into from the C code
def callback():
    print "Calling back into python WORKS!!!!"
    
# You can optionally make your python code available to the C program.
# compile expects a list of three-tuples containg first the name
# to use in the C source, the ctypes prototype, and the actual
# python function object to be called.
imports = (('callback', ctypes.CFUNCTYPE(None), callback),)

# The C source code
code = """
int fib(int n)
{
    if (n <= 2)
        return 1;
    else
        return fib(n-1) + fib(n-2);
}

void test_callback() {
    callback();
}
"""

# compile returns a dictionary of callables. imports is optional.
lib = compiler.compile(exports, code, imports)

print "c fib result:", lib['fib'](30)
print "should see the same text repeated twice:"
callback()
lib['test_callback']()
Tiny C also supports importing functions from a dynamic link library so I might implement that also, but I haven't needed it yet. [Edited by - smr on December 1, 2009 2:40:26 PM]
ibebrett
ibebrett
that looks awesome
smr
smr
Thanks!
mrbastard
mrbastard
Cool stuff! I'd love to see more.
[size="1"]
smr
smr
I think I'll make this available once I get it working under Windows and Mac OS X. I'm also considering automatically grabbing the exported functions by analyzing the C source code. More to come later.
swiftcoder
swiftcoder
Quote:
Original post by smr
I'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]
smr
smr
Quote:
Original post by swiftcoder
Quote:
Original post by smr
I'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?


It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
swiftcoder
swiftcoder
Quote:
Original post by smr
Quote:
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.
Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]
smr
smr
Quote:
Original post by swiftcoder
Quote:
Original post by smr
Quote:
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.


Aww, shucks... I guess I was so excited about the idea of this that I hadn't thought that part through. I have to give the tiny c library a place to put the code after it's compiled, which I'm doing now by allocating an array in python of the correct size then passing tiny c the address of its buffer. I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.
swiftcoder
swiftcoder
Quote:
Original post by smr
I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.
In theory at least, you can configure it so that the mmap'ed file shouldn't ever touch the physical disk, so the performance difference should be very much. Plus, you get ctypes to do the legwork of symbol table passing.

I would love to take a look when you have a working implementation - might fit right in with my planet renderer...
Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]
smr
smr
So I've coded up a version that I *think* will work on a linux system with NX. Unfortunately I don't have one of these so I was wondering if someone with a linux machine with NX would test for me?
Vectorian
Vectorian
On Windows you should use VirtualAlloc and VirtualProtect to make sure the memory you allocate is executable (msdn)
smr
smr
Quote:
Original post by Vectorian
On Windows you should use VirtualAlloc and VirtualProtect to make sure the memory you allocate is executable (msdn)


Already working on it ;)
smr
smr
Quote:
Original post by smr
Quote:
Original post by Vectorian
On Windows you should use VirtualAlloc and VirtualProtect to make sure the memory you allocate is executable (msdn)


Already working on it ;)


OK, I've got it working under windows. I need to clean it up a bit before I can release it. I think I am setting up the execute access properly, but I don't have a machine with NX, so I can't be sure.
swiftcoder
swiftcoder
Quote:
Original post by smr
I need to clean it up a bit before I can release it. I think I am setting up the execute access properly, but I don't have a machine with NX, so I can't be sure.
I have access to a couple of Red Hat boxes running SELinux, etc. if that is any help. Want me to give it a shot?
Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]
smr
smr
I'll release what I have as soon as I can get the setup script to work properly on linux and windows. This is my first go at creating a setup script. The package will build and everything works on Windows, but on linux I'm having some problems getting the libraries to go in the right places.
smr
smr
I've uploaded the python module, dynamic libraries for Windows and Linux and example program to my gdnet page. This isn't an installable setup package. I couldn't get that to work on Linux. The setup script on linux copies the libraries to my package's dist-packages folder, but I can't seem to figure out a portable way to get the path to these libraries unless I hard code them. I'll work on this later.

For now, you can copy the tcc.py module and the lib folder into your application's directory and everything should work on Linux and Windows. This does not work on Mac OS X. It shouldn't be difficult to get working for osx, I just couldn't get Tiny C to build on my Mac.

This module should work on systems using NX protection. I can't test it myself because I don't have a system with NX. Would someone with NX mind testing this for me and letting me know whether it works or not? To test, first try running example.py without any modifications. If that works, set tcc.TCC_TEST_NX = True directly after importing the tcc module. If you receive a page fault with TCC_TEST_NX set true but no page fault when reset False, then then everything is working the way it should.
wodinoneeye
wodinoneeye


My question would be :

What use would this be put to -- on-the-fly C lisp-like Lambda functions ???

It would have to be some kind of dynamicly defined heavy number-crunching (code reused alot to offset the overhead of the initial code-building and compilation stage, over a less efficient code interpretor (possibly a data option control or byte code system working for a limited parameterizing scheme).


If its something that changes constantly (code being rebuilt frequently) the overhead might make the non-native compilation more efficient.

Since the code is building C code it has to have finite/limited logic to do that operation (unless the code is being made from use input fragments, ie- equations) and even that could be a wizard that build bytecode or activated precanned flavors in static code. Again unless if is build once and run ALOT, so to amortize the problem->C code->native code conversion overhead.


--------------------------------------------[size="1"]Ratings are Opinion, not Fact
smr
smr
Quote:
Original post by wodinoneeye


My question would be :

What use would this be put to -- on-the-fly C lisp-like Lambda functions ???

It would have to be some kind of dynamicly defined heavy number-crunching (code reused alot to offset the overhead of the initial code-building and compilation stage, over a less efficient code interpretor (possibly a data option control or byte code system working for a limited parameterizing scheme).


If its something that changes constantly (code being rebuilt frequently) the overhead might make the non-native compilation more efficient.

Since the code is building C code it has to have finite/limited logic to do that operation (unless the code is being made from use input fragments, ie- equations) and even that could be a wizard that build bytecode or activated precanned flavors in static code. Again unless if is build once and run ALOT, so to amortize the problem->C code->native code conversion overhead.


The C code is intended to be only built once and the Tiny C compiler is very fast. Have a look at the TCC website to see some metrics. And you are correct, there is quite a bit of overhead when marshaling across the C / python border, so you should always profile before you decide to convert some algorithm from python to C. And when you do replace some python code with C, you should think hard about how you'll marshal the data and how often. In many circumstances it will be more efficient to wad up a big chunk of data to be processed with C code once rather than iterating over it in python, crossing the python-C barrier for each item.

Though building this module has been a fun experience, I'll admit that I'm not entirely sure how useful it will be. There's no debug support for the C code, so for anything non-trivial you'll probably have to code and debug with standard C tools anyway then copy-paste the code into python. Since you've already gone through all of that trouble, you might as well just compile your code as a dynamic library and load it with ctypes. Any other compiler will probably produce better code than TCC because TCC only performs a handful of optimizations, trading fast code for fast compiling.
swiftcoder
swiftcoder
Quote:
Original post by wodinoneeye
What use would this be put to -- on-the-fly C lisp-like Lambda functions ???
Assembling composite functions at runtime: i.e. function composition as supported by lisp/python, but in C.

For example, I have a set of procedural generation algorithms, which can be composed at runtime. At the moment, these are implemented as a large set of C functions in a DLL, each of which takes a function pointer and a void argument pointer to chain with the previous function. These are then loaded using ctypes, and some clever (and blatantly unsafe) ctypes tricks are used to chain the C functions together.

With smr's on-the-fly compiler, I can just splice the actual source of the functions together (as I already do for GLSL), and compile a single large function which performs the whole generation step. For the general case, I have a feeling that the removal of all the function/void pointers and related hacks will offset the performance loss of using TinyC, and the maintenance will be much reduced.
Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.