stackless python not utilizing multicore?

Run multiple processes and pass data between them via sockets (Twisted is perfect).

The nice thing about stackless is that it's inherently concurrent, whether you communicate via local heap memory, via sockets or via internet, it makes no difference.

Quote:

I have to schedule() often then the overall system load increases.

This type of concurrency is well suited for coarse-grained tasks. So getHealth(cb), getMana(cb), getName(cb) isn't the best approach. Instead, try to perform larger tasks - handleDamage(cb), and in cb you perform all calculations (health, mana, name, lookup tables, etc.)

Quote:

Why does it not execute independent tasks in parallel?

But it does. In an incredibly efficient and light-weight manner. It's just cooperative multi-threading.

only one tasklet is scheduled at a time.?!

What does "cooperative Multithreading" mean?

Quote:

Original post by hydroo only one tasklet is scheduled at a time.?!

Only one is being executed at a time. All are scheduled whenever they have incoming messages.

Quote:

What does "cooperative Multithreading" mean?

It means that rather than relying on OS to interrupt and schedule threads on its own, programmer does so. The benefit is that tasks get scheduled at well defined and logically sane points, the downside is that if a task takes too long, it can hog the system.

The pyProcessing module is another one to look at. I don't know if it works with stackless (or indeed if any regular python modules work with stackless), but it's been accepted in to the Python standard libraries.

Python and stackless are multithread capable. Having said that they aren't very good at it. Python and therefore stackless has a GIL(global interpreter lock) so only one thread is allowed in the interpreter at any one time. Typically the way to get multi threaded performance is have a thread call out to a C function that releases the GIL then does stuff and reacquires the GIL before returning to python. Also the micro threads in stackless are "green" threads. They all execute on the same os level thread, that is why you can have so many of them with out choking.

Quote:

Original post by stonemetal Python and stackless are multithread capable. Having said that they aren't very good at it. Python and therefore stackless has a GIL(global interpreter lock) so only one thread is allowed in the interpreter at any one time. Typically the way to get multi threaded performance is have a thread call out to a C function that releases the GIL then does stuff and reacquires the GIL before returning to python. Also the micro threads in stackless are "green" threads. They all execute on the same os level thread, that is why you can have so many of them with out choking.

The GIL in python is one of my biggest gripes with the design of it. Tcl, for instance, has a GIL per interpreter, which is fine. Python, however, has a GIL per process. Which prevents you from utilizing multiple interpreters per thread from running concurrently, even if all of the information they'll use is completely independant of the other. *sigh*.

Of course, you could patch this in a custom python implementation, but they've basically stated they aren't going to fix it.

This is from memory only - no guarantees that this is correct in the details.

when i was researching a way of building a lightweight framework for message passing i was suprised that none of the more high-level language approaches offered a seemless method to take advantage of multi-core hardware. i looked at lua (great coroutine support but still single threaded), stackless python (has the GIL - so you need IPC or sockets [edit] and process seperation), ocaml (Gc doesnt play nicely in concurrent environment) despite having a couple of continuation libs offerings that could be cool. javascript (spidermonkey?) i think is basically single threaded.

the only stuff i am aware of is c++ boost::asio which has a design allowing for mixing of preemptive multithreading and cooperative multitasking (event driven - but you can layer over the top with coroutines) and erlang which scales nicely with hardware but suffers a bit as a general programming environment compared with newer fp style langugaes (ie weak types compared to ML derived langugages offering much richer expressiveness).

Quote:

Original post by Washu The GIL in python is one of my biggest gripes with the design of it. Tcl, for instance, has a GIL per interpreter, which is fine. Python, however, has a GIL per process. Which prevents you from utilizing multiple interpreters per thread from running concurrently, even if all of the information they'll use is completely independant of the other. *sigh*. Of course, you could patch this in a custom python implementation, but they've basically stated they aren't going to fix it.

Have you seen pyProcessing!? The benchmarks in the PEP are particularly interesting.

Quote:

Original post by the_edd Quote: Original post by Washu The GIL in python is one of my biggest gripes with the design of it. Tcl, for instance, has a GIL per interpreter, which is fine. Python, however, has a GIL per process. Which prevents you from utilizing multiple interpreters per thread from running concurrently, even if all of the information they'll use is completely independant of the other. *sigh*. Of course, you could patch this in a custom python implementation, but they've basically stated they aren't going to fix it. Have you seen pyProcessing!? The benchmarks in the PEP are particularly interesting.

Yes, but that's just it: They are PROCESSES, not threads. Which makes it particularily hard when you're dealing with C interop between say a low level engine and high level python scripts.

More reason to extend Python rather than embed it. ;)

Quote:

Original post by Washu Quote: Original post by the_edd Have you seen pyProcessing!? The benchmarks in the PEP are particularly interesting. Yes, but that's just it: They are PROCESSES, not threads. Which makes it particularily hard when you're dealing with C interop between say a low level engine and high level python scripts.

Maybe so (I don't know if this an issue for the OP), but I'm sure you'll agree that there are some very big advantages to this approach too.

Quote:

Original post by the_edd Maybe so (I don't know if this an issue for the OP), but I'm sure you'll agree that there are some very big advantages to this approach too.

Not really, just a lot of disadvantages mostly. Processes switch slower than threads do, you have to deal with the kernel to perform any inter-process communication, including between the two python modules, synchronizing information across your C modules is difficult at best, and as someone above suggested, is easier to accomplish if you extend python rather than embed it.

In fact, I can't really think of a single advantage that kind of processing has that isn't something of a generic advantage that all methodologies of performing something similar has (aka, something agnostic to python completely).

didnt see that there are this many responses.

I am currently moving, so I dont have the time to dig some more.

What I was trying to do was a basic architecture which enables using multiple machines/clusters + multiple cores per node (on a cluster).
Kind of threading + MPI + sockets ... I am not sure about details.
( I could leave out the threading part yes, but why not reaping the benefits of smp)
I am just gathering information.

Quote:

Original post by hydroo What I was trying to do was a basic architecture which enables using multiple machines/clusters + multiple cores per node (on a cluster). Kind of threading + MPI + sockets ... I am not sure about details. ( I could leave out the threading part yes, but why not reaping the benefits of smp)

If you're already using distributed processes, use processes, not threads. In some scenarios, threads may get you slightly better performance, but the programming model for then is necessarily more error-prone, doubly so when you know you already have a perfectly working multi-process model which you could use anyway.

You are right in a away.

But I don't want such a "half-assed" solution, and the performance gain is possibly huge, depending on the situation.

So I guess I've gotta go with python thread/threading or threadpool.

Quote:

Original post by Washu Quote: Original post by the_edd Maybe so (I don't know if this an issue for the OP), but I'm sure you'll agree that there are some very big advantages to this approach too. Not really, just a lot of disadvantages mostly. Processes switch slower than threads do, you have to deal with the kernel to perform any inter-process communication, including between the two python modules, synchronizing information across your C modules is difficult at best, and as someone above suggested, is easier to accomplish if you extend python rather than embed it. In fact, I can't really think of a single advantage that kind of processing has that isn't something of a generic advantage that all methodologies of performing something similar has (aka, something agnostic to python completely).

I think you're making many assumptions about the application. I don't know what the OP has in mind but for some kinds of applications:

- shared nothing can make "parallel programming" much easier
- being able to spread computations across multiple machines *and* multiple cores
- on some systems (e.g. Linux) a thread is damn well near the same thing as a process

Quote:

Original post by the_edd - on some systems (e.g. Linux) a thread is damn well near the same thing as a process

Performance wise this depends on how data transfer is achieved. afaik MPI (lam/mpich) use sockets. It ofcourse uses not shared mem. (I Don't know about the influence of OS on this, though).
But one could simply benchmark it. I will. Sometime sooner.

Quote:

Original post by the_edd - on some systems (e.g. Linux) a thread is damn well near the same thing as a process

Umm, nope. In fact, this is why things like Apache have moved away from "fork", because processes AREN'T at all like threads. They have a great deal more overhead than a thread, switching between them requires a ton of work for the kernel, and communication between them requires using either fairly limited resources (such as shared memory, which requires the kernel to allocate space in its tables to reference said shared memory so that both processes can actually see it), or using communication methods that require you to talk to the kernel, such as sockets.

Meanwhile, two threads of the same process run in the same address space as each other, have access to each others variables, and can communicate far more quickly. Switching between them is also quite a bit less challenging.

Quote:

Original post by Washu Umm, nope. In fact, this is why things like Apache have moved away from "fork", because processes AREN'T at all like threads.

My impression was that they moved because they were using thread-per-socket or process-per-socket to favor task-based concurrency (reactor, pro-actor style).

Running one process per core with IPC shouldn't incur any drastic overhead compared to having one thread per core within single process.

Spawning either process or thread is indeed incredibly expensive.

Quote:

Meanwhile, two threads of the same process run in the same address space as each other, have access to each others variables, and can communicate far more quickly. Switching between them is also quite a bit less challenging.

A lot of benefit with regards to concurrency can be had from reducing shared state as much as possible.

The benchmarks on IPC (windows) showed the access rates that are effectively equal to memory bandwidth. I also don't remember any notable cost using SHM under Linux.