Python, Pygame, events, and maybe threads?

Open your Python documentation, go to the Global Module Index and look up the 'threading' module. If you've ever used threads before you'll feel right at home. ;)

Beware though that Python threads cannot take advantage of multi core systems because the GIL (Global Interpreter Lock) effectively prevents threads from really running simultaneously.

Quote:

Original post by theOcelot The ideal solution would be to simply turn off the event system. I've already tried pygame.event.set_allowed(None), but it didn't seem to work. Is there some trick that will make it work or is it just a dead end?

It's a dead-end. You need the event system for the GUI to work at all. That's how Pygame collects input from Windows (or KDE, Gnome, whatever you're using) and passes it to the renderer and internal window management. It also means that you're unlikely to see any benefit by changing graphical libraries. (Although I would generally always recommend Pyglet over Pygame.)

Quote:

Is it possible to do threads in regular python? It seems that I could make a thread that does nothing but pump the event loop. But I've read enough here to know that nothing in multi-threading is as easy as it seems.

Googling for 'python threads' would have answered this. :P But yes, you don't want a thread that does nothing but pump the event loop, primarily because the event loop is intrinsic to your main thread. What you end up having to do in this case is synchronise the two so that the main thread tells your 'pumping' thread when it is safe to pump. This means the main thread has had to explicitly call a certain function to do this - so it may as well just have pumped the event queue itself. Thus, this is pointless.

The answer is probably to do the reverse: do your heavy lifting work in the background thread, and have the main thread handle the event queue. But then you have a new problem - if your heavily-worked thread is performing graphical operations, well, they don't usually work from a background thread. So you're back to square one. But that's just how these things go - no matter what you do in concurrent/distributed computing, there are always some resources you can use independently, and some you have to share, and the shared ones (eg. event queues for a process, a graphics card, even a printer or hard disk) can only be used by one process at a time. There are ways of hiding or moving this restriction but ultimately it's still there.

What exactly is your application doing that takes so long that you don't get to call pygame.event.pump() often enough? I'm sure there's a workaround that can be found.

Quote:

Original post by Kylotan What exactly is your application doing that takes so long that you don't get to call pygame.event.pump() often enough? I'm sure there's a workaround that can be found.

It's waiting for user input at the console. That's what I meant by being controlled by an interactive session. I've been using Python as a programmable calculator for my math homework, and I want this to fit in with that.

It just seems that since pygame has a default handler for all the events anyway, I ought to be able to tell it to handle them automatically, without bothering me.

So I guess I'm stuck with threads no matter what? I meant to say in the OP that I have never done anything threaded.

So I guess I'll have the main thread that basically just runs a game loop, reading information about the graphs from the list. The other 'thread' will just be the one that sits at the command line and edits the list as the user desires. Sound like it'll work?

Hello?

How much trouble do I actually have to go to synchronize the list anyway? Could someone point me to some article that will let me get a handle on threading so I can understand the Python docs?

Quote:

Original post by theOcelot Hello? How much trouble do I actually have to go to synchronize the list anyway? Could someone point me to some article that will let me get a handle on threading so I can understand the Python docs?

Let's put it this way. It's enough trouble for people to not be able to instantly provide you with finished solution and tutorial on concurrency in the one hour and ten minutes you're apparently willing to wait.

Quote:

Original post by Antheus Quote: Original post by theOcelot Hello? How much trouble do I actually have to go to synchronize the list anyway? Could someone point me to some article that will let me get a handle on threading so I can understand the Python docs? Let's put it this way. It's enough trouble for people to not be able to instantly provide you with finished solution and tutorial on concurrency in the one hour and ten minutes you're apparently willing to wait.

Actually, I waited twenty-five hours and ten minutes, until my thread had gone onto the second page. Look at the date.

I didn't ask for a finished solution, or even really a tutorial. I just want a link with the basics, which I figured people would probably have lying around to spring on us poor beginners in parallelism.

I know you're used to people asking for ready-made solutions, but that's not what I want, and that's not what I said.

Quote:

Original post by theOcelot Quote: Original post by Kylotan What exactly is your application doing that takes so long that you don't get to call pygame.event.pump() often enough? I'm sure there's a workaround that can be found. It's waiting for user input at the console. That's what I meant by being controlled by an interactive session. I've been using Python as a programmable calculator for my math homework, and I want this to fit in with that.

Ok, there's your first mistake, wanting to mix console sessions with a graphical interface. Generally this is just not attempted because it's awkward mixing a sequential control flow with an iterating event loop at the same time.

Quote:

It just seems that since pygame has a default handler for all the events anyway, I ought to be able to tell it to handle them automatically, without bothering me.

But it won't even get the events unless you explicitly pass them on. That's what pumping or polling the event queue is for.

Quote:

I meant to say in the OP that I have never done anything threaded.

Then I wholeheartedly suggest you give up on that now, and perhaps opt for removing the interactive session, instead handling input as part of your event loop.

Quote:

So I guess I'll have the main thread that basically just runs a game loop, reading information about the graphs from the list. The other 'thread' will just be the one that sits at the command line and edits the list as the user desires. Sound like it'll work?

Not safely, if they're just editing the same list. If you're lucky, you can put a mutex around access to that list to keep things safe.

Quote:

Could someone point me to some article that will let me get a handle on threading so I can understand the Python docs?

Unfortunately not. Proper threading is a complex subject more suited to several weeks of study than to an article somewhere. Diving in without a good understanding of all the theory behind distributed systems is what gets most people into trouble.

Thanks for replying, Kylotan. I had given this thread up for dead.

I figured that I would have to put a mutex or something on the list no matter what. Since the two parts that access the list can't actually run simultaneously (thanks to the GIL), I thought that it would work. Why am I "lucky" if it works? The main loop is only reading the list.

I'm ok with studying for a week or five or more before messing with it. It is, after all, for school [smile]. And I like theory. But an overview would still help in understanding how all the parts fit together as I study each of them. I've been googling for a while and haven't found anything that shows, for instance, exactly the way a lock is actually used (do you subclass from an object to produce a lockable object, do you construct a lock with an object, some combination?).

EDIT: finally found something that starts to explain locks in python. It appears that acquiring a lock (by itself) just prevents switching to another thread until the lock is released. Is that right?

[Edited by - theOcelot on October 20, 2008 8:16:10 PM]

Quote:

Original post by theOcelot I figured that I would have to put a mutex or something on the list no matter what. Since the two parts that access the list can't actually run simultaneously (thanks to the GIL), I thought that it would work. Why am I "lucky" if it works? The main loop is only reading the list.

The GIL does not save you from concurrency problems in Python. It does not make anything safer. All it does is ensure that Python is only executing 1 thread at a time, in exactly the same way that a single-core CPU only executes 1 thread at a time. Both are still perfectly capable of having 2 threads working on the same data and completely mangling what they were trying to achieve. Even reads are not safe if you don't know what you are doing. Imagine Thread One needs to read the last element of the list. First it finds where the end of the list is. And then, before it reads it, there's a context switch, and Thread Two deletes the last element, or adds a new element and moves the whole list in memory, anything like that. A little later, we return to Thread One, who reads what it thinks is the last element from the position it determined earlier, and... well, if you're lucky you get meaningless data back, and if you're unlucky, your program crashes because you read memory you aren't allowed to read. Actually, maybe the unlucky case is luckier because at least you see there's a problem. You could just end up with randomly corrupted data for ages and not see why it's happening.

The problems don't come from things executing at exactly the same time, but from previous assumptions made about shared resources becoming violated by other processes changing those resources. Any program that executes "IF x THEN y" has to make the assumption that x doesn't change by the time they execute y, or that it no longer matters. Sadly, you'll find that it usually does matter because statement y operates on value x. And you'll also find that a hell of a lot of any program in the world can be boiled down to "IF x THEN y".

I don't mean to be patronising, but this is why I say that threading is something you need to study, rather than read a tutorial on. I'm not speaking about you personally, but so many people on GameDev don't fully understand the implications of threading or distributed/concurrent programming and believe that they can just hack their way through it, fixing bugs as they find them, as they would in other areas. But it doesn't work like that. The only way to write concurrent code that works is to prove it correct from the beginning. Otherwise, you're just coding in bugs which you have very little chance of ever reproducing or debugging.

Quote:

EDIT: finally found something that starts to explain locks in python. It appears that acquiring a lock (by itself) just prevents switching to another thread until the lock is released. Is that right?

No. If that was possible, any single program could easily cripple the system just by acquiring a lock and choosing not to release it. Needless to say, that wouldn't be a great feature for an OS to offer you. A lock (aka mutex, binary semaphore), once acquired, prevents any other process from acquiring that same lock. Other threads can do whatever they like as long as they don't want that lock. If they ask for it, their thread will then pause until the lock becomes available.

Locks are the simplest and lowest-level synchronisation primitive available to you. They're also the easiest to misuse, resulting in deadlock, starvation, and other problems relating to safety and liveness properties. So the clever programmer would be better advised to use a safer system. If you are just passing data from one thread to another, you have a classic "producer-consumer" pattern, and in Python the best way to write that if using multiple threads is to use the Queue class. One thread can put() into it, the other can get() out of it, with no need for you to do your own locking or anything.

Suffice it to say, to the degree I thought you sounded patronizing, I also thought I deserved it. No problem there.

As for the other stuff, I started to get it after I edited the post. It's slowly coming together. Thanks for putting it all in the open, though. You confirmed some things I was starting to think, and corrected some others.

I'll look at the Queue thing. It might be a little tricky to do, since the information passed would have to be in the form of events that tell it to change it's state.

I think I've almost got it enough that I can study it effectively.