Twisted and foreign event loops
There's a new reactor in Twisted svn: threadedselectreactor. This reactor blocks on select() in a worker thread, so that it is easy to integrate with a foreign event loop without having to completely separate your Twisted code into its own thread. This is awesome, and mostly eliminates the need to have specific reactors for each useful foreign event loop (cfreactor, wxreactor, etc.). Especially when the existing integration reactor sucks -- wxreactor is more or less a worst-possible implementation, and the others aren't a whole lot better.
In order to integrate with foreign event loops, threadedselectreactor sports an extension to the reactor interface: interleave(waker). waker is a callable that takes a callable as an argument. Its job is to call this callable from the thread of the main event loop. So, whatever you're integrating with needs to be able to send messages from an arbitrary thread to the main thread. All of the event loops worth the fairy dust their bits are encoded on are designed to do this easily.
Other than that, the only other thing that needs to be considered is shutdown (if it matters to shut down cleanly). To do that, you can simply replace your "quit" function with an implementation that:
- Sets up an "after shutdown" reactor system event that will call "quitRightNow" when the reactor is dead
- Calls reactor.stop() to commence the self destruct sequence
wxPython already has wxCallAfter that works perfectly well as the waker argument for interleave, so you can Twistify a wxPython application in about six lines. Here's a full demo: doc/examples/core/threadedselect/wxdemo.py.
PyObjC has a PyObjCTools.AppHelper.callAfter that also serves as a too-trivial-to-be-true waker (coincidence? no -- I just added it). You can check out the demo from: doc/examples/core/threadedselect/Cocoa/SimpleWebClient/. The examples in PyObjC's Examples/Twisted/ have also been refactored to use this (instead of cfreactor, which should be considered dead weight).
pygame is more low-level and doesn't provide any conveniences beyond getting and posting events. So your waker should be some function that posts a USEREVENT to pygame, which you need to pick up and do something with. A minimal example of this is here: doc/examples/core/threadedselect/pygamedemo.py.
This reactor should trivially integrate with just about any event loop, including PyQt, GTK, etc. However, the margins of my blog aren't big enough to fit them all.
Besides dropping the threadedselectreactor into twisted.internet is there anything else that is needed to get this reactor working? I ask because I am not building from twisted source but desperately need this reactor. When I replace wxreactor with threadedselectreactor and add the interleave call to my OnInit I get exploding bits all over the place. Basically I can figure out where my first reactor call is by looking at the traceback.
[traceback from my first call to reactor.callLater... plus the following:]
threadedselectreactor.py, line 71 in callLater
self.wakeUp()
threadedselectreactor.py, line 67 in wakeUp
self.waker.wakeUp()
AttributeError: ‘NoneType’ object has no attribute ‘wakeUp’
wxPython 2.5.5.1
twisted 2.0.0 rc2 (win32 installer) [with threadedselectreactor.py in twisted.internet]
Comment by Jim McCoy — 2005-04-18 @ 4:26 pm
Use twisted from svn trunk, try it again. If it’s still broken, let me know. I really don’t care to support monkeypatched installs.
Comment by Bob Ippolito — 2005-04-18 @ 5:00 pm
Non-monkeypatched install, still happens.
win32, twisted trunk build using mingw compiler (r13605)
Reactor calls are made in main(), before MainLoop() is called. Same ‘NoneType’ object has no attribute ‘wakeUp’ error.
Comment by Jim McCoy — 2005-04-18 @ 5:28 pm
Sorry, can’t reproduce, and can’t imagine why this is happening. I don’t use win32. You’ll have to figure out why waker is not getting set.
Comment by Bob Ippolito — 2005-04-18 @ 5:47 pm
Okay, we will try to figure things out on this end. Thanks for the new reactor though, I have high hopes that we will eventaully be able to put it to good use.
Comment by Jim McCoy — 2005-04-19 @ 2:40 am
Thanks! I got this working with wxPython modal dialog boxes immediately.. Just what I needed!
I plan to use it a few other places too. very sweet.
Comment by drew — 2005-05-02 @ 9:43 pm
Hrm, I downloaded the whole twisted tree from svn yesterday, got that going, and am running the demo successfully, except that perhaps 1 time out of 15, when I click “Exit”, the window and the rest of the event loop freeze. I’m not new to Twisted, but I am to wx — can somebody point me to where should I start looking for the problem? I’m running wxPython 2.4.1 on Debian Sid.
Comment by Steve Freitas — 2005-08-09 @ 7:34 pm
Okay, I gave in and waded into the scary expanses of Debian experimental, and from a quick test I can say that the upgrade to wxPython 2.6.1.1 seems to have solved the issue. Alas, poor wx2.4, I knew thee well…
Comment by Steve Freitas — 2005-08-09 @ 7:49 pm
Bob, could you be so kind as to provide a little overview of how the reactor interacts with wx? I’m a bit confused.
I modified wxdemo.py’s OnInit() to have two code paths: a) To display a modal dialog, then do a reactor.connectTCP() and do some deferreds against the ClientFactory, or b) To skip the dialog, but otherwise do the same as a). Now, I called reactor.interleave() at the very top of OnInit(), before I did anything else. The problem which resulted was this: Code path b) simply wouldn’t run unless I either suffixed it with reactor.run(), or put a dummy wx dialog in. So now I’m getting the picture that the reactor doesn’t start up unless, at some point in OnInit(), you hand control to wx, or you fire it up by hand. So I’m not sure how I should be structuring this, and how this reactor interacts with wx.
I’m hoping you can shed a little light on the way I ought to be doing this. :-) Thanks!
Comment by Steve Freitas — 2005-08-10 @ 6:06 am
Post runnable code, or else nobody will care enough to try it and see what’s wrong.
Comment by Bob Ippolito — 2005-08-10 @ 7:02 am
Apologies if there’s some blog etiquette I don’t know about banning posting of longish code…
The problem I’m having can be illustrated by the following example. Run it, and after clicking “OK” it should tell you that it attempted to connect to a server. (There isn’t one, so the attempt fails, but that doesn’t matter — the point is that it actually tried.) Now, on line 9, change CODEPATH to False, then run it again. The only change is in OnItit() — it doesn’t make a dialog first. Because of that, it doesn’t attempt to connect. That is, it doesn’t unless you uncomment the “reactor.run()” line. So I’m obviously doing something really wrong with the reactor here, but I’m not sure what. Any help is appreciated. Thanks!
from twisted.internet import threadedselectreactor # Requires Twisted from SVN
threadedselectreactor.install()
from twisted.cred import credentials
from twisted.internet import defer, error, reactor
from twisted.python import failure
from twisted.spread import pb
import wx
CODEPATH = True # Change to False to exercise other test path
class wxClientApp(wx.App):
def OnInit(self):
if CODEPATH:
# Make a modal dialog first
TestDialog(None, -1, ‘Click OK to start test’).ShowModal()
self.makeConnection() # The connection is attemped and should fail
else:
# Note the only difference here is we don’t make a modal dialog first
self.makeConnection() # The connection ISN’T attempted…
#reactor.run() # …unless we uncomment this line.
return True
def makeConnection(self):
factory = pb.PBClientFactory()
reactor.connectTCP(’localhost’, 57283, factory)
def1 = factory.login(credentials.UsernamePassword(’client1′, ‘123′),
client=pb.Referenceable()).addErrback(self.error)
reactor.interleave(wx.CallAfter)
def error(self, reason):
if isinstance(reason, failure.Failure):
print “Connection was attempted.”
else:
print reason
# Unless I did something silly, the following code can be ignored.
ID_TEXT = 10000
class TestDialog(wx.Dialog):
def __init__(self, parent, id, msg,
pos = wx.DefaultPosition, size = wx.DefaultSize,
style = wx.DEFAULT_DIALOG_STYLE):
wx.Dialog.__init__(self, parent, id, ”, pos, size, style)
TestDialogFunc(self, msg, True, True)
def TestDialogFunc(parent, msg, call_fit = True, set_sizer = True):
item0 = wx.BoxSizer(wx.VERTICAL)
item1 = wx.StaticText(parent, ID_TEXT, msg, wx.DefaultPosition, wx.DefaultSize, 0)
item0.Add(item1, 0, wx.ALIGN_CENTER|wx.ALL, 5)
item2 = wx.Button(parent, wx.ID_OK, “OK”, wx.DefaultPosition, wx.DefaultSize, 0)
item0.Add(item2, 0, wx.ALIGN_CENTER|wx.ALL, 5)
if set_sizer == True:
parent.SetSizer(item0)
if call_fit == True:
item0.SetSizeHints(parent)
return item0
app = wxClientApp(False)
app.MainLoop()
Comment by Steve Freitas — 2005-08-10 @ 2:20 pm
Hrm, code doesn’t indent well here. Bob, feel free to trim the mess. I’ve posted a workable copy of the code here.
Comment by Steve Freitas — 2005-08-10 @ 2:24 pm
from twisted.internet import wxreactor
wxreactor.install()
app = wx.App()
reactor.registerWxApp(app)
reactor.run()
Comment by Andrei — 2006-06-14 @ 9:59 am
Thanks for the tip on the new way of integrating Twisted and wxPython, Andrei.
For anybody else who is interested, the following were enlightening:
http://twistedmatrix.com/trac/ticket/1235
http://twistedmatrix.com/documents/current/api/twisted.internet.wxreactor.html
Comment by Steve Freitas — 2006-06-29 @ 5:12 pm