[nycbug-talk] twisted python resources

[Isaac Levy]

Yo Bob, Michael, Pete, all,

On Jan 29, 2005, at 9:47 AM, Bob Ippolito wrote:

> On Jan 28, 2005, at 20:10, <lists at genoverly.net> wrote:
>
>> On Fri, 28 Jan 2005 19:27:48 -0500
>> Pete Wright <pete at finn.nomadlogic.org> wrote:
>>
>>> so i'm looking to write a "simple" messaging client
>>> for my network in python.  it's more of an exercise to
>>> get me up to speed on programming in python,

Ooh- yummy...  Can we beta test it sometime for nycbug chat? :)  (I 
don't care if it's a learning app, I just think it's cool to chat 
nycbug using something written by one of us in exile...)

>>
>> I am no guru, I just like to tinker.  But, I went through a very 
>> similar
>> exercise two summers ago.  You are in for a fantastic ride!
>>
>> I wanted to write a private network chat program for day traders to 
>> swap
>> information.  It actually leaned more toward a sqwack box IM app.  I
>> had a server, two flavors of client (desk mgr, trader), and an admin
>> client.   It was backed by a MySQL database for authentication, users
>> preferences, and all the messeages that went flying around. There was 
>> a
>> LOT more to it, but you get the idea.

Yummy.

>>
>> I looked at twisted but found it to be overly complex; and a whole 
>> layer
>> removed from the programming language.  As my goal was to learn
>> asynchronous socket programming; specifically python sockets, I went 
>> to
>> the bottom rung and built my way up.  Here's some of the imports from
>> the server app.
>>
>> import os
>> import sys
>> import asyncore
>> import asynchat
>> import socket
>> import threading
>> import signal
>> import marshal

Well, I wanted to throw in .02? on this- I agree with both Michael and 
Bob here, for different reasons-

I've not truly made the time, or had the project, to dive seriously 
into Twisted- and casually browsing it, I'll agree it's daunting at 
first glance, (even though I work regularly in Python).

I agree with Michael, insomuch as I have experienced most tasks can be 
accomplished very completely and elegantly using just the base python 
libs.  The core of the language is delightfully complete, and the core 
libraries are designed to be a very comprehensive set of tools for 
common, and uncommon tasks.

However, on the other side, experience tells me Bob is right too, some 
tasks (mostly tasks involving heterogeneous environments [www anything, 
basically]) can inadvertently drop you deep into a DEEP rabbit-hole- 
network sockets totally fitting that bill.


> The difference between Twisted and "low level" attempts at socket 
> programming (either with asyncore/asynchat or directly with 
> select/poll) is that Twisted accounts for lots of platform 
> differences, and has years of notches on its belt.  Doing it on your 
> own might appear to work, but it won't be as scalable, might have some 
> really hard to track down bugs related to your misunderstanding of the 
> dark secrets that the BSD sockets API is hiding from you, and it won't 
> be easy to integrate with other services.
>
> I felt the same way as you for my first asynchronous socket 
> application (4 or 5 years ago, when Twisted was in a much poorer state 
> too),

Yay- Go Swocket!  :)

> but after I had learned first-hand what the 'hard' problems were I 
> realized that Twisted already had solutions for them, plus benefits 
> such as existing implementations of various protocols and a nice 
> thread pool implementation.
>
> I also see that you're importing the marshal module... which, if used 
> over the wire, is terribly terribly insecure.


> You can crash an app with a bad marshal,

Yes.  Experienced this once.  Not fun, at all.


> and marshals are specific to a particular Python API version.  If you 
> have a client running Python 2.3 and a server running Python 2.4, 
> marshal will probably not work.

Didn't know that.  Run like hell, at least to Python Pickles IMO.


> It will definitely not work with alternative Python implementations 
> such as Jython.  If it is used for persistence, you may not be able to 
> read your persistent datastore if you upgrade Python (though in 
> practice you probably will, Python makes no such guarantee).  Twisted 
> solves this problem as it has its own safe serialization mechanism 
> (jelly) used by a nice RPC API called Perspective Broker.  If you are 
> using marshal for just persistence, use pickle instead, it is not 
> tightly bound to the Python implementation.

Going to have to second this one, but without direct experiences here- 
just an academic understanding of the implementation.  I saw a lecture 
on Perspecive Broker by it's author at PyCon a while back, was 
seriously impressive.  He'd elegantly solved a lot of mechanical 
problems I had experienced using xmlrpc/soap type stuff long ago, I was 
very impressed and have been looking for an excuse to dive into some 
PB/jelly ever since.


> Also, if I were designing such an app, I would probably use SQLite or 
> PostgreSQL rather than MySQL.  At the low-mid range, SQLite 3 is 
> godawful fast and makes backup and migration really really easy.  
> Going up a few notches, PostgreSQL is more powerful than MySQL in that 
> it supports more stuff useful for development, has less stupid 
> "gotchas", and a much better license.  Also, in the case of 
> PostgreSQL, the Python DB-API adapters available are a whole hell of a 
> lot better written, and there is even a pure python implementation 
> available <http://barryp.org/software/bpgsql>.

 From my Zope experiences, (where various SQL DB's are regularly tied to 
the backend), I feel it's important to second this sentiment:
The Python MySQL scene is in my opinion, simply very messy.  Tons of 
rough edges to cut your hands open on.  (I found myself hacking C to 
fix the PyMsql module for a particular system just weeks ago for work- 
and then there's all these weird transactional annoyances... not really 
a fun path.)  I guess because MySQL is so stable and popular, the 
python libs are suffering from a bit of 'everyone knows how to deal 
with the rough edges' syndrome or something...

PostgreSQL is indeed much better supported, I say that because the 
adapters I've used just transparently work as expected, no complaints 
yet, but I don't know squat about the SQLite side.


> The best way to learn Twisted programming these days seems to be the 
> step by step tutorial of writing a Finger server taking advantage of 
> some of the various features Twisted has to offer 
> <http://twistedmatrix.com/documents/current/howto/tutorial/index>.  
> Other than that, #twisted on IRC is usually helpful, as-is the 
> <twisted-python at twistedmatrix.com> list.  Besides that, I have 
> significant (years) of experience with Twisted and I am a contributor 
> (though I don't have much time to hack on Twisted these days), so I 
> can provide some help on or off this list.
>
> It may be particularly useful if you kept a log of your learning 
> experience so that you can clean it up and write an article.  The 
> article would not only help other sysadmin/developer types looking to 
> write simple applications to better their network, but it will also 
> help the Twisted folks patch a few holes in the learning curve.
>
> -bob

Dude- I'll totally second that learning-Log idea.  Dunno if it's 
appropriate for nyc*bug site, but I bet nobody would dissuade you from 
posting something about your experiences there :) - weather or not the 
app succeeds for you...

Rocket-
.ike