[nycbug-talk] twisted python resources

[Bob Ippolito]

On Jan 28, 2005, at 20:10, <lists at genoverly.net> wrote:

> On Fri, 28 Jan 2005 19:27:48 -0500
> Pete Wright <pete at finn.nomadlogic.org> wrote:
>
>> so i'm looking to write a "simple" messaging client
>> for my network in python.  it's more of an exercise to
>> get me up to speed on programming in python,
>
> I am no guru, I just like to tinker.  But, I went through a very 
> similar
> exercise two summers ago.  You are in for a fantastic ride!
>
> I wanted to write a private network chat program for day traders to 
> swap
> information.  It actually leaned more toward a sqwack box IM app.  I
> had a server, two flavors of client (desk mgr, trader), and an admin
> client.   It was backed by a MySQL database for authentication, users
> preferences, and all the messeages that went flying around. There was a
> LOT more to it, but you get the idea.
>
> I looked at twisted but found it to be overly complex; and a whole 
> layer
> removed from the programming language.  As my goal was to learn
> asynchronous socket programming; specifically python sockets, I went to
> the bottom rung and built my way up.  Here's some of the imports from
> the server app.
>
> import os
> import sys
> import asyncore
> import asynchat
> import socket
> import threading
> import signal
> import marshal

The difference between Twisted and "low level" attempts at socket 
programming (either with asyncore/asynchat or directly with 
select/poll) is that Twisted accounts for lots of platform differences, 
and has years of notches on its belt.  Doing it on your own might 
appear to work, but it won't be as scalable, might have some really 
hard to track down bugs related to your misunderstanding of the dark 
secrets that the BSD sockets API is hiding from you, and it won't be 
easy to integrate with other services.

I felt the same way as you for my first asynchronous socket application 
(4 or 5 years ago, when Twisted was in a much poorer state too), but 
after I had learned first-hand what the 'hard' problems were I realized 
that Twisted already had solutions for them, plus benefits such as 
existing implementations of various protocols and a nice thread pool 
implementation.

I also see that you're importing the marshal module... which, if used 
over the wire, is terribly terribly insecure.  You can crash an app 
with a bad marshal, and marshals are specific to a particular Python 
API version.  If you have a client running Python 2.3 and a server 
running Python 2.4, marshal will probably not work.  It will definitely 
not work with alternative Python implementations such as Jython.  If it 
is used for persistence, you may not be able to read your persistent 
datastore if you upgrade Python (though in practice you probably will, 
Python makes no such guarantee).  Twisted solves this problem as it has 
its own safe serialization mechanism (jelly) used by a nice RPC API 
called Perspective Broker.  If you are using marshal for just 
persistence, use pickle instead, it is not tightly bound to the Python 
implementation.

Also, if I were designing such an app, I would probably use SQLite or 
PostgreSQL rather than MySQL.  At the low-mid range, SQLite 3 is 
godawful fast and makes backup and migration really really easy.  Going 
up a few notches, PostgreSQL is more powerful than MySQL in that it 
supports more stuff useful for development, has less stupid "gotchas", 
and a much better license.  Also, in the case of PostgreSQL, the Python 
DB-API adapters available are a whole hell of a lot better written, and 
there is even a pure python implementation available 
<http://barryp.org/software/bpgsql>.

The best way to learn Twisted programming these days seems to be the 
step by step tutorial of writing a Finger server taking advantage of 
some of the various features Twisted has to offer 
<http://twistedmatrix.com/documents/current/howto/tutorial/index>.  
Other than that, #twisted on IRC is usually helpful, as-is the 
<twisted-python at twistedmatrix.com> list.  Besides that, I have 
significant (years) of experience with Twisted and I am a contributor 
(though I don't have much time to hack on Twisted these days), so I can 
provide some help on or off this list.

It may be particularly useful if you kept a log of your learning 
experience so that you can clean it up and write an article.  The 
article would not only help other sysadmin/developer types looking to 
write simple applications to better their network, but it will also 
help the Twisted folks patch a few holes in the learning curve.

-bob