In Part 1 of this series, we looked at the low-level aspects of Twisted, such as defining custom protocols. In a lot of ways, these low-level aspects of Twisted are the easiest to jump into. Even though asynchronous, non-blocking styles are somewhat novel for developers accustomed to threading, a new protocol can follow the examples in the Twisted Matrix documentation. The higher-level tools for Web development are undergoing more rapid flux, and have more API details to learn. In fact, Twisted's Web templating framework, woven, while becoming quite sophisticated, is unstable enough that I will only touch on it here.
A note on the name of the Twisted library is worthwhile. "Twisted Matrix Laboratories" is the name a geographically diverse group of developers call themselves, with a certain levity. The Python library for event-driven network programming is called just "Twisted" -- my last column did not carefully distinguish the group from the product.
Enhancing the Weblog server
We looked earlier at a slightly-better-than-trivial server that used a custom protocol, with custom servers and clients, to remotely monitor hits to a Web site. For this installment, let's enhance that functionality with a Web-based interface. A certain URL can be used, in our scenario, to monitor the hits a Web site receives.
There is a very simple approach to a Web-based Weblog server that has nothing to do with Twisted per se. Suppose you simply let a Web page like Weblog.html list some information about the latest few hits to a Web site. In keeping with the prior examples, we will display the referrer and resource of a hit, but only when the request has a status code of
200 (and referrer is available). You can see an example of such a page (that is not being updated for content) at my Web site (see Resources for a link).
We need to do two things: (1) Put a
<meta http-equiv=refresh ...> tag in the HTML header to keep the display up to date, and (2) Rewrite the Weblog.html file itself intermittently when new hits occur. The second task requires only a background process that is left running. For example:
Listing 1. logmaker.py Weblog refresher script
from webloglib import log_fields, TOP, ROW, END, COLOR import webloglib as wll from urllib import unquote_plus as uqp import os, time LOG = open('../access-log') RECS =  PAGE = 'www/weblog.html' while 1: page = open(PAGE+'.tmp','w') RECS.extend(LOG.readlines()) RECS = RECS[-35:] print >> page, TOP odd = 0 for rec in RECS: hit = [field.strip('"') for field in log_fields(rec)] if hit[wll.status]=='200' and hit[wll.referrer]!='-': resource = hit[wll.request].split() referrer = uqp(hit[wll.referrer]).replace('&',' &') print >> page, ROW % (COLOR[odd], referrer, resource) odd = not odd print >> page, END page.close() os.rename(PAGE+'.tmp',PAGE) time.sleep(5)
The precise HTML used is contained in the module
Webloglib, along with some constants for log field positions. You can download that module from the URL listed in the Resources section.
Notice here that you do not even need to use Twisted as a server -- Apache or any other Web server works fine.
Creating a Twisted Web server
Running a Twisted Web server is quite simple -- perhaps even easier than launching other servers. The first step in running a Twisted Web server is creating a .tap file, as we saw in the first installment. You can create a .tap file by defining an application in a script, including a call to
application.save(), and running the script. But you can also create a .tap file using the tool
mktap. In fact, for many common protocols, you can create a server .tap file without any special script at all. For example:
mktap web --path ~/twisted/www --port 8080
This creates a fairly generic server that serves files out of the base directory ~/twisted/www on port 8080. To run the server, use the tool twistd to launch the created Web.tap file:
twistd -f web.tap
For servers of types other than HTTP, you could also use other names in place of
manhole, and others. Some of those name familiar servers; others are special to Twisted. And more are added all the time.
Any static HTML files that happen to live in the base directory are delivered by the server, much as with other servers. In addition, you may also serve dynamic pages that have the extension .rpy -- in concept, these are like CGI scripts, but they avoid the forking overhead and interpreter startup time that slows down CGI. A Twisted dynamic script is arranged slightly differently than a CGI script; in the simplest case it can look something like:
Listing 2. www/dynamic.rpy Twisted page
from twisted.web import resource page = '''<html><head><title>Dynamic Page</title></head> <body> <p>Dynamic Page served by Twisted Matrix</p> </body> </html>''' class Resource(resource.Resource): def render(self, request): return page resource = Resource()
The file-level variable
resource is special -- it needs to point to an instance of a
twisted.web.resource.Resource child, where its class defines a
.render() method. You can include as many dynamic pages as you like within the directory served, and each will be served automatically.
Using Twisted to update a static page
The timed callback technique presented in the first Twisted installment can be used to periodically update the Weblog.html file discussed above. That is, you can substitute a non-blocking
twisted.internet.reactor.callLater() call for the
time.sleep() call in
Listing 3. tlogmaker.py Weblog refresher script
from webloglib import log_fields, TOP, ROW, END, COLOR import webloglib as wll from urllib import unquote_plus as uqp import os, twisted.internet LOG = open('../access-log') RECS =  PAGE = 'www/weblog.html' def update(): global RECS page = open(PAGE+'.tmp','w') RECS.extend(LOG.readlines()) RECS = RECS[-35:] print >> page, TOP odd = 0 for rec in RECS: hit = [field.strip('"') for field in log_fields(rec)] if hit[wll.status]=='200' and hit[wll.referrer]!='-': resource = hit[wll.request].split() referrer = uqp(hit[wll.referrer]).replace('&',' &') print >> page, ROW % (COLOR[odd], referrer, resource) odd = not odd print >> page, END page.close() os.rename(PAGE+'.tmp',PAGE) twisted.internet.reactor.callLater(5, update) update() twisted.internet.reactor.run()
There is not much difference between
tlogmaker.py -- both can be launched in the
background and left running to update the page
refresher.html. What would be more interesting
would be to build
tlogmaker.py directory into a
Twisted server, rather than simply have it run in a background process.
Easy enough; we just need two more lines at the end of the script:
from twisted.web import static
resource = static.File("~/twisted/www")
The call to
may also be removed. With these changes, create a server using:
mktap web --resource-script=tlogmaker.py --port 8080
And run the created
web.tap server using
twistd, as before. Now the Web server itself
will refresh the page Weblog.html every five
seconds, using its standard core dispatch loop.
Making the Weblog a dynamic page
Another approach to serving the Web log is to use a dynamic page to
generate the most recent hits each time they are requested. However, it
is a bad idea to read the entire
file each time such a request is received -- a busy Web site is likely to
have many thousands of records in a log file; reading those repeatedly is
time consuming. A better approach is to let the Twisted server itself
hold a file handle for the log file, and only read new records when
In a way, having the server maintain a file handle is just what
tlogmaker.py does, but it stores the latest records
in a file rather than in memory. However, that approach forces us to
write the whole server around this persistence function. It is more
elegant to let individual dynamic pages make their own persistence
requests to the server. This way, for example, you can add new stateful
dynamic pages without stopping or altering the long-running (and generic)
server. The key to page-allocated persistence is Twisted's
registry. For example, here is a dynamic page that serves the
Listing 4. www/Weblog.rpy dynamic Weblog page
from twisted.web import resource, server from persist import Records from webloglib import log_fields, TOP, ROW, END, COLOR import webloglib as wll records = registry.getComponent(Records) if not records: records = Records() registry.setComponent(Records, records) class Resource(resource.Resource): def render(self, request): request.write(TOP) odd = 0 for rec in records.getNew(): print rec hit = [field.strip('"') for field in log_fields(rec)] if hit[wll.status]=='200' and hit[wll.referrer]!='-': resource = hit[wll.request].split() referrer = hit[wll.referrer].replace('&',' &') request.write(ROW % (COLOR[odd],referrer,resource)) odd = not odd request.write(END) request.finish() return server.NOT_DONE_YET resource = Resource()
One thing that is initially confusing about the registry is that it is never imported by Weblog.rpy. An .rpy script is not quite the same as a plain .py script -- the former runs within the Twisted environment, which provides automatic access to
register among other things. The
request object is another thing that comes from the framework rather than from the .rpy itself.
Notice also the somewhat new style of returning the page contents. Rather than just return an HTML string, in the above, we cache several writes to the
request object, then finish them up with the call to
request.finish(). The odd-looking return value
server.NOT_DONE_YET is a flag to the Twisted server to flush the page content out of the
request object. Another option is to add a
Deferred object to the request, and serve the page when the callback to the
Deferred is performed (for example, if the page cannot be generated until a database query completes).
Creating persistent objects
Notice the little conditional logic at the top of Weblog.rpy. The first time the dynamic page is served, no
Records object has yet been added to the registry. But after that first time, we want to keep using the same object for each call to
records.getNew(). The call to
registry.getComponent() returns the registered object of the appropriate class if it can; otherwise, it returns a false value to allow testing. Between calls, of course, the object is maintained in the address space of the Twisted server.
A persistence class should best live in a module that the .rpy file imports. This way, every dynamic page can utilize persistence classes you write. Any sort of persistence you like can be contained in the instance attributes. However, some things such as open files cannot be saved between shutdowns of the server (simple values, however, can be persisted between server runs, and are saved in a file such as web-shutdown.tap). The module
persist that I use contains one very simple class,
Counter, that is borrowed from the Twisted Matrix documentation, and another,
Records, that I use for the Weblog dynamic page:
Listing 5. Persistence support module persist.py
class Counter: def __init__(self): self.value = 0 def increment(self): self.value += 1 def getValue(self): return self.value class Records: def __init__(self, log_name='../access-log'): self.log = open(log_name) self.recs = self.log.readlines() def getNew(self): self.recs.extend(self.log.readlines()) self.recs = self.recs[-35:] return self.recs
You are free to put whatever methods you like in persistence classes -- the registry simply holds instances in memory between different calls to dynamic pages.
For next time
In this installment, we have looked at the basics of Twisted Web servers. A basic server (or even one with minor custom code) is easy to set up. But greater power is available in the
twisted.web.woven module, which provides a templating system for Twisted Web servers. In outline, woven provides a programming style similar to PHP, ColdFusion, or JSP, but arguably with a more useful division between code and templates than those other systems offer (and of course,
twisted.web.woven lets you program in Python). In Parts 3 and 4 of this series, we will also address dynamic pages and Web security.
- Twisted Matrix comes with quite a bit of documentation and many examples. Browse around the Twisted Matrix Laboratories home page to glean a greater sense of how it works and what has been implemented with it (or see the next installments here at developerWorks).
- Read David's introduction to Twisted in Part 1 of this series (developerWorks, June 2003).
- In Part 3 of this series, David looks at dynamic Web serving and shows how to generate dynamic Web pages using the Woven application for templating pages (developerWorks, August 2003).
- In Part 4 of this series, David looks at specialized protocols and servers contained in the Twisted package, with a focus on secure connections (developerWorks, September 2003).
- A simple version of a Weblog server was presented in the tip "Asynchronous SAX: Use Simple API for XML as a long-running event processor" (developerWorks, May 2003).
- Learn more about the Apache Web server in the tutorial "Customizing Apache for maximum performance" (developerWorks, June 2002).
- You can see an example Weblog.html page at David's site.
- You can download the Webloglib module and other things associated with this Twisted series at David's site as well.
- Find more articles for Python developers in the developerWorks Linux zone.
Dig deeper into Linux on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Experiment with new directions in software development.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.