OK more Python, some code and a rant -- it all happens here.
Rant On First let me get this out of my system - Python has a great standard library and it's great seeing packages extended, mature and versions come out that provide additional capability or performance. Also the Cheeseshop and Sourceforge host many great Python projects - so why do we see so many of these packages and frameworks re-create standard features such as logging? For example I've been using both Twisted and Django lately and both include their own custom logging which seems to add nothing over the capabilities of the standard library logging package. I know most of us are used to print statements (or System.out.println for my Java colleagues) and it's a slippery slope to create debug() methods and then move them to their own package and soon you have your own logger. Now when we have particular needs we sometimes have to roll our own solution, but hopefully these requirements get fed back into the main project for the community to benefit from. Rant Off
In the case of logging eventually we deploy systems and these logs are important, we collect them, analyze them and correlate events across systems to determine faults. It's this thinking that led to the Common Base Event format which IBM and others brought to OASIS and which has now become the WSDM Event Format or WEF. The purpose is to provide a common, XML based, event record format that can be used to log events for later correlation and analysis. An example WEF event is shown below, generated by an extension package for Python's common logging. This package(available as loggingx-0.1.1.zip) uses the standard extension mechanisms of the logging library, in this case providing a customer event formatter object (WEFFormatter) that can convert Python event records into the XML shown below. There are also libraries for other languages, notably in Java you can use the Apache MUSE project.
<ManagementEvent ReportTime="2007-02-14T18:12:45Z" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://docs.oasis-open.org/wsdm/muws1-2.xsd"> <EventId>urn:uuid:e0bf22a6-bc80-11db-bfac-0014a4dafba5</EventId> <SourceComponent processId="5676" locationType="path" threadId="2344" component="logtest" componentType="Module" application="d:\Python25\python.exe" location="d:\my projects\eclipse3.2m3\logging\src\test\logtest.py" componentIdType="Module"> </SourceComponent> <ReporterComponent executionEnvironment="win32" componentIdType="Module" subComponent="WEFFormatter" component="logging" componentType="Module" location="skjohnt60p" locationType="Hostname"> </ReporterComponent> <Situation> <SituationCategory> <ReportSituation> <Log></Log> </ReportSituation> </SituationCategory> <SituationTime>2007-02-14T18:12:45Z</SituationTime> <Priority>50</Priority> <Severity>4</Severity> <Message>Oops, that didnt really work now did it.</Message> </Situation> <extendedContent> <extendedDataElements type="xsd:int" name="Line"> <values>36</values> </extendedDataElements> <extendedDataElements type="xsd:string" name="Exception"> <values>integer division or modulo by zero</values> <children type="xsd:string" name="Traceback"> <values>File: D:\My Projects\eclipse3.2m3\logging\src\test\logtest.py, Line: 34</values> </children> <children type="xsd:string" name="Type"> <values>ZeroDivisionError</values> </children> </extendedDataElements> </extendedContent> </ManagementEvent>
The following snippet shows how to connect this custom formatter to the logger, setting the log level to DEBUG so we should see lots of messages as we go. If you provide a 'format' value on the call to basicConfig() it will be ignored by the WEFFormatter which doesn't support any configuration of the output format.
import logging # standard library! from loggingx.wef import WEFFormatter
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout) logger = logging.getLogger('') handler = logging.StreamHandler() handler.setFormatter(WEFFormatter()) logger.addHandler(handler)
The example event above was actually reporting a Python exception using the exception() method on the logger object. This method includes exception information in the event record before passing to the formatter object and our WEFFormatter uses the extendedContent tag to encompass details about the exception (the traceback information and the type of the exception) in the formatted event. The test code which generated the XML above is shown in the snippet below.
try: 1/0 except Exception, e: logger.exception("Oops, that didnt really work now did it.")
However gobs and gobs of XML are not necessarily the most effective of efficient native format for log records (and there are many applications that have well known existing log formats - such as Apache) and so the loggingx package also includes a module for logging events to a SQL database using the Python standard DBAPI. This module allows multiple applications to log to the same database and therefore provides an efficient method for log analysis across applications. In this case we simply created a customer log Handler object, called DatabaseHandler, which can take a set of initialization parameters to choose the database API and provide connection details. Below is a snippet showing how to use the default configuration (SQLite3 - provided with Python 2.5) and how we add this handler to the logger. Note that this handler will ignore any provided formatter as it has to do it's own 'formatting' that is extract data from the log record and insert into a SQL row.
handler = DatabaseHandler() logger = logging.getLogger('') logger.addHandler(handler)
However, for analysis applications expecting to use WEF (such as those provided in the IBM Autonomic Tool Kit) an additional module in the package provides a function to extract records from a log database and format according to any provided log formatter object. The snippet below shows how the run_extract() function works, taking an existing, opened, connection object and any log formatter object and the identifier of the first log record to extract. In this manner we could not only extract log records into WEF events but perhaps into Apache style log records or more.
from loggingx.db.extract import run_extract from loggingx.wef import WEFFormatter
db = os.path.join(os.environ['TEMP'], 'pylogging.db') connection = sqlite3.connect(db) run_extract(connection, 0, WEFFormatter())
Hopefully we will over time see more and more projects using the standard logging and, if they have special needs to extend the package themselves or contribute these changes back to the community.[Read More]