Contents


Writing plug-ins in Python

Learn to write your first plug-in by extending a command-line tool in Python

Comments

In a previous article for IBM developerWorks I wrote about the joy of creating command-line tools with Python. This article takes the command-line tools to the next level by creating plug-ins to extend them. Both plug-ins and command-line tools offer a convenient way to extend the functionality of existing code. Used together, they can be a very powerful tool.

To get started writing the plug-in, you are going to use an open source Python package I wrote called pathtool, which uses generators to walk a filesystem and yield a file object. This library was specifically written to allow developers to extend it by writing their own filters that do something to a file object, and then return the result.

The actual Python module code is a bit larger than you would like to see for an article, so I will only post a snippit of the API that you will actually use:

Listing 1. pathtool API
def path(fullpath, pattern="*", action=(lambda rec: print_rec(rec))):
    """This takes a path, a shell pattern, and an action callback
    This function uses the slower pathattr function which calculates checksums
    """ 
    for rec in pathattr(fullpath):
        for new_record in match(pattern, rec):  #applies filter
            action(new_record)  #Applies lambda callback to generator object

Looking at this example, you can tell that the path function takes a mandatory path-positional argument along with a optional pattern keyword argument, and an optional action keyword argument called lambda callback. The default callback for path just print out the filename as an example. A developer would just need to easy_install pathtool. See the resources section for information on using the easy_install command, and then import the module and call the function as follows:

      from pathtool import path
      path("/tmp", pattern="*.mp3", action=(lambda rec: print_rec(rec)))

Note: I have included the source code for pathtool with this article for added convenience. One thing to point out about this example is the use of lambda. You can read a link to a Python tutorial entry in the resources section about them, but in a nutshell, a lambda is a convenient way to tell a function to "call" another function.

Writing an pluggable command-line tool

Now that we have a general idea of how to use a path-walking library that includes a callback, it is time to jump into actually writing a command-line tool that is extensible using plug-ins. Look at the finished version first, and then I'll break it down into smaller pieces:

Listing 2. Command-line tool with plug-ins
#!/usr/bin/env python
# encoding: utf-8
"""
pathtool-cli.py 0.1

A commandline tool for walking a filesystem.
Takes Action callback plugins in a plugin directory

action=(lambda rec: print_rec(rec))

"""

from pathtool import path
import optparse
import re
import os
import sys

try:
    plugin_available = True
    from plugin import *
    from plugin import __all__ #note this is the registered plugin list
except ImportError:
    plugin_available = False
        
def path_controller():
    descriptionMessage = """
    A command line tool for walking a filesystem.\
    Takes callback 'Action' functions as plugins.\
    
    example: pathtool_cli /tmp print_path_ext
    """
    p = optparse.OptionParser(description=descriptionMessage,
                                prog='pathtool',
                                version='pathtool 0.1.1',
                                usage= '%prog [starting directory][action]')
    p.add_option('--pattern', '-p',
                help='Pattern Match Examples: *.txt, *.iso, music[0-5].mp3\
                plain number defaults to * or match all.  \
                Uses UNIX standard wildcard syntax.',
                default='*')

    p.add_option('--list', '-l',
            action="store_true",
            help='lists available action plugins',
            default=False)
                    
    options, arguments = p.parse_args()
    if options.list:
        try:
            print "Action Plugins Available:"
            if plugin_available:
                for p in __all__:
                    print p
        finally:
            sys.exit(0)
            
    if len(arguments) == 2:
        fullpath = arguments[0]
        try:
            action_plugin = eval(arguments[1])   
            #note we expect the plugin author to write a method with our naming convention
            #path(fullpath,options.pattern,action=(lambda rec: move_to_tmp.plugin(rec)))
            path(fullpath, options.pattern,action=(lambda rec: action_plugin.plugin(rec)))
        except NameError:
            sys.stderr.write("Plugin Not Found")
            sys.exit(1)
    else:
        print p.print_help()

def main():
    path_controller()

if __name__ == '__main__':
    main()

Running this example results in the following output:

# python pathtool_cli.py
Usage: pathtool [starting directory][action]

     A command line tool for walking a filesystem.    Takes callback 'Action'
functions as plugins.         example: pathtool_cli /tmp print_path_ext

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -p PATTERN, --pattern=PATTERN
                        Pattern Match Examples: *.txt, *.iso, music[0-5].mp3
                        plain number defaults to * or match all.
                        Uses UNIX standard wildcard syntax.
  -l, --list            lists available action plugins

From the output of the command, you can see that the tool expects to take a fullpath, and then an "action." The action is a plug-in that a developer creates. I have added a command-line option list that allows the user of the command-line tool to see what plug-ins are available. Look at the output of this:

# python pathtool_cli.py -l
Action Plugins Available:
move_to_tmp
print_file_path_ext

Without knowing too much about how the tool works, someone can probably guess what an action does by the name of it. Because the print_file_path_ext action I have written just prints the file, path, ext, go ahead and run that and see what it looks like:

# python pathtool_cli.py /tmp print_file_path_ext
/tmp/foo0.txt | foo0.txt | .txt
/tmp/foo1.txt | foo1.txt | .txt
/tmp/foo10.txt | foo10.txt | .txt
/tmp/foo2.txt | foo2.txt | .txt
/tmp/foo3.txt | foo3.txt | .txt
/tmp/foo4.txt | foo4.txt | .txt
/tmp/foo5.txt | foo5.txt | .txt
/tmp/foo6.txt | foo6.txt | .txt
/tmp/foo7.txt | foo7.txt | .txt
/tmp/foo8.txt | foo8.txt | .txt
/tmp/foo9.txt | foo9.txt | .txt

I created ten temp files using touch foo{0..10}.txt earlier, and now the command-line tool used a plug-in that it found to print the fullpath, the filename, and the extension, all separated by a "|" character.

Simple plug-in architecture explained

Up until now, I have been expecting you to trust me, without telling you where these "magic" plug-in actions are coming from. The first thing to look at is the import statement at the top of the module. Try the following:

    plugin_available = True
    from plugin import *
    from plugin import __all__ #note this is the registered plugin list
except ImportError:
    plugin_available = False

This import statement gives away the secret to the surprisingly simple plug-in architecture. The official Python documentation generally discourages the use of the syntax "from package import *", but if there is a good reason for it, such as writing plug-ins, then the plug-in author is responsible for creating an entry in the __init__.py file located in the plug-in directory. It should look like this:

"""Lists all of the importable plugins"""
__all__ = ["move_to_tmp", "print_file_path_ext"]

By setting this, it allows all of the modules inside of the package, or directory, to be imported as *. Next I import the actual __all__ list to use as a way of showing the user what plug-ins are available. Finally, there is one more small bit of magic. Because the command-line tool doesn't know until runtime what plug-in action to use, use eval to convert the action string on the command-line into a callable function. This is the magic line:

action_plugin = eval(arguments[1])

Generally, the use of eval should be used with extreme caution, but in this case, it is reasonable to tell our tool to use plug-in methods.

A look at a plug-in

Now that you understand how the plug-in architecture is supposed to work, look at an actual plug-in. Note that in order for this architecture to work, there needs to be a plug-in directory in the current working directory or the Python site-packages directory. This particular plug-in is called print_file_path_ext.py, and it has a method called plug-in. This is an expected API that a plug-in developer must conform to:

Listing 3. Example plug-in
#!/usr/bin/env python
# encoding: utf-8
"""
prints path, name, ext, plugin
"""

def plugin(rec, verbose=True):
    """Moves matched files to tmp directory"""
    path = rec["path"]
    filename = rec["filename"]
    ext = rec["ext"]
    print "%s | %s | %s" % (path, filename, ext)

This plug-in is a very simple function, but it takes a rec parameter that is a dictionary that the pathtool module generates. That dictionary includes the following API:

{"path": path, "filename": file, "ext": ext, "size": size,
"unique_id": unique_id, "mtime": mtime, "ctime": ctime}

In this example, I use the keys of the dictionary to print the values for that particular file object each time it is called. A plug-in author could write many other useful actions that could convert files, rename files, archive files, and more.

Summary

This article demonstrated a reasonably simple plug-in architecture that can be a useful way to extend command-line tools in Python. There are a few things that should be noted, though. First, there is a more sophisticated plug-in system available with easy_install, which is included in the references. This plug-in system allows a user to create "entry points" that define plug-ins for a particular tool. Second, the way our command-line tools is written only allows for one "action" plug-in. I will leave it as an exercise to the reader to modify the command-line tool such that it could receive a limitless amount of "chainable" callback actions.

One potential gotcha with creating chainable plug-ins is that the design must take into account the nature of the API it is using. In our case we are building on a foundation of generators that yields. In order for our tool to continue "chaining" plug-ins together, they must do their work, yet still yield the dictionary record back. I hope this article inspired you to write your own plug-ins for command-line tools, as well.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=335078
ArticleTitle=Writing plug-ins in Python
publish-date=09022008