Writing plug-ins in Python

Learn to write your first plug-in by extending a command-line tool in Python

Learn how to extend your Python command-line tools by writing plug-ins.

Share:

Noah Gift, Software Engineer, Giftcs

Photo of Noah GIftNoah Gift is the co-author of "Python For Unix and Linux" by O'Reilly. He is an author, speaker, consultant, and community leader, writing for publications such as IBM developerWorks, Red Hat Magazine, O'Reilly, and MacTech. His consulting company's website is www.giftcs.com, and his personal website is www.noahgift.com. Noah is also the current organizer for www.pyatl.org, which is the Python User Group for Atlanta, GA. He has a Master's degree in CIS from Cal State Los Angeles, B.S. in Nutritional Science from Cal Poly San Luis Obispo, is an Apple and LPI certified SysAdmin, and has worked at companies such as, Caltech, Disney Feature Animation, Sony Imageworks, and Turner Studios. In his free time he enjoys spending time with his wife Leah, and their son Liam, playing the piano, and exercising religiously.


developerWorks Contributing author
        level

02 September 2008

Also available in Chinese

Introduction

What is a plug-in?

One of the more popular examples of the use of plug-ins to extend an application is the Firefox plug-in community. There are Firefox plug-ins for Flash players, Web development, managing RSS feeds, and more. One of the more common uses of plug-ins is to enable third-party developers to extend an application. This is a very powerful idea, as it allows the original developers to harness the power of the unknown to extend their product in ways they may have not even imagined. Please see the Resources section for more information on plug-ins.

In a previous article for IBM developerWorks I wrote about the joy of creating command-line tools with Python. This article takes the command-line tools to the next level by creating plug-ins to extend them. Both plug-ins and command-line tools offer a convenient way to extend the functionality of existing code. Used together, they can be a very powerful tool.

To get started writing the plug-in, you are going to use an open source Python package I wrote called pathtool, which uses generators to walk a filesystem and yield a file object. This library was specifically written to allow developers to extend it by writing their own filters that do something to a file object, and then return the result.

The actual Python module code is a bit larger than you would like to see for an article, so I will only post a snippit of the API that you will actually use:

Listing 1. pathtool API
def path(fullpath, pattern="*", action=(lambda rec: print_rec(rec))):
    """This takes a path, a shell pattern, and an action callback
    This function uses the slower pathattr function which calculates checksums
    """ 
    for rec in pathattr(fullpath):
        for new_record in match(pattern, rec):  #applies filter
            action(new_record)  #Applies lambda callback to generator object

Looking at this example, you can tell that the path function takes a mandatory path-positional argument along with a optional pattern keyword argument, and an optional action keyword argument called lambda callback. The default callback for path just print out the filename as an example. A developer would just need to easy_install pathtool. See the resources section for information on using the easy_install command, and then import the module and call the function as follows:

      from pathtool import path
      path("/tmp", pattern="*.mp3", action=(lambda rec: print_rec(rec)))

Note: I have included the source code for pathtool with this article for added convenience. One thing to point out about this example is the use of lambda. You can read a link to a Python tutorial entry in the resources section about them, but in a nutshell, a lambda is a convenient way to tell a function to "call" another function.

Writing an pluggable command-line tool

Now that we have a general idea of how to use a path-walking library that includes a callback, it is time to jump into actually writing a command-line tool that is extensible using plug-ins. Look at the finished version first, and then I'll break it down into smaller pieces:

Listing 2. Command-line tool with plug-ins
#!/usr/bin/env python
# encoding: utf-8
"""
pathtool-cli.py 0.1

A commandline tool for walking a filesystem.
Takes Action callback plugins in a plugin directory

action=(lambda rec: print_rec(rec))

"""

from pathtool import path
import optparse
import re
import os
import sys

try:
    plugin_available = True
    from plugin import *
    from plugin import __all__ #note this is the registered plugin list
except ImportError:
    plugin_available = False
        
def path_controller():
    descriptionMessage = """
    A command line tool for walking a filesystem.\
    Takes callback 'Action' functions as plugins.\
    
    example: pathtool_cli /tmp print_path_ext
    """
    p = optparse.OptionParser(description=descriptionMessage,
                                prog='pathtool',
                                version='pathtool 0.1.1',
                                usage= '%prog [starting directory][action]')
    p.add_option('--pattern', '-p',
                help='Pattern Match Examples: *.txt, *.iso, music[0-5].mp3\
                plain number defaults to * or match all.  \
                Uses UNIX standard wildcard syntax.',
                default='*')

    p.add_option('--list', '-l',
            action="store_true",
            help='lists available action plugins',
            default=False)
                    
    options, arguments = p.parse_args()
    if options.list:
        try:
            print "Action Plugins Available:"
            if plugin_available:
                for p in __all__:
                    print p
        finally:
            sys.exit(0)
            
    if len(arguments) == 2:
        fullpath = arguments[0]
        try:
            action_plugin = eval(arguments[1])   
            #note we expect the plugin author to write a method with our naming convention
            #path(fullpath,options.pattern,action=(lambda rec: move_to_tmp.plugin(rec)))
            path(fullpath, options.pattern,action=(lambda rec: action_plugin.plugin(rec)))
        except NameError:
            sys.stderr.write("Plugin Not Found")
            sys.exit(1)
    else:
        print p.print_help()

def main():
    path_controller()

if __name__ == '__main__':
    main()

Running this example results in the following output:

# python pathtool_cli.py
Usage: pathtool [starting directory][action]

     A command line tool for walking a filesystem.    Takes callback 'Action'
functions as plugins.         example: pathtool_cli /tmp print_path_ext

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -p PATTERN, --pattern=PATTERN
                        Pattern Match Examples: *.txt, *.iso, music[0-5].mp3
                        plain number defaults to * or match all.
                        Uses UNIX standard wildcard syntax.
  -l, --list            lists available action plugins

From the output of the command, you can see that the tool expects to take a fullpath, and then an "action." The action is a plug-in that a developer creates. I have added a command-line option list that allows the user of the command-line tool to see what plug-ins are available. Look at the output of this:

# python pathtool_cli.py -l
Action Plugins Available:
move_to_tmp
print_file_path_ext

Without knowing too much about how the tool works, someone can probably guess what an action does by the name of it. Because the print_file_path_ext action I have written just prints the file, path, ext, go ahead and run that and see what it looks like:

# python pathtool_cli.py /tmp print_file_path_ext
/tmp/foo0.txt | foo0.txt | .txt
/tmp/foo1.txt | foo1.txt | .txt
/tmp/foo10.txt | foo10.txt | .txt
/tmp/foo2.txt | foo2.txt | .txt
/tmp/foo3.txt | foo3.txt | .txt
/tmp/foo4.txt | foo4.txt | .txt
/tmp/foo5.txt | foo5.txt | .txt
/tmp/foo6.txt | foo6.txt | .txt
/tmp/foo7.txt | foo7.txt | .txt
/tmp/foo8.txt | foo8.txt | .txt
/tmp/foo9.txt | foo9.txt | .txt

I created ten temp files using touch foo{0..10}.txt earlier, and now the command-line tool used a plug-in that it found to print the fullpath, the filename, and the extension, all separated by a "|" character.

Simple plug-in architecture explained

Up until now, I have been expecting you to trust me, without telling you where these "magic" plug-in actions are coming from. The first thing to look at is the import statement at the top of the module. Try the following:

    plugin_available = True
    from plugin import *
    from plugin import __all__ #note this is the registered plugin list
except ImportError:
    plugin_available = False

This import statement gives away the secret to the surprisingly simple plug-in architecture. The official Python documentation generally discourages the use of the syntax "from package import *", but if there is a good reason for it, such as writing plug-ins, then the plug-in author is responsible for creating an entry in the __init__.py file located in the plug-in directory. It should look like this:

"""Lists all of the importable plugins"""
__all__ = ["move_to_tmp", "print_file_path_ext"]

By setting this, it allows all of the modules inside of the package, or directory, to be imported as *. Next I import the actual __all__ list to use as a way of showing the user what plug-ins are available. Finally, there is one more small bit of magic. Because the command-line tool doesn't know until runtime what plug-in action to use, use eval to convert the action string on the command-line into a callable function. This is the magic line:

action_plugin = eval(arguments[1])

Generally, the use of eval should be used with extreme caution, but in this case, it is reasonable to tell our tool to use plug-in methods.

A look at a plug-in

Now that you understand how the plug-in architecture is supposed to work, look at an actual plug-in. Note that in order for this architecture to work, there needs to be a plug-in directory in the current working directory or the Python site-packages directory. This particular plug-in is called print_file_path_ext.py, and it has a method called plug-in. This is an expected API that a plug-in developer must conform to:

Listing 3. Example plug-in
#!/usr/bin/env python
# encoding: utf-8
"""
prints path, name, ext, plugin
"""

def plugin(rec, verbose=True):
    """Moves matched files to tmp directory"""
    path = rec["path"]
    filename = rec["filename"]
    ext = rec["ext"]
    print "%s | %s | %s" % (path, filename, ext)

This plug-in is a very simple function, but it takes a rec parameter that is a dictionary that the pathtool module generates. That dictionary includes the following API:

{"path": path, "filename": file, "ext": ext, "size": size,
"unique_id": unique_id, "mtime": mtime, "ctime": ctime}

In this example, I use the keys of the dictionary to print the values for that particular file object each time it is called. A plug-in author could write many other useful actions that could convert files, rename files, archive files, and more.

Summary

This article demonstrated a reasonably simple plug-in architecture that can be a useful way to extend command-line tools in Python. There are a few things that should be noted, though. First, there is a more sophisticated plug-in system available with easy_install, which is included in the references. This plug-in system allows a user to create "entry points" that define plug-ins for a particular tool. Second, the way our command-line tools is written only allows for one "action" plug-in. I will leave it as an exercise to the reader to modify the command-line tool such that it could receive a limitless amount of "chainable" callback actions.

One potential gotcha with creating chainable plug-ins is that the design must take into account the nature of the API it is using. In our case we are building on a foundation of generators that yields. In order for our tool to continue "chaining" plug-ins together, they must do their work, yet still yield the dictionary record back. I hope this article inspired you to write your own plug-ins for command-line tools, as well.


Download

DescriptionNameSize
Sample CLI Plug-in Codecli_plugin_code.zip15KB

Resources

Learn

Get products and technologies

  • IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=335078
ArticleTitle=Writing plug-ins in Python
publish-date=09022008