Introduction

In a previous article for IBM developerWorks I wrote about the joy of creating command-line tools with Python. This article takes the command-line tools to the next level by creating plug-ins to extend them. Both plug-ins and command-line tools offer a convenient way to extend the functionality of existing code. Used together, they can be a very powerful tool.

To get started writing the plug-in, you are going to use an open source Python package I wrote called pathtool, which uses generators to walk a filesystem and yield a file object. This library was specifically written to allow developers to extend it by writing their own filters that do something to a file object, and then return the result.

The actual Python module code is a bit larger than you would like to see for an article, so I will only post a snippit of the API that you will actually use:

def path(fullpath, pattern="*", action=(lambda rec: print_rec(rec))):
    """This takes a path, a shell pattern, and an action callback
    This function uses the slower pathattr function which calculates checksums
    """ 
    for rec in pathattr(fullpath):
        for new_record in match(pattern, rec):  #applies filter
            action(new_record)  #Applies lambda callback to generator object
       
       

Looking at this example, you can tell that the path function takes a mandatory path-positional argument along with a optional pattern keyword argument, and an optional action keyword argument called lambda callback. The default callback for path just print out the filename as an example. A developer would just need to easy_install pathtool. See the resources section for information on using the easy_install command, and then import the module and call the function as follows:

      from pathtool import path
      path("/tmp", pattern="*.mp3", action=(lambda rec: print_rec(rec)))
      

Note: I have included the source code for pathtool with this article for added convenience. One thing to point out about this example is the use of lambda. You can read a link to a Python tutorial entry in the resources section about them, but in a nutshell, a lambda is a convenient way to tell a function to "call" another function.

Writing an pluggable command-line tool

Now that we have a general idea of how to use a path-walking library that includes a callback, it is time to jump into actually writing a command-line tool that is extensible using plug-ins. Look at the finished version first, and then I'll break it down into smaller pieces:

#!/usr/bin/env python
# encoding: utf-8
"""
pathtool-cli.py 0.1

A commandline tool for walking a filesystem.
Takes Action callback plugins in a plugin directory

action=(lambda rec: print_rec(rec))

"""

from pathtool import path
import optparse
import re
import os
import sys

try:
    plugin_available = True
    from plugin import *
    from plugin import __all__ #note this is the registered plugin list
except ImportError:
    plugin_available = False
        
def path_controller():
    descriptionMessage = """
    A command line tool for walking a filesystem.\
    Takes callback 'Action' functions as plugins.\
    
    example: pathtool_cli /tmp print_path_ext
    """
    p = optparse.OptionParser(description=descriptionMessage,
                                prog='pathtool',
                                version='pathtool 0.1.1',
                                usage= '%prog [starting directory][action]')
    p.add_option('--pattern', '-p',
                help='Pattern Match Examples: *.txt, *.iso, music[0-5].mp3\
                plain number defaults to * or match all.  \
                Uses UNIX standard wildcard syntax.',
                default='*')

    p.add_option('--list', '-l',
            action="store_true",
            help='lists available action plugins',
            default=False)
                    
    options, arguments = p.parse_args()
    if options.list:
        try:
            print "Action Plugins Available:"
            if plugin_available:
                for p in __all__:
                    print p
        finally:
            sys.exit(0)
            
    if len(arguments) == 2:
        fullpath = arguments[0]
        try:
            action_plugin = eval(arguments[1])   
            #note we expect the plugin author to write a method with our naming convention
            #path(fullpath,options.pattern,action=(lambda rec: move_to_tmp.plugin(rec)))
            path(fullpath, options.pattern,action=(lambda rec: action_plugin.plugin(rec)))
        except NameError:
            sys.stderr.write("Plugin Not Found")
            sys.exit(1)
    else:
        print p.print_help()

def main():
    path_controller()

if __name__ == '__main__':
    main()

       
       

Running this example results in the following output:

# python pathtool_cli.py
Usage: pathtool [starting directory][action]

     A command line tool for walking a filesystem.    Takes callback 'Action'
functions as plugins.         example: pathtool_cli /tmp print_path_ext

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -p PATTERN, --pattern=PATTERN
                        Pattern Match Examples: *.txt, *.iso, music[0-5].mp3
                        plain number defaults to * or match all.
                        Uses UNIX standard wildcard syntax.
  -l, --list            lists available action plugins

From the output of the command, you can see that the tool expects to take a fullpath, and then an "action." The action is a plug-in that a developer creates. I have added a command-line option list that allows the user of the command-line tool to see what plug-ins are available. Look at the output of this:

# python pathtool_cli.py -l
Action Plugins Available:
move_to_tmp
print_file_path_ext

Without knowing too much about how the tool works, someone can probably guess what an action does by the name of it. Because the print_file_path_ext action I have written just prints the file, path, ext, go ahead and run that and see what it looks like:

# python pathtool_cli.py /tmp print_file_path_ext
/tmp/foo0.txt | foo0.txt | .txt
/tmp/foo1.txt | foo1.txt | .txt
/tmp/foo10.txt | foo10.txt | .txt
/tmp/foo2.txt | foo2.txt | .txt
/tmp/foo3.txt | foo3.txt | .txt
/tmp/foo4.txt | foo4.txt | .txt
/tmp/foo5.txt | foo5.txt | .txt
/tmp/foo6.txt | foo6.txt | .txt
/tmp/foo7.txt | foo7.txt | .txt
/tmp/foo8.txt | foo8.txt | .txt
/tmp/foo9.txt | foo9.txt | .txt

I created ten temp files using touch foo{0..10}.txt earlier, and now the command-line tool used a plug-in that it found to print the fullpath, the filename, and the extension, all separated by a "|" character.

Simple plug-in architecture explained

Up until now, I have been expecting you to trust me, without telling you where these "magic" plug-in actions are coming from. The first thing to look at is the import statement at the top of the module. Try the following:

    plugin_available = True
    from plugin import *
    from plugin import __all__ #note this is the registered plugin list
except ImportError:
    plugin_available = False

This import statement gives away the secret to the surprisingly simple plug-in architecture. The official Python documentation generally discourages the use of the syntax "from package import *", but if there is a good reason for it, such as writing plug-ins, then the plug-in author is responsible for creating an entry in the __init__.py file located in the plug-in directory. It should look like this:

"""Lists all of the importable plugins"""
__all__ = ["move_to_tmp", "print_file_path_ext"]     

By setting this, it allows all of the modules inside of the package, or directory, to be imported as *. Next I import the actual __all__ list to use as a way of showing the user what plug-ins are available. Finally, there is one more small bit of magic. Because the command-line tool doesn't know until runtime what plug-in action to use, use eval to convert the action string on the command-line into a callable function. This is the magic line:

action_plugin = eval(arguments[1]) 

Generally, the use of eval should be used with extreme caution, but in this case, it is reasonable to tell our tool to use plug-in methods.

A look at a plug-in

Now that you understand how the plug-in architecture is supposed to work, look at an actual plug-in. Note that in order for this architecture to work, there needs to be a plug-in directory in the current working directory or the Python site-packages directory. This particular plug-in is called print_file_path_ext.py, and it has a method called plug-in. This is an expected API that a plug-in developer must conform to:

#!/usr/bin/env python
# encoding: utf-8
"""
prints path, name, ext, plugin
"""

def plugin(rec, verbose=True):
    """Moves matched files to tmp directory"""
    path = rec["path"]
    filename = rec["filename"]
    ext = rec["ext"]
    print "%s | %s | %s" % (path, filename, ext)

This plug-in is a very simple function, but it takes a rec parameter that is a dictionary that the pathtool module generates. That dictionary includes the following API:

{"path": path, "filename": file, "ext": ext, "size": size,
"unique_id": unique_id, "mtime": mtime, "ctime": ctime}

In this example, I use the keys of the dictionary to print the values for that particular file object each time it is called. A plug-in author could write many other useful actions that could convert files, rename files, archive files, and more.

Summary

This article demonstrated a reasonably simple plug-in architecture that can be a useful way to extend command-line tools in Python. There are a few things that should be noted, though. First, there is a more sophisticated plug-in system available with easy_install, which is included in the references. This plug-in system allows a user to create "entry points" that define plug-ins for a particular tool. Second, the way our command-line tools is written only allows for one "action" plug-in. I will leave it as an exercise to the reader to modify the command-line tool such that it could receive a limitless amount of "chainable" callback actions.

One potential gotcha with creating chainable plug-ins is that the design must take into account the nature of the API it is using. In our case we are building on a foundation of generators that yields. In order for our tool to continue "chaining" plug-ins together, they must do their work, yet still yield the dictionary record back. I hope this article inspired you to write your own plug-ins for command-line tools, as well.

Back to top

Download

Information about download methods

Resources

Learn

• Learn about eggs with Ian Bicking PyCon Plug-in Presentation.
  
  
• Creating command-line tools in Python (developerWorks, March 2008): Learn how to create a simple command-line tool.
  
  
• Plug-in: Read the Wikipedia entry for plug-ins.
  
  
• Firefox Plug-in: Help your browser perform specific functions like viewing special graphic formats or playing multimedia files.
  
  
• Dynamic Discovery of Services and Plug-ins with Setuptools supports creating libraries that "plug in" to extensible applications and frameworks.
  
  
• Python Lambda Statements can be used wherever function objects are required.
  
  
• Pathtool is an efficient API to walking a filesystem.
  
  
• The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of IBM® AIX® systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX? Visit the New to AIX and UNIX page to learn more.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the AIX and UNIX forums:
  • AIX Forum
  • AIX Forum for developers
  • Cluster Systems Management
  • IBM Support Assistant Forum
  • Performance Tools Forum
  • Virtualization Forum
  • More AIX and UNIX Forums
  
  

About the author

Noah Gift is the co-author of "Python For Unix and Linux" by O'Reilly. He is an author, speaker, consultant, and community leader, writing for publications such as IBM developerWorks, Red Hat Magazine, O'Reilly, and MacTech. His consulting company's website is www.giftcs.com, and his personal website is www.noahgift.com. Noah is also the current organizer for www.pyatl.org, which is the Python User Group for Atlanta, GA. He has a Master's degree in CIS from Cal State Los Angeles, B.S. in Nutritional Science from Cal Poly San Luis Obispo, is an Apple and LPI certified SysAdmin, and has worked at companies such as, Caltech, Disney Feature Animation, Sony Imageworks, and Turner Studios. In his free time he enjoys spending time with his wife Leah, and their son Liam, playing the piano, and exercising religiously.

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Writing an pluggable command-line tool
• Simple plug-in architecture explained
• A look at a plug-in
• Download
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers