Oh, no! Not another "theoretical" discussion! Keep reading, though -- this discussion of metaprogramming theory can help you separate fact from fiction in the sometimes wild marketing claims of various metaprogramming tools.
This article explores the theoretical limits of metaprogramming and the Object Management Group's (OMG) Model Driven Architecture (MDA). This architecture separates business and application logic from the underlying technology platform. Learn about the problems that the MDA can solve and see an example of a textual system that uses MDA.
The theoretical limits of metaprogramming
Knowing the possibilities for the future of metaprogramming and the inherent limits of metaprogramming can help you choose the right tools and ask vendors the right questions.
Graphical and textual metaprogramming
Parts 1 and 2 in this series focused on textual metaprogramming systems; however, many metaprogramming systems are graphical in nature. Let's begin by examining a program represented both textually and graphically and see what the similarities and differences are. This example uses regular expressions.
Regular expressions are basically a small sub-language built for text processing, used to match and replace strings of characters. Learn more about regular expressions in developerWorks tutorials (see Resources).
Listing 1 a small regular expression in Perl that substitutes every occurrence of <h1> with
<h2> in an HTML string:
Listing 1. Perl regular expression for substituting HTML tags
$html_str =~ s!<(/?)h1>!<${1}h2>!g;
|
The language for doing a regular expression is not Perl; it is a domain-specific language that Perl happens to have a hook into. However, the exact syntax of the regular expression is not that important; regular expressions can be written in many ways.
Basically, whether you are communicating to the computer in a full-featured programming language or a small domain-specific language, you need a way to communicate to the compiler or interpreter the symbols of that language. You can do by typing code, but you could also express the above regular expression like this:
Figure 1. A Perl regular expression as a diagram
This works not only for domain-specific languages, but also for general programming languages. Because compilers operate on symbols, you can express symbols textually or graphically. Reasons for text expressions include:
- Typing is often faster than having to position graphs.
- More training and tools are geared toward text-based programming systems.
- Text forms are often easier to think about and manipulate programmatically.
In fact, for regular expressions, entering the code graphically is indeed more difficult. In the two representations of the regular expression, each graph has the same amount of programming information. Thus, graphical representations of programming concepts do not reduce the amount of programming involved.
If you have used visual tools for programming, your experience may differ, due to the benefits of domain-specific languages, not specifically the graphical approach. Let's look at an example from Windows® programming, a Win32 resource script for building dialog boxes:
Listing 2. Example Win32 resource script
IDD_TEST DIALOG DISCARDABLE 0, 0, 200, 200
STYLE DS_MODALFRAME | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CAPTION "Hello There!"
FONT 8, "MS Sans Serif"
BEGIN
DEFPUSHBUTTON "OK",IDOK,108,8,50,14
END
|
The script in Listing 2 requires you to "put in" the same amount of information as if you had used a graphical tool to build the dialog box and position it. The difference is that it is much easier to select widgets and position them with a mouse than it is to type out their names and then type in their coordinates. The amount of information you convey to the computer is the same in both cases.
So why does the operation require less programming than coding it by hand in C? The restricted, domain-specific sub-language requires less programming, not the visual or textual nature of the tool. This suggests the following conclusions:
- Domain-specific languages may reduce the amount of programming involved.
- Some domain-specific languages are easier to enter textually, while others are easier to enter graphically.
- Whether a program in a domain-specific language is entered textually or graphically, the same amount of information flows from the programmer to the computer.
- Because graphical languages still must provide symbols to the computer for compilation or interpretation, graphical specifications are still be required to follow formal language parameters. Such languages cannot be ad-hoc any more than computer languages can be. Graphical representations have syntax and semantics that are just as formalized as those of textual languages.
However, you can add more non-programmatic information to graphical representations. For example, you can:
- Spatially arrange different components to make more sense to other people in a graphical environment.
- Add graphs, drawings, and other graphical explanatory elements and visual cues directly to the source code to help explain a process.
- Assign code to regions that are hidden or exposed, depending on the detail of the view (for example, you can hide error-checking code when you want to analyze just the algorithm).
Graphical models take programming to a new level, not because less information is passed from the programmer to the computer, but because more information can be passed from programmer to programmer in a more natural way.
Metaprogramming, programmers, and domain experts
Now that we've established that it is the domain-specific sub-language that reduced the amount of programming involved, not necessarily the graphical nature, can we ever reach a point where programming is done completely by domain experts? In other words, can we skip the actual programmers to do the work, and let the domain experts simply use metaprogramming tools to design and build their entire applications?
To answer this question, think about what programmers do for a programming project other than write code. While the domain expert may try to specify all possible scenarios for a particular transaction, some may be missed. For example, let's say you created an application that automatically filled in the city for a given ZIP code. The person who requested the feature may not be aware that some ZIP codes lie in multiple cities. What happens then? Is the user prompted? Is one automatically chosen? These underspecified corners of programming are called edge cases. In my own experience, the most important role of the programmer/analyst is to identify all of the edge cases that the domain expert has not thought of. Programmers tend to think about issues of process more methodically and therefore are often in the best position to help the domain expert identify and expound upon all of the edge cases.
Domain experts, left to their own devices, will often leave large areas of the system underspecified no matter how hard they try not to. Performing ad-hoc solutions is usually just a part of doing business, and the need to establish absolute specifications just isn't in their realm of everyday thinking.
The second thing that programmers do is understand fully the syntax, semantics, abilities, and limitations of the languages that they program with. As mentioned above, graphical languages are no less subject to issues of exact semantics than textual languages are since they are both ultimately reduced to symbols. Therefore, even in the best, most simplified graphical environments, it is still utterly necessary to have a full understanding of the tool being used and its semantics.
Based on these two observations, I am confident that there will always be a need for programmers as long as there are programs to write. Eventually programming may be embedded well enough into the schools and curriculums that there won't be a need for programming as a separate discipline. Instead of programming being simplified to a point where programmers are not needed, public education would be expanded to a point where such simplifications would not be necessary!
The important point in all of this discussion is that the role of being a domain expert is and will continue to be separate from the role of being a programmer, though those roles are sometimes played by the same people.
Ultimately, graphical metaprograms are still programs, thus they require a programmer to implement them correctly.
Working at multiple levels of abstraction
One of the biggest problems in metaprogramming is the fact that the programmer almost always has to work with multiple levels of abstraction. The programmer not only needs to know the details of how to program the domain-specific languages and the general-purpose languages, but also:
- The details of how they are each implemented
- How to communicate between them
- What sort of impedance mismatches are between them.
Often, most of the code can use the metaprogramming tools available to accomplish the task. But the hardest part comes when they have to be mixed. For example, if the metaprogramming system auto-generates a method within a class, but later that method must be tweaked, how do you implement something like that?
In some instances and with some tools, modifying the generated code immediately removes the program from the purview of the metaprogramming system. The metaprogram is no longer of any use because the change must be made at a lower level. There are some solutions to this problem, but ultimately it is very difficult if not impossible to solve completely.
The best systems are those that allow you to work at multiple levels of abstraction within the metaprogramming system itself. For example, Part 2 of this series discussed how the Scheme macro system allows an easy mixing of macro and non-macro code. Ideally you need to work at both a high and a low level from within the metaprogramming tool itself and simply to mark for the tool where you need to transition to a lower level.
Now let's look at the OMG's Model Driven Architecture and the problems it can help you solve.
A high-level view of the Model Driven Architecture
Now that you have a better understanding of what is and isn't possible with metaprogramming tools, let's turn to the concept of enterprise metaprogramming and see how it is both similar to and different from other metaprogramming practices.
Enterprise metaprogramming ultimately is just metaprogramming. What differentiates it are these two defining qualities:
- The nature of the problems they try to solve
- The cardinality of the relationship between source and destination languages
The first difference between traditional metaprogramming and enterprise metaprogramming are the problems each tries to solve. Traditionally, metaprogramming has been mostly about these things:
- Reducing the software-development release cycle length
- Forcing programs to follow system constraints
- Making the programming language follow the domain model more closely
Enterprise metaprogramming involves these as well, but also adds other considerations. First of all, one problem has always been that models are often developed at the beginning of a project and then become irrelevant as the project progresses. The realities of coding mean that the model is not very accurate, and the act of changing the model continually only delays the project.
It seems almost wasteful to design a model at the beginning and then not use it. It also seems wasteful to develop a model once in a charting program and then redevelop it later as code. Therefore, the Model Driven Architecture aims to make the models developed at the beginning of the project part of the code itself, so that it stays current and is not redundant to the actually coding process.
Another problem is a related one, that of communication. Code is very difficult for a non-programmer to read. However, if code were written in a graphical manner, then it would be much easier to discuss coding matters with non-coders. This would actually make the code more difficult to develop, though, since aesthetics would have to be included in the coding, but ultimately it would make the communication between the programmer and the domain experts much more efficient since the domain expert could comment on the code itself, represented in a graphical, domain-friendly nature.
Another problem is portability. Businesses often use very different tools. This is not necessarily because of a lack of technological forethought, but is simply a reality of business since:
- Many businesses purchase other businesses, and therefore their ITs must be combined.
- Businesses have so many small needed that it is impossible for an IT department to come up with a cohesive solution to all needs.
Therefore, it should simply be assumed that in a large organization there will be a diverse set of technologies that must be brought together. Because of this, the metaprogramming system should be customizable to produce output for a wide variety of systems.
The final problem is maintenance. Ultimately, systems are not just built; they must be maintained. Because the model is tied directly into the code itself, maintenance becomes easier. Programmers can communicate to future programmers graphically and use not only text, but also graphics and charts to explain their programs. In fact, in many cases the graphs and charts will be the program!
Platform-specific and -independent models
The core concept of the Model Driven Architecture -- which can solve or alleviate the problems mentioned above -- is the platform-independent model. Essentially, the MDA has three tiers:
- The code. Ultimately, all of our models will become code. This is the bottom-most tier of the MDA.
- The platform-specific model, or PSM. The metaprogramming systems described in the previous articles in this series are essentially platform-specific models. Although the models exist at a higher level than the programming languages themselves, they are ultimately directly tied to them. The translation and relationship between the platform-specific model and the code are usually fairly obvious. This model is often abbreviated as PSM.
- The platform-independent model, or PIM. This makes enterprise metaprogramming unique. PIMs, as you may have guessed, get translated into PSMs. What makes them different is that the PIM is usually translated into multiple PSMs and is usually not tightly tied to any destination PSM or language.
Having a PIM that can be translated into multiple PSMs is useful for two reasons:
- Using the same program for two different platforms -- because of heterogeneous systems or the need to transition from a legacy system to a newer system
- Having a single descriptive model translate into multiple aspects of a system
The second reason is by far the more important of the two. For example, let's say that you have a data model that you want to use in a C++ database program. With a platform-independent model, you can write a single source file and have it translate into multiple aspects of the final system. For example, the data model could translate into:
- A skeleton C++ class, including operations for serializing the class to the database
- An SQL script for creating the database on which the data model is based
- A simple user interface for data loading
So, from a single platform-independent model, you would get three separate platform-specific models. Note that the difference between a platform-specific model and code is merely a difference of degree (think of a programming language as a platform-specific model for its object code). The important idea is that of translating a single source file into multiple platform-specific ones, each covering a different aspect of the system.
Coding in the Model Driven Architecture requires an ability to program on all three levels based on the task at hand, and perhaps even an ability to write translators between source models and destination models:
- Some parts of an application can be written as a PIM in which most of the legwork is done by the metaprogramming system.
- Other parts must be done as a PSM in which the features of the model are heavily tied to the purpose and abilities of the language.
- And some things can be written in code, as usual.
The benefits to the Model Driven Architecture are that:
- The models for the system have continued usefulness for the life of the software.
- Much of the redundancy that is common in applications with multiple aspects is eliminated.
- Communication between the programmer, other programmers, and domain experts is enhanced.
Example: Building code from Dia diagrams
One of the most interesting things about the MDA is that while there are specifications involved, it is at its core mostly an idea, a vision, and a process. A tool is an MDA tool if it implements some or all of the vision of the MDA. Of course, being an OMG standard, UML (the Unified Modeling Language) is the preferred modeling language, but the vision of MDA greatly supersedes the technologies that it uses.
For our example, we are going to create an MDA tool that takes class diagrams from the graphing program Dia and produces both SQL code for database tables and C++ code for simple C++ classes. This doesn't live up to the full expectation of MDA tools, but it gives a good example of generating multiple platform-specific models (well, we're generating actual code in this case) from a single platform-independent model.
To create the diagram, fire up Dia and use its UML graphing capabilities. When you start Dia, it looks like this:
Figure 2. The Dia program
There is a menu of object types on the main toolbox. Choose the UML object set. The first object is a UML class. Select it and then add it to your diagram. Your diagram will look something like this:
Figure 3. A UML class in a new drawing
If you double-click the new class glyph, you see a screen like this:
Figure 4. Basic class information
This dialog box lets you set up the class information. Since this example is limited, the only field on this tab that is of use is the Name field. For our example, we're calling this the Person class. Next, add attributes, so go to the Attributes tab. Here we will add three fields: id, name, and address of types int, text, and text, respectively. Your screen should look like this:
Figure 5. Editing class attributes
Click OK on the dialog box to finish. You can add additional classes if you want. Once you are done, save the diagram using this dialog box:
Figure 6. Dia's save dialog box
Be sure when saving that the "compress diagram files" box is not checked.
A quick overview of the Dia file format
A Dia file is simply an XML file using Dia's own schema and namespace, either compressed or uncompressed (our program only deals with uncompressed files). The layout is simple. Here are some portions of the file we just made:
Listing 3. Example Dia file
<?xml version="1.0" encoding="UTF-8"?>
<dia:diagram xmlns:dia="http://www.lysator.liu.se/~alla/dia/">
<dia:diagramdata>
<dia:attribute name="background">
<dia:color val="#ffffff"/>
</dia:attribute>
...
...
</dia:diagramdata>
<dia:layer name="Background" visible="true">
<dia:object type="UML - Class" version="0" id="O0">
<dia:attribute name="obj_pos">
<dia:point val="11.6,6.35"/>
</dia:attribute>
...
...
<dia:attribute name="name">
<dia:string>#Person#</dia:string>
</dia:attribute>
...
<dia:attribute name="attributes">
<dia:composite type="umlattribute">
<dia:attribute name="name">
<dia:string>#id#</dia:string>
</dia:attribute>
<dia:attribute name="type">
<dia:string>#int#</dia:string>
</dia:attribute>
<dia:attribute name="value">
<dia:string>##</dia:string>
</dia:attribute>
...
...
</dia:composite>
</dia:attribute>
<dia:attribute name="operations"/>
<dia:attribute name="template">
<dia:boolean val="false"/>
</dia:attribute>
<dia:attribute name="templates"/>
</dia:object>
</dia:layer>
</dia:diagram>
|
This file has a lot of information in it, but only a small part of it
is useful to our purposes. Notice the
XML trees that begin with something like <dia:object type="UML - Class" version="0" id="O0">.
And from these, notice the name of the
class and the name and type of each attribute.
The name of the class is in an attribute named name. The attributes are all in a subtree called attributes with each attribute being a umlatrribute with corresponding name and type fields. Therefore, within the sea of XML tags and data, sort through them and grab the portions of interest.
The MDA implementation is a simple SAX parser written in Scheme using the SSAX module (for more information about Scheme, SAX, and the SSAX module, see Resources). The parser reads from one .dia file and writes to an SQL file and a C++ source file. For each class it creates a C++ class and a database table.
To make the example as simple as possible, only attributes are supported, not operations, and no database loading/saving functions are implemented. However, adding those features are only incremental steps away from the parser shown in Listing 4.
Here is the code for the MDA application, written for use with Chicken Scheme (other schemes would probably require only slight modification to the line that loads in the SSAX module):
Listing 4. Full program listing for MDA application
;;Load the Scheme SAX parser
(require-extension ssax)
;;Dia's string values are surrounded with '#'s. This
;;removes the '#'s.
(define (munge-dia-value val)
(let (
(str-len (string-length val)))
(if (<= str-len 2)
""
(substring val 1 (- str-len 1)))))
;;The SAX parser function
(define (dia-mda-parser input-port cpp-out-port sql-out-port)
(let (
;;State variables -- these would probably be more correct
;; to keep with the seed, but putting them
;; here makes them easier to manage
;this is used for proper handling of comma-separated lists
(first #t)
;these are used to store the type and name of an attribute
;before its read to be printed out
(attr-type "")
(attr-name ""))
;;Write the C++ header
(display "#include <string>" cpp-out-port)(newline cpp-out-port)
(display "using namespace std;" cpp-out-port)(newline cpp-out-port)
(display "typedef string text;" cpp-out-port)(newline cpp-out-port)
;;SSAX Macro to build the parser
((SSAX:make-parser
;;on-element function
NEW-LEVEL-SEED
(lambda (elem-gi attrs namespaces expected-content seed)
;;The element name is the cdr of the name list
(define element (cdr elem-gi))
;;common attributes
(define name-attr (assq 'name attrs))
(define type-attr (assq 'type attrs))
;;Giant state switch - determine what element and state
;; we are in and what state we should
;; change to, if any
;;
;;Note that the return value of this function (and the others)
;;becomes the new "seed" value, so we are using it to record
;;the current state. More information about the inner
;;workings of SSAX are in the references section.
(cond
;Check to begin a top-level class
((and (eq? seed 'top) (eq? element 'object) (equal? (cdr type-attr) "UML - Class"))
(set! first #t) ;reset the "comma" state
'object)
;Check whether to be looking for the class name
((and (eq? seed 'object) (eq? element 'attribute) (equal? (cdr name-attr) "name"))
'class-name)
;Check whether we are ready to read the class name
((and (eq? seed 'class-name) (eq? element 'string))
'read-class-name)
;Check to see if we should switch to attribute mode
((and (eq? seed 'object) (eq? element 'attribute) (equal? (cdr name-attr)
"attributes")) 'class-attrs)
;Check to see if we should start looking for attribute info
((and (eq? seed 'class-attrs) (eq? element 'composite) (equal? (cdr type-attr)
"umlattribute")) 'class-attribute)
;Check whether we should be looking for the attribute name
((and (eq? seed 'class-attribute) (eq? element 'attribute) (equal? (cdr name-attr)
"name")) 'class-attribute-name)
;Check whether we should be reading the attribute name
((and (eq? seed 'class-attribute-name) (eq? element 'string))
'read-class-attribute-name)
;Check whether we should be looking for the attribute type
((and (eq? seed 'class-attribute) (eq? element 'attribute) (equal? (cdr name-attr)
"type")) 'class-attribute-type)
;Check whether we should be reading the attribute type
((and (eq? seed 'class-attribute-type) (eq? element 'string))
'read-class-attribute-type)
(else seed)))
;;after-element function
FINISH-ELEMENT
(lambda (elem-gi attrs namespaces parent-seed seed)
;;often-used values
(define element (cdr elem-gi))
(define type-attr (assq 'type attrs))
(cond
;After finishing the umlattribute element, write out the attribute information
((and (eq? element 'composite) (equal? (cdr type-attr) "umlattribute"))
(if first
(set! first #f)
(display ", " sql-out-port))
(display (string-append attr-name " " attr-type) sql-out-port)
(display (string-append "\t" attr-type " " attr-name ";") cpp-out-port)
(newline cpp-out-port))
;After finishing the UML Class object element, write the closing characters
((and (eq? element 'object) (equal? (cdr type-attr) "UML - Class"))
(display ");" sql-out-port)(newline sql-out-port)
(display "};" cpp-out-port)(newline cpp-out-port))
(else #f))
;Restore parent information
parent-seed)
;;character data function
CHAR-DATA-HANDLER
(lambda (s1 s2 seed)
(cond
;Read the class name and write out the appropriate statements
((eq? seed 'read-class-name)
(display "class " cpp-out-port)
(display (munge-dia-value s1) cpp-out-port)
(display " {" cpp-out-port) (newline cpp-out-port)
(display "public:" cpp-out-port) (newline cpp-out-port)
(display "create table " sql-out-port)
(display (munge-dia-value s1) sql-out-port)
(display " (" sql-out-port))
;Read and save the attribute name
((eq? seed 'read-class-attribute-name)
(set! attr-name (munge-dia-value s1)))
;Read and save the attribute type
((eq? seed 'read-class-attribute-type)
(set! attr-type (munge-dia-value s1)))
(else #f))
seed)
) input-port 'top)))
;;;Main Program;;;
(let (
;;Open Files;;
(tables-in (open-input-file "tables.dia"))
(cpp-out (open-output-file "tables.cpp"))
(sql-out (open-output-file "tables.sql")))
;;Process Data;;
(dia-mda-parser tables-in cpp-out sql-out)
;;Close Files;;
(close-input-port tables-in)
(close-output-port cpp-out)
(close-output-port sql-out))
|
For simplicity, the input and output file names are
hardcoded, so be sure to save the Dia file in the same directory with
the name tables.dia. Save the program as mda.scm. To run the program, simply do csi mda.scm. With the data we put into Dia, it produces two output files. The SQL output looks like this:
Listing 5. SQL output from the MDA program
create table Person (id int, name text, address text);
|
The C++ file looks like this:
Listing 6. C++ output from the MDA program
#include <string>
using namespace std;
typedef string text;
class Person {
public:
int id;
text name;
text address;
};
|
So, for under 200 lines of code, you can create diagrams instead of writing SQL and C++. Of course, it's even more fun when someone else writes the conversion program.
The Model Driven Architecture is a powerful idea. It is rooted in a strong tradition of metaprogramming and yet goes beyond traditional methods using its concept of platform-independent models.
Learn
-
For more information about regular expressions, see the Web's premier site on the subject.
-
The OMG has a section of its site dedicated to the MDA.
-
MDA Explained is an excellent book covering the basics of the MDA.
-
The SAX Project is a great resource site for SAX, the Simple API for XML.
-
The official SSAX page has lots of information about SSAX and other aspects of manipulating XML in Scheme.
-
"XML Matters: Investigating SXML and SSAX" (developerWorks, October 2003) is a wonderful guide to manipulating XML in the Scheme programming language.
-
Chicken Scheme has its own SSAX package/parser. Learn more about Chicken, a compiler for the Scheme programming language.
-
How to Design Programs is the best resource for Scheme beginners.
-
In the developerWorks Linux zone, find more resources for Linux developers.
-
Stay current with developerWorks technical events and Webcasts.
Get products and technologies
-
The Design Pattern Toolkit (alphaWorks, March 2005) is an Eclipse-based toolkit for developing MDA transformations.
-
For a true open-source graphical MDA tool, check out AndroMDA.
-
Eclipse has an MDA tool called the Eclipse Modeling Framework.
-
Order the SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
-
With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.
Discuss
-
Check out developerWorks
blogs and get involved in the developerWorks community.
Jonathan Bartlett is the author of the book Programming from the Ground Up, an introduction to programming using Linux assembly language. He is the lead developer at New Medio, developing Web, video, kiosk, and desktop applications for clients.
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
