Today at the IMPACT 2008 conference, IBM made an announcement covered by the press about its new Web 2.0 Portal Software capabilities. That announcement included the following content from the Lotus Forms team:
"With diverse data sources and inefficient, time consuming processes needed to turn data into useful information, more companies are turning to web-based forms. In response, IBM announced today a new lightweight entry-level product code-named TotalForms which the company plans to ship in beta in June."
"TotalForms is an easy-to-use version of IBM's Lotus Forms software that will enable nontechnical people to quickly create, publish and route electronic forms submitted via the web. It can be used for a variety of tasks including customer satisfaction surveys, job applications and product orders. Based on an open and scalable nonproprietary software platform and Web 2.0 technology, TotalForms will integrate with IBM WebSphere Portal to provide one familiar interface for customers."
So you know, Total Forms, will offer not just web forms based on XForms, but also a web-based design experience for forms. More importantly, though, the value of Total Forms is that it provides a server-side repository for completed form data as well as listing and reporting functions. This means that users will have a point-and-click gestural design experience and run-time capability for the full, end-to-end Web 2.0 application. This puts the computing power into the hands of the people who have the problems that computing power can solve.
Modified by John M. Boyer
Explanatory power is a bit of a loaded term. I believe that we can come to a good understanding of what it is and how it related to machine learning by comparing and contrasting linear regression with neural nets.
The IBM Analytics Education Series has a good introductory analytics presentation that includes brief descriptions of neural nets and linear regression. It's a good video worth your viewing, though it does make one point with which I don't agree.
The speaker says that a challenge with neural nets in business applications is that they are black box, meaning that you can understand the inputs and the outputs but not really how it is deriving the outputs. Later, the speaker says that linear regression is a preferred technique because it has a very strong predictive and explanatory power.
It's not really true that linear regression has more explanatory power than neural nets. Rather, it is easier to understand the problems and the answers that can be solved by linear regression. By comparison, neural nets tend to be used to provide cognitive computing power to harder problems than linear regression can solve.
To put this another way, when you use linear regression, you actually begin by assuming linearity of the relation you want to predict. As the speaker points out, you can also make a non-linear assumption, and you can accommodate this using a data transformation, for example. But the high order bit is that you are asked to assume the data relationship, and that assumption is what is giving you the illusion of explanatory power. You can explain that the data follows a line, but this is due to your own assumption. Note that an important aspect of completing a linear regression model is determining the R2 or goodness of fit of the model. This is the part where you make sure that your assumption of linearity is valid. And if the assumption is invalid, then the model has no predictive value, so it does not matter that you can explain how it operates.
Under the interpretation that explanatory power is akin to predictive power, it turns out that neural nets have greater predictive power because they can produce results for a wider array of applications than linear regression can. There a neat table that relates the cognitive power of a neural net to the number of hidden layers. From the table, you can see that when a relationship actually is linear, a neural net can solve it without even using any hidden layers of neurons. When one or two hidden layers of neurons is present, neural nets transcend the capabilities of linear regression, in part because they do not require you to make any assumption about what the data relationship actually is.
And that's where the confusions comes in. The linear regression model requires you to assume linearity and so you know at least what geometric shape the relationship looks like. The neural net requires no such assumption, but nor does the trained neural network give you any hint at what the relationship is. The lack of knowing the relationship is confused for having less explanatory power.
But if you look at this a bit more abstractly, the trained linear regression model has the same exact problem of not providing any additional insight. A neural net is really just a pile of numbers giving constant weights to the neural connections that can convert inputs to outputs. Similarly, a linear regression model is just a pile of numbers that give constant weights to inputs to be linearly combined into an output. Sure you know the data relationship, but that's because you assumed it. The actual linear regression model gives you no insight into why one dimension has a large slope constant where another has a small slope.
An analogy I like to use is that the value of the neural net is not diminished by our inability to explain how it is that the little gray cells which implement our personal neural nets can produce the cognitive results that they do, and who among us would prefer to have cognitive powers defined by linear regression instead?
In terms of explanatory power, our biological neural nets perform an additional key function that we have not hitherto been able to achieve with artificial neural nets. We are able to construct additional information in the output that reveals causal relationships, or insights into the reasons for the phenomena it predicts. Put simply: we say why something is true. We provide a rationale. This is an aspect of explanatory power that, when achieved, dramatically increases the value and utility of any cognitive analytic. Theorem provers and Prolog programs have been able to do this for the applications to which they apply. In the area of unstructured information processing and data mining, you can see a demo of this concept in Watson Paths.
Someone recently asked me to explain why the XForms way of binding form controls to XML data with XPath was better than, say, using XSLT templates to match data with XPaths and output an XHTML form. Seemed a fair question.
An XSLT describes a transformation step that happens entirely before the user gets involved, so all the limitations of current XHTML forms still applies to the result of the transformation.
In comparison, XForms defines an intelligent, interactive layer that governs the fill experience for the data. Applying an XSLT to data does not, in and of itself, describes what happens as the user interacts with the data to create new content, so the author of the XSLT is left to encode the old script and HTML devices into the XSLT.
XForms standardizes the following:
- schema-based validation as you fill out the form
- dynamic validation constraints applied as you fill out the form
- calculated dependent values (like tax and total values on a purchase order)
- accessibility text
- access to the web services to let an SOA enrich the fill experience
- preservation of data hierarchy in the UI
- access to template-based UI repetition responsive to data mutations
- user interface switching (presenting panels of controls conditionally or in sequence)
- dynamic user interface properties (readonly, visibility/display)
- XML data submissions to the Web 2.0 server tier
Not only does XForms standardize these things, but it standardizes how they work together.
To further drive home the point, note that XSLT transformation could actually be used as a sympathetic technology with XForms. You could, for example, write an XSLT that transforms an XML Schema into data+XForms rather than data+DHTML. Then, you'd still get all of the above value adds because XForms is a standardized interactivity layer that describes what happens after the XSLT generates the initial rendition.
I did a customer site visit out in Washingto DC earlier this week (I live on the west coast of Canada). On the plane next to me there was a guy who had a book to read with a title that was something like "Three men, a raft and the most dangerous river". It appeared to be about three guys who navigated the Amazon in a raft.
Perhaps it was just my mood at the time, but it struck me as a little ironic to read a book about it. It's just so much the kind of thing you have to do.
Well, while we're in the land of giant leaps... forms applications are like that too. You can read, but you have to *do* too.
For my own case, the Workplace Forms beta download is accompanied by a reasonably substantial sample form that I wrote to help illustrate for you a number of the important features that are available, such as a wizard-like or guided-interview modality attached to the front end of a "traditional" form.
The internal markup is highly commented and laid out so that you can really get an understanding at the pointy bracket level even if you only have a text editor for exploration (though the beta also includes our eclipse-based designer product).
So, speaking of doing (and also to fulfill my promise to talk about the nitty gritty), it seemed a good idea to share with you one of the interesting things I learned from this form.
The typical form is designed to be used in a particular business context. In general, you will not find that a form (of the size and scope that Workplace Forms targets) is easily purposed to serve multiple locales or business contexts. The "mortgage preapproval" sample form was created to give people in many locales some idea of what the product set can do, but it's not really the intent to address all locale-specific peculiarities. For example, a more realistic form in Russian would have the formulae and inputs adjusted to collect monthly salary information rather than yearly information. And in some locales, it isn't even appropriate to collect interest on a loan, nor is it appropriate to accept a service like a loan for free, so interest is something that is informally negotiated rather than being driven by an exact compound interest formula.
Anyway, as I said we just wanted to provide a form that would give a good idea, so we ended up with the task of translating the form to many languages.
Something I didn't expect happened because I put some localizable strings into certain XML attributes, like an XForms calculate or the value attribute of an xforms:setvalue. The word "Electricity" becomes "L'Electricitie" (pardon my omission of accented characters).
The addition of the apostrophe is problematic when in an attribute because the attribute contains an XPath expression. So you have double quotes around the attribute value, so that single quotes are used around the literal value, and the apostrophe conflicts with that. Moreover, the apos character entity is of no help because that is an encoding consumed by the XML parser, not the XPath parser, so it is converted from a character entity to the literal character by the XML parser so that the XPath parser still receives the offending character.
So, it seems forms have to be designed so that strings intended for humans don't appear in XML attributes that contain XPath expressions. My apologies to my European friends, who no doubt know this already. As I said, pardon my French.
I've recently realized that there are a few milestones to celebrate all at once here: This blog turned 5, my W3C XForms group age is 10, and this is the 100th exciting episode of the IBM Forms dW blog. I took a look back at what you, the readers, seem to like best, and those 2000-3000 hit entries are at the intersection of strong technical content and the confluence of open standards. How serendipitous!
The power and value of the XFDL markup language underlying IBM Forms comes from the way it brings together all the features of open standards needed to design sophisticated solutions, including XML, XML Schema, XForms, XML Signatures, and XPath. In this entry, I'd like to continue the schema driven design story by talking to you about the information architect's half of the equation. Specifically, we'll cover the XML Schema constructs that the IBM Forms Designer converts into various user interface controls like XFDL popups skinning XForms select1 elements and XFDL tables skinning XForms repeat elements. If you haven't yet watched the short demo videos mentioned in the prior blog entry, take a moment to do so now because I'd like for us to dig into parts of the "Medical Preapproval" schema used in that demo. The top-level element definitions for the schema look like this:
<element name="medicalPreapproval" type="health:medicalPreapprovalType"/>
<element name="patient" type="health:patientType"/>
<element name="contactInfo" type="health:contactInfoType">
<element name="details" type="health:detailsType"/>
In this sample, the data needed fora medical insurance preapproval consists of some basic patient data like name and date of birth, contact information like email and address, and details of the medical procedures required. When the IBM Forms Designer is provided with this schema, it can generate an XML data conformant to the schema in an XForms instance element, and then the form author can drag and drop the XML elements in the instance view onto the Design canvas. For example, if the form author drags and drops the "patient" element, then the IBM Forms Designer creates an XFDL pane that skins an XForms group element. Within the group, various XFDL items like text input fields and calendar pickers are created and mapped to the child elements of the "patient" element so that the textual content can be collected by the form. So you can get an idea of how to write XML schema for structured content, here's a part of the patientType definition:
<element name="lastName" type="xsd:string"/>
<element name="firstName" type="xsd:string"/>
<element name="SSN" type="health:serialNumberType"/>
<element name="dateOfBirth" type="xsd:date"/>
At design time, text entry fields are created for these data items, and any specialized schema rules, such as the regex pattern definition for the SSN, are applied at XForms run-time. In case you're curious, here's how you'd set up a regex pattern, in this case a very simple one that takes 9 digits but also allows an empty string:
For data types like the xsd:date (above) and xsd:boolean, you will get an alternative form control, such as calendar picker or checkbox. The IBM Forms Designer will even generate a calendar picker if you have a type derived from xsd:date. For example, you may want your schema to allow a date or an empty value. So, if you change the type of the dateOfBirth element above to health:dateOrEmpty, then you can use the declaration below to achieve that effect:
<element name="dateOfBirth" type="health:dateOrEmpty"/>
By default, the label text generated for each element is based on the element name. Camel-casing is used to determine when spaces should be added to the label. For example, the default label generated for the "patient" group is "Patient", and the default label for the "lastName" element is "Last Name". However, the information architect is provided greater control over the labels in the schema. In the example above, an appinfo annotation is used to define the desired label of "Contact Information:" for the contactInfo element. The same appinfo annotation can be used to control the labels of the UI controls bound to leaf (text content) nodes. For example, below is a definition that could go into the patientType definition, but it provides an alternative label for an element that has a name that is perhaps medically accurate but perhaps less desirable:
<element name="sex" type="health:genders">
Continuing with this example, the obvious choices of Male and Female would be defined in a schema enumeration, like this:
An enumeration like this in the schema causes an XFDL "popup" item (a dropdown menu) to be generated, along with an xforms:select1 element. The dropdown menu entries are Male and Female, and the user's choice at run-time places the word Male or Female in the "sex" element. But maybe you'd prefer to have the data be based on codes like M and F, but still show menu entry labels like Male and Female. Again, the appinfo annotation comes to the rescue:
I've used Gender in these examples for brevity, but you can use enumerations to describe other lists, such as for States and Provinces. The IBM Forms Designer will generate an XFDL popup item by default, but the Designer also lets you right-click convert the popup to other user interface items that can legally skin an xforms:select1, such as a radiogroup, checkgroup or list box.
Also, sometimes what you need is a combobox, so that the user can access a dropdown menu but also have the option to type an open-ended answer. For example, the open-ended response for a gender might provide medically relevant information. The information architect can control this in the XML schema by creating an open-ended enumeration, i.e. an enumeration unioned with a string:
Last but not least, we have the mechanism that signifies the need to create an XFDL table and xforms:repeat. Here's an excerpt of the health:detailsType definition:
<element name="reason" type="xsd:string"/>
<element name="recommendedProcedures" type="health:recommendedProceduresType"/>
<element name="procedure" minOccurs="0" maxOccurs="unbounded">
<element name="name" type="xsd:string"/>
<element name="qty" type="health:positiveIntegerOrEmpty">
<element name="price" type="health:decimalOrEmpty"/>
<element name="lineValue" type="health:decimalOrEmpty"/>
<element name="total" type="health:decimalOrEmpty"/>
Within the recommendedProcedures element, the schema indicates that a "procedure" element can appear more than once (maxOccurs is unbounded in this example, but any value greater than 1 will also work). The IBM Forms Designer responds with the XFDL table editor. It reads the child elements in the sequence to determine the default set of table columns. Notice the use of the appinfo annotation to control the column header label text. When the form author drags and drops the details element, XFDL fields sking xforms:input elements are created for elements like reason and total, but the table editor also produces an XFDL table and xforms:repeat whose nodeset attribute contains the XPath needed to bind to however many procedure elements are present in the data at form run-time. The table editor also generates XFDL buttons containing xforms:trigger elements that provide the "add row" and "delete row" capabilities using xforms:insert, xforms:delete and xforms:setfocus actions.
Now hopefully you have a more complete picture of how IBM Forms concentrates the combined value of all these standards-- XML schema, XPath, XForms, XML-- into one solution creation machine. Well, that may have been a tad long, but we're celebrating three milestones here :-) Besides, the length of this entry is a simple reflection of how much cool stuff you can now exploit in your XML Schemas to help your form authors create the forms that feed the right XML to your backend systems.
On a recent ad for a movie or TV program, I heard an irreverent but nonetheless funny line about there being no team in I. Good for a laugh, but it did remind me of that more deeply satisfying idiomatic saying: There is no 'I' in team. As well, it reminded me that I wanted to say a thing or two more about a recent interview I did with Jonathan Erickson for the Dr. Dobb's Journal Developer's Diary.
The purpose of the diary is to provide snapshots in the lives of real developers at various stages of their careers to find out what they do, what makes life harder for them, and what makes their job easier. I was honored to be selected, and Dr. Dobb's did a good job of relating what I said. If you read the print version, you'll see that they said 'Turing incomplete' instead of 'Turing complete' but that was immediately corrected in the online version. But the main thing I wanted to talk to you about was the question of what makes my job easier because I talked about two things, but they only had space for one.
In response to the question of what makes my life easier, I would say that I see my job as a senior technical leader being comprised of both challenging problems to work on as well as interesting people to work with. On the problem-solving side, I made mention of the important role that algorithm analysis has played in not only effective problem-solving, but in seeing where the problems really are and how to either solve or sidestep them as needed. However, I'd like to at least make public my thoughts on the people side of my role, too.
Specifically, I would like to call your attention to the Workplace Forms team and the XForms community. Whether I am directly participant in a collaboration or simply observing the members of either group, I have to say that the level of dedication you demonstrate inspires me. You, the people, certainly give meaning to my own work, and I am proud to be working with you to build Workplace Forms and XForms.
Gaining customer insight has very high business value. In the last blog, I showed you how you can gain customer insight without introducing bias by randomizing the answers to each question you ask. In a prior blog, I showed you how you can make the presentation layer be dynamically responsive to the type of question you are asking. In this blog, we'll cover how to randomly select the subset of questions to be asked of a particular user. This is important in avoiding survey fatigue and low response rates that can result from surveys that take too long to complete.
As in the prior blog entry that presented survey items of different types, suppose you have an XForms instance that includes any number of items of interest, where each <item> includes a prompting question, a type of answer, storage space for an answer, and so forth. The goal is to have the form template initialize itself with a small subset of these <item> elements, and once again to do so randomly, i.e. without bias, so that over a large customer population, we will get a close-to-equal response rate on all items of interest.
Let the following two XForms instances exist:
<xforms:instance id="AllItems" xmlns="">
<!-- All survey item elements -->
<xforms:instance id="ChosenItems" xmlns="">
<items random="" i="" number="3">
<!-- Initially empty, will contain "number" item elements chosen from AllItems -->
Data initialization in XForms can be done in the xforms-ready event but is more typically done in the xforms-model-construct-done event because it is more efficient to manipulate the data before the user interface is created. So, you'd randomly choose some survey items from all the items like this:
<xforms:action if="count(instance('ChosenItems')/item) = 0"
<xforms:setvalue ref="instance('ChosenItems')/@random" value="random(true())"/>
<xforms:action while="instance('ChosenItems')/@i <= instance('ChosenItems')/@number">
<xforms:setvalue ref="instance('ChosenItems')/@random" value="random(false())"/>
The if statement on the main action ensures that the random selection only happens once, when a copy of the form template is used for the first time, thereby becoming a form. The first setvalue action seeds the random number generator, and the second action initializes the loop variable. The main loop randomly chooses "number" items from AllItems and places them in the ChosenItems. Each selected item is then deleted from AllItems to avoid duplicate items.
The "number" of questionnaire items to select is shown to be 3 in the example, but it is really up to the designer(s) of the survey to set that number into the form template based on how many questions they feel their customers will answer without impeding the return rate due to survey fatigue.
Finally, rather than binding a user interface to the 'AllItems' data instance, you simply bind it to the 'ChosenItems instance instead. An XForms repeat iterates the user inteface controls within it for each of a set of nodes identified by the XPath in its nodeset attribute, like this:
<!-- XForms user interface controls, like input and select1, go here -->
Inside the repeat is where you could use simple XForms input controls for typed answers or select1 controls for choice-based answers, or this is where you could use dynamic user interface techniques to show different kinds of user interface controls based on a type of survey item.
Well, I have always looked at Workplace Forms as an eco-friendly product line since, after all, it does dramatically cut down on the use of actual paper, which cuts down on the cutting down of actual trees. But, this post is actually quite a bit more technical than that. In this case, we're going to talk a little about the the *Workplace Forms Designer 2.7* (part of the 2.7 GA release this week) and the three wizards it includes to help form authors with creating dynamic tables and the components and formulae that make them dynamic. The focus, of course, will be the new wizard in 2.7.
The first *Table Wizard* helps you to create a repeating set of user interface controls over some data. You get a chance to set up nice presentational features like column headers, column dividers and row highlighting, and you also get Add and Delete buttons for the table. The second *Row Operation Wizard* allows the form author to express a formula over two columns of a table, with the result being placed into a third column. The formula is defined on the underlying data, of course, via an XForms calculate. The wizard is ideal for setting up that "line total = quantity * unit price" formula that shows up in every purchase order. It could also be used as the starting point for setting up the XForms constructs for a more complex calculation.
Although we've made improvements to these wizards in 2.7 (and will do so again in the future), the new kid on the block is the *Column Summation Wizard*, which is ideal for setting up a summation over a column of data, e.g. for taking the subtotal of a purchase order. The XPath expression written by this wizard is very interesting and relates to the title of this post. All you do as the form author is gesture at the user interface object in the column to be summed and at the user interface object where the result will go. Here is a sample of the underlying XPath calculate that results:
This formula requests the summation of all the line "total" values on each row of a purchase order form, and if the result is not a number (NaN) for some reason such as bogus user input, the formula produces an empty string result rather than the summation value.
This is helpful in and of itself because XPath, XML Schema and other W3C technologies thend to regard an empty string as not a number. So if you try to add or multiply a data node that has no value yet (like the quantity of items desired), then NaN starts to percolate through your form unless you take special measures.
But the really special sauce is the XPath predicate shown here:
[.!='']. An XPath predicate is used to filter out nodes that have some undesirable property. In this case, we want to get rid of the empties and only sum up the data nodes that have a non-empty value. The dot means "this node", so the expression in square brackets says "this node must be not equal to the empty string".
Suppose for a moment that we didn't add this extra tidbit to ignore the empties. Suppose a user has a partially filled out purchase order form with three or four rows of data leading to a non-empty column total. If the user then adds one or more empty rows to add more data, then at that moment the column total formula would update itself to calculate the total over rows that included empties, which would result in a NaN, which the 'if' part of the expression converts to an empty string result. So the user would get treated to this odd experience of having the column total disappear each time he tries to add another row to the purchase order. The extra predicate created by the *Column Summation Wizard* says to ignore the empties so that the user will have a chance to fill in a row of data before it contributes to the column summation.
For over 15 years now, the solutions built with the software products from the IBM Lotus Forms team have been based on a simplifying system architecture depicted below. This architecture uses a sophisticated, intelligent XML document as the unit of information that flows among collaborators in a business process.
To support such high-value intelligent interaction, the document has a numberof layers as depicted below. These layers have different responsibilities. Foundationally, the document format is an XML format, which means it is based on thestandard that allows introspection of the document to inject or extract information.This also means the full power of XML signatures can be brought to bear to meet archival and security requirements, particularly non-repudiation of the transactioneffected by the business process based on the document. The XForms layer allows representation of a rich interaction layer for collection of structured data content.
Our work on Lotus Forms has been focused on the collection of structured contentneeded to drive a transaction through a business process, and this has included defining the XML vocabulary called XFDL to allow application developers to build thehigh precision user interface needed to drive interaction with end-users.
But consider for a moment the great number of situational applications that areconstantly being created and deployed around the globe that are based on the requirement to have a mix of structured data and unstructured content. In other words, consider the case ofoffice documents with high business value. We're talking purchase contracts, supplieragreements, insurance polices, RFP responses, financial arrangements, and the like. These are documents that can make effective use of multipage free-flowing unstructuredcontent and graphics interspersed with fill-in-the-blanks content for names, dates, addresses, monetary values, milestones and so forth.
These documents support an important class of live, interactive applications in which multiple collaborators work together to solve a semi-structured content problem.The OASIS Technical Committee for the Open Document Format, in their wisdom, incorporatedthe XForms model into ODF, and this delightful feat of standards reuse allows us to reap the benefits of ODF's unstructured content creation features together with the interactive structured data collection and both the data and document submission facilities of XForms 1.1 at the same time. The layered document picture looks pretty much the same, except as you can see below the on-the-glass presentationlayer is ODF, and otherwise the overall business process system architecture is the same as depicted above.
The result of these considerations is a focus on Interactive Office Documentsas a strategic means of going Beyond Office, i.e. going beyond the traditional usage patternsfor office documents that have become entrenched due to the office document limitations of the past and instead solving the real business problems that cause people to select the useof office documents in the first place. Using XForms 1.1, office documents based on ODF can be first class participants in collaborative business processes.
Information system architects, and even vertical industry standardization or government IT standardization bodies, derive immense value from XML schema definitions that describe the data structures and data types expected in valid transactions of the information system. However, these assets are focused on defining the completed transaction. In many a presentation, I've talked to potential customers about how IBM Forms documents express far more value because they are about the human interaction that takes place during the fill experience needed to produce those completed transactions.
As a bit of an aside, it's important for the technically minded reader to be familiar with the "sell" side of this equation. It is important to be able to easily justify technology adoption decisions with business owners who need to be able to understand how you will be able to better server your customers, reduce development and maintenance costs, increase competitiveness, eliminate vendor lock-in, etc.
In this blog, I'd like enumerate various benefits that you get from the standards basis of IBM Forms documents and their implementations, but I'd also like to separate the enumerations into two lists: 1) benefits above using XML Schema that you get just from using the XForms markup within an IBM Form, and 2) the benefits you get from the XFDL processor (XFDL is the XML vocabulary that IBM Forms provides as a presentation layer for XForms).
Firstly, an XForm is a clearly a superset of XML Schema since an XForm can incorporate the XML Schema if it is available and provide its validation information set to the fill experience and the submission experience. But an XForm provides many additional benefits, including the ability to:
Express data validation constraints that are based on other XML data values entered during the fill experience.
Automatically compute data values based on other input entered during the fill experience, rather than requiring users to perform the error-prone task of calculating and inputting summative results manually.
Describe the inputs, outputs and triggering mechanisms of web services to invoke, such as to use entered XML data to obtain database query results that fill other parts of the form or to invoke server-side validation or calculation logic from a business rules engine.
Control whether a data value is readonly or whether the user can enter a value, for example based on conditions related to a business process step or an access rule.
Express user interface controls that indicate which XML data values will be available to the presentation layer for input or output.
Conditionally show or hide the user interface controls (and hence their presentation) in response to conditions, such as those that may relate to a business process step or access rule.
Define a step-wise fill experience to guide a user systematically through incremental successes in entering valid data.
Provide customized help and validation error messages to help users fix data input errors when they occur.
Provide prompting label text to be associated with the presentation of each input or output control, both visually and aurally (for accessibility)
Associate a selection list with any XML data node in a way that constrains user input capability to the provided list
Define labels, help messages, validation error messages and selection lists in more than one human language within the same form, thereby ensuring citizens receive the same form logic and interaction behaviors regardless of which official language they select to request government services
Associate repeated data with a logically tabular set of user interface controls and encode the means by logical rows of the table are added or removed in response to insertions or deletions of data
While the above benefits indicate what additional behaviors and features of XML forms can be expressed above those that can be expressed by an XML Schema alone, XForms is also more interesting for standardization of a forms repository due to what it does not express. An XForm does not rigorously bind its many behavioral benefit to a specific presentation layer of the form. The intention of this language architecture was to address multimodal requirements of forms applications, e.g. rendition on a desktop, tablet, smartphone, telephone call-in voice service or instant messaging interaction. Different presentation layer implementations can address these requirements, and such implementations can even be provided by different vendors. The XForms working group also anticipated that there would be a wide array of varied technical requirements for presentation layers, and this language architecture allows XForms to be used with fundamentally different XML presentation languages that address these disparate requirements. Examples range from ODF for flowing text with fill-in-the-blanks fields to XFDL with its high-precision contract-style layout capability.
Due to the above mentioned language architecture, XForms markup does not comprise a well-formed XML document until it is incorporated into a presentation layer XML document. XFDL in the XML vocabulary used in IBM Forms to provide a presentation layer for XForms.
Of course, in a "baseline standard" version of a form, the default presentation layer associated with the XForms markup can be minimal in nature so that the only benefit is to provide a well-formed XML document to host the XForms markup. Interestingly, once you have this from XFDL, then the result is in fact an XML document, and so it can be processed by readily available XML processing tools like XSLT. These XML tools can be usd to automate creation of different versions of the baseline form that may have a richer presentation description, alternative natural language usage, or even different presentation layer markup. In addition, various consumers could use simple XML tooling to rebrand the forms.
The Extensible Forms Description Language (XFDL) in an XML vocabulary describing the presentation layer and richly interactive behaviors of modern web-based electronic forms. This XML vocabulary was first introduced to the W3C in 1998 (http://www.w3.org/TR/NOTE-XFDL
) and over the years, the versions of the language have consumed XML data-processing components of the W3C XML technology stack as they have been standardized by the W3C, especially including XML Schema, XForms, and XML Signatures. XFDL is a royalty-free open format whose current version specification, which can be obtained from -
XFDL is a host XML language for the XForms standard, and so the many benefits of XForms described above are inherited by XFDL. The implementations of XFDL as a presentation layer language add the following benefits over the core XForms processing:
very high precision control over the layout and rendition of the user interface
integration of XML Signatures with both XForms and a high precision user interface
comprehensive treatment for accessibility, localization and language support
dynamically computable presentation layer effects based on data value changes
multiple page support and ability to easily integrate web analytics within the form
ability to print or generate a PDF snapshot of the form or of just the agreement style pages
built-in ability to save the form locally and reload it later to continue the fill experience
a built-in file attachments folder system, including the ability of digital signatures to protect files attached into the form
a “public data” map for mapping XForms instance data to business process/workflow variables and case management properties
integrations to standard application server processing of form results and to run-time processing by JSR 168 and JSR 286 compliant portlets
the choice of zero-install operation within a web browser, using a server translator module
the choice of a client install to support both offline and online processing
In a larger sense, though, this only part of the benefit derived from XFDL. Still more benefit is derived from the availability of the XFDL forms visual design environment, which gives form authors integrated access to GUI features for XForms, for XML Signatures, for schema-driven design, for web sevice connections, and for the XFDL language benefits included in the list. The design environment even includes a converter that helps preserve the layout of PDF Forms that are brought over into XFDL. Finally, the XFDL forms visual design environment also provides features for maintaining a collection of forms, such as SVN and other team repository plugins and management of form parts.
Modified by John M. Boyer
Ever since my first blog entry in this recent series on artificial intelligence, I've been highlighting the lesser, calculational nature of machine intelligence and learning-- as well as the valuable role it nonetheless can play in driving more effective human understanding and decisions. I've been doing this by articulating mainly what machines do, as that is the primary interest of mine and most who would read a developerWorks blog. Still, our interests will be served by taking an entry to discuss human learning as a counterpoint or contrast.
The multiple linear regression example in my last post is a good example to start with because it highlights the difference between accuracy versus understanding. If there is a linear relationship among the data, then an MLR can have a very high predictive accuracy, but it has no explanatory power whatsoever. The MLR model does not have, nor does it convey, any understanding as to why the relationship exists.
Let's see how this predictive accuracy rates in terms of human intelligence and learning. In this case, we can benefit from an instance of that delightful human propensity to apply ideas to themselves. Specifically, we humans have applied our learning abilities to the phenomenon of our learning abilities, with many useful results including Bloom's Taxonomy.
According to Bloom's taxonomy, the very lowest level of cognitive learning is the knowledge level, or the ability to remember and recall what is learned. When you think about it, you realize that an MLR model, like many predictive analytics, is really a storage mechanism for something that has been machine learned from data. In MLR, we store the constants of a linear formula as the representation of what has been learned from linearly related data.
The next higher level of Bloom's taxonomy is comprehension, which is where understanding and true explanatory power begin to surface. But human learning is so much more sophisticated than the knowledge level of machine learning that there are a number of levels above comprehension. There's the application level, in which we can use our knowledge to solve new problems, including being able to explain why the new solution works. The analysis level drills deeper into our ability to make inferences and generalizations. The synthesis level begins to get at our ability to be creative with what we've learned and come up with new ideas and solutions. Finally, the evaluation level gets at our ability to be subjective and judge quality and creativeness of ideas and solutions. We are beginning to see some faint glimmers of some elements of some of these levels in cognitive computing efforts like IBM Watson, but it is early days indeed.
While we're on the subject of human learning and Bloom's Taxonomy, it makes sense to digress for a bit and mention the IBM Social Learning product. This is a SaaS educational platform intended to help enterprises achieve a Smarter Workforce. A few reasons for the digression are
this is a product I currently work on,
both it and the whole Smarter Workforce initiative are being featured at the IBM Connect 2014 conference being held right now, and
learning is a key ingredient of how a human workforce becomes smarter.
The IBM Social Learning product has a very nice feature that enables educational administrators to implement Bloom's Taxonomy in their learning materials. A component of the product is the Kenexa LCMS, or learning content management system, which includes various subcomponents like a course designer and a metadata dictionary. The educational administrator can add any metadata tag, such as "Learning Goal", and any tag values, such as "Basic Knowledge", "Comprehension", "Application", etc. Once this is done, the educational administrator can use the metadata tag values to classify any learning item in the LCMS accord to Learning Goal. Once these classified learning materials are published, learners can use the "Learning Goal" as a new faceted search criterion in the platform's learning library. A learner would be able to isolate and focus on "knowledge" level learning in a subject area before proceeding to comprehension and then application, for example. This will enable learners to effectively use the natural way in which their learning blooms, i.e. Bloom's Taxonomy.
Finally, there is an aspect of human learning that goes beyond Bloom's taxonomy, and it's an area that is highlighted by the IBM Social Learning product. There is a very important word in the product title: Social. This is crucial because it underscores the central role of communication and collaboration in the human learning process. We are an order of magnitude more effective at learning based on our interconnectedness to others who think and learn, rather having access to just data. This is pertinent to the advancement of artificial intelligence because "social" goes quite beyond the computing architecture underlying a lot of today's machine learning efforts.
Modified by John M. Boyer
Your intelligent behavior is based on sentient *understanding*. Sentient schmentient. I'll bet my intelligent behavior can outfox yours. I've done so with my friend Wiley from Canidae
, and he's a genius! So, let's see how much good your sapience does you, shall we?
The rules of the contest
are simple. You get the top six "houses" and the "store" on the top left. I get the bottom six houses and the bottom right store. We each start out with 6 seeds in each of our 6 houses, and 0 seeds in our stores. To win, you have to get more than half of the seeds into your store (for you knuckle draggers, that's 37 or more). I'll let you go first, so you already start with advantage.
To take your turn, you pick one of your houses that contains seeds. That house is emptied, and its seeds are "sowed" one at a time in a counterclockwise fashion, including your store but excluding mine. So, it takes 13 seeds to traverse from a starting house, through your store, through my houses, and back to your (now empty) starting house. Every seed that goes into your store gets you closer to victory.
You can earn a seed or two from your move, but there are a few more rules that can earn you lots of seeds. First, if the last seed you sow lands in your store, you get another turn, and you can have multiple extra turns if you make your moves in the right order. Second, if the last seed you sow lands in an empty house, then you earn that seed from the empty house and all seeds in the house of mine immediately below the empty house. I call this a "big take". Third, if I run out of seeds in all my houses, then you earn all the seeds in your houses. Of course, I can also earn lots of seeds by these same rules, which is why YOU'RE GOING TO LOSE MEAT BAG!
I will take it easier on you at first, but I'll play harder if you earn the privilege. And there's a special message for you, a badge of distinction, if you manage to beat me when I play my hardest. Ooops. You... win?!? Wake up! Your teetering bulb is dreaming!
SPOILER ALERT. PLAY A WHILE, BEFORE LOOKING ANY FURTHER.
OK, so hopefully you've played enough to know you're not going to be getting that badge of distinction anytime soon (unless you have some of the rare talents of Ted Neustaedter
). But also hopefully you're coming to the understanding that I really have no clue what I'm doing when I beat you. What I'm doing is mechanical, not miraculous. I'm being no more intelligent, really, than a calculator squaring a five digit number. Now, when one of you meat bags does it
, it actually is miraculous. But the miracle is that you can do it at all on your hardware given that it is designed more for sentient understanding of what mechanical operations like squaring are, what they're good for, and what to combine them with.
I am just doing the fine-grain operations of my Minimax algorithm
, but it is you who understands our contest at a higher level than that. That's why machine intelligence
like mine is best applied as an expert advisor
. For example, if you hit "Invoke Expert Advisor", you are asking me to advise you in the limited domain where my simulated intelligence would seem like real intelligence.
Keep using that expert advisor button and see how much faster you earn that special "badge of distinction" message. Go ahead. You won't be able to do it entirely without also sprinkling in your own intelligence at some points. This will be because you will hit some key points where your sentient understanding recognizes a *pattern* that emerges that will allow you to see how to beat my mechanical intelligence, where even my own advice is unable to do so. What will most likely happen is that you'll use the advice to hold your own for most of the game. My advice will help you avoid moves that give me extra turns and "big take" opportunities. But at some point, you may see that I am beginning to be starved of seeds in my houses. You, as an expert, will have this insight sooner than I see it coming using my mechanical calculations because your sentient intelligence truly understands what is going on at that higher level.
But of course, you would have a much harder time getting to that point without my advice. And that is what makes machine intelligence like advanced analytics on big data and machine learning technologies like IBM Watson invaluable to you. In short, expert advisors can turbocharge the smarts in your smarter workforce.
Modified by John M. Boyer
David Lee Roth and Eddie Van Halen have been trying to get us to do it for decades: "JUMP!" Douglas Hofstadter would qualify that with "... out of the system!" Here's what that means.
Machine intelligent entities like James Blog
exist within a certain system, conforming to a prescribed set of rules, and they really can't escape the confines and constraints of that programming. Within that limited domain, they do calculate wonderful results that can seem intelligent. In an early version, I found myself adding a logger so I could see why James Blog was not making some moves that seemed very good. Time and again, I would find that the good move now set up the conditions for a better opponent move later, which is exactly what the artificial intelligence is supposed to detect and avoid.
The algorithm does this so well that it is really hard to beat, especially on the maximum lookahead value I set, which was 6. Frankly, if you're new to this game, you have to work to beat even the initial lookahead level setting of 2, which means that James Blog only looks at its own moves and your countermoves to see what will produce the greatest net gain in seeds relative to you.
Because it is hard to beat this little game and see the special winners message, this opened up a delightful opportunity to talk about an important capacity of human intelligence that could be exemplified by determining the winners message without winning. I used a Zen-like characterization of a "winless win" as a nod to Hoftstadter's style in the book Gödel, Escher, Bach.
Put simply, we are not limited in our thinking to the confines of the system. We regularly "take it up a level" or "think outside the box". In this case, the system is a blog entry presented in a web page. So you can jump out of the system by using the View Source feature of your web browser to take a look at James Blog's code, where you will find the winners message: "I, for one, welcome my non-computer overlord." The message is an allusion to Ken Jennings' capitulation to IBM Watson, which was an awesome pop culture nod to The Simpsons-- awesome because both Jeopardy and the Watson AI are about sorting out exactly those kinds of allusions.
Frankly, I had a lot of fun with allusions, both in the blog entry and while holding the programmer challenge to achieve this winless win. For example, James mentions that he outfoxes his friend Wiley, alluding to the famous coyote, who is in the same animal family as a fox (Canidae), which is a tiny aural tweak from Canada, where I live. So, James can beat his wiley creator. Similarly, in tweets and status updates, I made numerous allusions to The Matrix movie, such as when I nearly used Morpheus's command to Neo: "Quit trying to hit me and hit me." The exception is that I changed the 'h' to a 'g', making 'git', which is what we use to get source code.
This kind of wordplay and allusion bears some similarity to "jumping out of the system". Hofstadter calls it contextual slipping, or my favorite word for it: counterfactualization. We take some piece of reality that we know about, and we ask "what if this were different?" We slip, or change, some piece of that reality to see if we end up with something new and useful. I find the notion of counterfactualization fascinating because it seems like a good operationalization of some other really important words: creativity, playfulness, humour, imagination.
Still, it might be a while between when we can efficiently and effectively operationalize contextual slipping and when we can generalize that to achieve machine intelligence that can jump out of any system in the way that I asked programmers to do with James Blog. At some point, I realized that there is a beautiful geometric analogy that helps explain why. In the book Flatland, the Sphere is able to escape the plane via the use of a third geometric dimension that is physically orthogonal to the two that comprise the plane. In this way, Sphere is able to see Square's inner workings. That is a great analogy with what we did by jumping out of the web page using View Source to see James Blog's inner workings. There was a whole different, higher level of understanding about what James was and how we could know more about it, and it is fitting to say we got that winners message by thinking outside the box.
Next blog will be a developer's tour of the particular machine intelligence algorithm built into James Blog. After that, will be a discussion of the relationships between machine intelligence, machine learning, and predictive analytics, so stay tuned!
The hit rate on my last blog announcement about IBM Forms 4.0 makes it clear that the release is pretty big news. I'm very pleased about that because it is a *major* release, both technically and in terms of the corporate branding. Corporate branding is not a decision lightly made, so it took the time it took, and meanwhile we didn't want to delay the technical features hitting the market, so you'll see some references to "Lotus Forms" under the covers of the main IBM branding. Not to fear, we're working on that for the next release.
Meanwhile, now that IBM Forms 4.0 is released, I've made a number of demonstration videos to show you key aspects of the IBM Forms Design and Run-time products. They're available on YouTube in HD, and they're broken into small segments, so you can really take them in pieces according to your interests.
To start, here's a 5-part series that takes you through run-time features with a demo form that ships with the product. These videos help you understand the value proposition of IBM Forms as well as new features of the release that will change your mind about what forms are and what they do.
- The New IBM Forms 4.0 - Introduction, Web 2.0 Look and Feel, AJAX Updates (LARGER YouTube HD)
- The New IBM Forms 4.0 - Dynamic Tables, Form Parts, Integrating Business Analytics (LARGER YouTube HD)
- The New IBM Forms 4.0 - Dojo Controls in Dialogs and on the Forms, Integrating Web Analytics (LARGER YouTube HD)
- The New IBM Forms 4.0 - High Precision Layout, Digital Signatures, Completing a Business Transaction (LARGER YouTube HD)
We also now have several videos that show off the IBM Forms Designer. This release really raises the bar on what level of data interactivity can be created through design GUI configuration rather than grunge coding, and many powerful XML data processing features of the IBM Forms Designer are directly attributable to W3C XForms. This includes Design by XML Schema, Dynamic Tables, Web Service Connections, Interaction Rules (e.g. constraints), and Wizard Creation:
Finally, here's a video series on the new FormParts feature (I didn't make these, but they're excellent and really belong in this post). FormParts allows you to create components that can be reused and kept up to date across a forms library.
If you want to know more about IBM Forms 4.0, then come to our wiki
or to our website
, where you can get more information, case studies and free trial downloads. Thanks for watching!
One of the New IBM Forms 4.0 Demonstrations
has just won the top-rated video award at Lotusphere
. This is a fantastic victory for the Websphere Portal segment of Lotus, which focuses on products like IBM Forms that create and provide exceptional web experiences. It is also a victory for the W3C XForms standard
since the Wizard Creator
featured in the video is, as far as I know, the first point-and-click design experience for an XForms switch. This feature exemplifies the principle that declarative markup languages result in more powerful application design environments because they express what the author wants. With an imperative languages, a design environment must either operate at a much lower level (less powerful) or do some wicked reverse engineering/pattern matching to discern what the author wanted from how he did it.
So, XForms has definitely been my friend in helping to create an award-winning feature in IBM Forms. Wanna meet my newest friend? Here we are, just me and Watson, celebrating our victories at the closing session of Lotusphere 2011:
To put it in Watson's terms:
Category: Lotusphere Best Demo Video Winners
Answer: An IBM Form that uses XForms to express Wizard interfaces for forms.
Question: What is a Smarter Web Application?
MC: Congratulations Watson, you're ready for prime-time! Wanna join my social network?