In "enterprise" service-oriented architectures (SOAs), XML is a clearly entrenched format for server-to-server communications. But there's a movement afoot to define a "Web SOA" as a separate entity that describes server-to-client communications... based on JSON as the format.
The rationale is that JSON parses "a hundred times faster" in a web browser than does XML. This is like the tail wagging the dog. It is a myopic, accept-the-status-quo approach to web application computing, and it just won't do.
First of all, the difference between parsing different serializations is negligible (when done right), and nodes are nodes are nodes, so if there is a 100-fold difference, it is because one parsing method is direct and the other is doing something phenomenally suboptimal and probably easy to fix. One way to solve this is to alter all the system architectures on the web and introduce the complexity of a data format transformation to cross the illusory chasm between a "Web SOA" and the "Enterprise SOA". Another idea would be to put the pressure where it rightly belongs, on web browser makers to fix the feature of efficient XML processing.
Particularly as vertical industry standardization continues, such as is happening in both insurance and healthcare, this latter solution is really the only viable one in the long term. XForms technologies like Lotus Forms and Ubiquity XForms wrap a logical client application around the exact XML that is needed by the server side because this is the best way to minimize system development and maintenance costs. Once there is a defined XML data model, there should not be some other second data model introduced into web applications for some reason as vapid as poor implementation of the required feature.
This conversion is not a trifling matter either; have you seen how ugly the JSON is that is actually capable of preserving XML fidelity (i.e. supporting round-tripping)? Trying to build a reasonable client-side around that JSON is not nearly such a pretty sight as when the data representational limitations of JSON are accepted in an application. If the full representational power of XML schema is used to define an application data model, then we need to be able to carry that out to the client without significant alteration so that we can make references to the data transparently, not through the lens of some horrendous transformation. And make no mistake, XML schemas for vertical industry standards are full-featured indeed.
Finally, I would be interested in hearing about any research into whether the overall throughput of a web application is increased or decreased by the introduction of JSON down to the client. It's nice that parsing gets a hundred times faster on the client, but when you're server is trying to handle more than a hundred concurrent users, and the server has to translate from XML to JSON (and back again), then it sure seems like the use of JSON is pushing processing burden in the wrong direction.
Modified by John M. Boyer
As an interesting possible counterexample to my last blog about MLR models not understanding the knowledge they learn, consider the neural network. Our brains are neural networks, and we are capable of learning at all levels of Bloom's Taxonomy, not just the knowledge level. Shouldn't artificial neural networks be able to achieve the same things?
The answer is no, not really. Our brains biologically, chemically and physically perform in ways that we scarcely understand, so our name for the thing we call "artificial neural network" is no less anthropomorphizing than when we say that a computer program of today "understands" anything.
Still (again), this is not to say that they aren't incredibly useful and effective. It's just that they are based on straightfoward and well-understood mechanical methods such as feed forward activation of neural outputs via sigmoidal threshold functions applied to inputs and back propagation of synaptic weight adjustments based on easily quantified classification errors. Before going any further, let's have a quick look at a diagram of an artificial neural network (ANN):
The ANN has an output layer on the right that is a classifier for input patterns received on the left. For example, an ANN for optical character recognition could have an input layer of an 8x8 matrix of bits, and the output layer could be an 8-bit code that indicates an ASCII character. The hidden layer(s) of neurons help the ANN to represent more sophisticated phenomena, though there is seldom need for more than one hidden layer. The "synaptic" connections between the neurons in the layers are weighted numbers, and the neurons apply the weights to the inputs and then feed the results into a Sigmoid function that essentially decides, like a transistor or switch, whether or not to fire the output.
An ANN is "trained" by giving it a sequence of input patterns for which the correct output pattern is known. The input pattern feeds forward through the ANN to produce an output. If there is a difference between the ANN output and the correct output, then the differential error is back propagated through the ANN to adjust the weights so that future occurrences of that input pattern are more likely to produce the correct output.
The synaptic weights, then, essentially represent the knowledge that the ANN "learns" from the input patterns. This is analogous to the constants that are "learned" by an MLR model. In fact, all elements of the ANN and MLR model architectures are analogous. The ANN input layer maps to the the independent X variables, the ANN output layer maps to the dependent Y variable, and the transition from input X values and the Y value that is achieved in MLR by multiplication and addition is achieved by a feed forward through synaptic connections, hidden layer neurons and Sigmoid functions in an ANN.
With such a one-to-one architecture mapping between ANNs and MLR models, it is easier to see them as having similar intellectual power. That's not to say they're equivalent, as ANNs are far more powerful. It's just that they're roughly the same (low) order of magnitude with respect to human intellect, and in terms of Bloom's Taxonomy, we call that order of magnitude "knowledge storage/retrieval".
Despite being in the lowest order of magnitude of intellect, the realm of today's artificial intelligence includes many interesting knowledge storage/retrieval techniques that are worth comparing and contrasting to see the range and limits of their power and the use cases they address. Stay tuned!
Information system architects, and even vertical industry standardization or government IT standardization bodies, derive immense value from XML schema definitions that describe the data structures and data types expected in valid transactions of the information system. However, these assets are focused on defining the completed transaction. In many a presentation, I've talked to potential customers about how IBM Forms documents express far more value because they are about the human interaction that takes place during the fill experience needed to produce those completed transactions.
As a bit of an aside, it's important for the technically minded reader to be familiar with the "sell" side of this equation. It is important to be able to easily justify technology adoption decisions with business owners who need to be able to understand how you will be able to better server your customers, reduce development and maintenance costs, increase competitiveness, eliminate vendor lock-in, etc.
In this blog, I'd like enumerate various benefits that you get from the standards basis of IBM Forms documents and their implementations, but I'd also like to separate the enumerations into two lists: 1) benefits above using XML Schema that you get just from using the XForms markup within an IBM Form, and 2) the benefits you get from the XFDL processor (XFDL is the XML vocabulary that IBM Forms provides as a presentation layer for XForms).
Firstly, an XForm is a clearly a superset of XML Schema since an XForm can incorporate the XML Schema if it is available and provide its validation information set to the fill experience and the submission experience. But an XForm provides many additional benefits, including the ability to:
Express data validation constraints that are based on other XML data values entered during the fill experience.
Automatically compute data values based on other input entered during the fill experience, rather than requiring users to perform the error-prone task of calculating and inputting summative results manually.
Describe the inputs, outputs and triggering mechanisms of web services to invoke, such as to use entered XML data to obtain database query results that fill other parts of the form or to invoke server-side validation or calculation logic from a business rules engine.
Control whether a data value is readonly or whether the user can enter a value, for example based on conditions related to a business process step or an access rule.
Express user interface controls that indicate which XML data values will be available to the presentation layer for input or output.
Conditionally show or hide the user interface controls (and hence their presentation) in response to conditions, such as those that may relate to a business process step or access rule.
Define a step-wise fill experience to guide a user systematically through incremental successes in entering valid data.
Provide customized help and validation error messages to help users fix data input errors when they occur.
Provide prompting label text to be associated with the presentation of each input or output control, both visually and aurally (for accessibility)
Associate a selection list with any XML data node in a way that constrains user input capability to the provided list
Define labels, help messages, validation error messages and selection lists in more than one human language within the same form, thereby ensuring citizens receive the same form logic and interaction behaviors regardless of which official language they select to request government services
Associate repeated data with a logically tabular set of user interface controls and encode the means by logical rows of the table are added or removed in response to insertions or deletions of data
While the above benefits indicate what additional behaviors and features of XML forms can be expressed above those that can be expressed by an XML Schema alone, XForms is also more interesting for standardization of a forms repository due to what it does not express. An XForm does not rigorously bind its many behavioral benefit to a specific presentation layer of the form. The intention of this language architecture was to address multimodal requirements of forms applications, e.g. rendition on a desktop, tablet, smartphone, telephone call-in voice service or instant messaging interaction. Different presentation layer implementations can address these requirements, and such implementations can even be provided by different vendors. The XForms working group also anticipated that there would be a wide array of varied technical requirements for presentation layers, and this language architecture allows XForms to be used with fundamentally different XML presentation languages that address these disparate requirements. Examples range from ODF for flowing text with fill-in-the-blanks fields to XFDL with its high-precision contract-style layout capability.
Due to the above mentioned language architecture, XForms markup does not comprise a well-formed XML document until it is incorporated into a presentation layer XML document. XFDL in the XML vocabulary used in IBM Forms to provide a presentation layer for XForms.
Of course, in a "baseline standard" version of a form, the default presentation layer associated with the XForms markup can be minimal in nature so that the only benefit is to provide a well-formed XML document to host the XForms markup. Interestingly, once you have this from XFDL, then the result is in fact an XML document, and so it can be processed by readily available XML processing tools like XSLT. These XML tools can be usd to automate creation of different versions of the baseline form that may have a richer presentation description, alternative natural language usage, or even different presentation layer markup. In addition, various consumers could use simple XML tooling to rebrand the forms.
The Extensible Forms Description Language (XFDL) in an XML vocabulary describing the presentation layer and richly interactive behaviors of modern web-based electronic forms. This XML vocabulary was first introduced to the W3C in 1998 (http://www.w3.org/TR/NOTE-XFDL
) and over the years, the versions of the language have consumed XML data-processing components of the W3C XML technology stack as they have been standardized by the W3C, especially including XML Schema, XForms, and XML Signatures. XFDL is a royalty-free open format whose current version specification, which can be obtained from -
XFDL is a host XML language for the XForms standard, and so the many benefits of XForms described above are inherited by XFDL. The implementations of XFDL as a presentation layer language add the following benefits over the core XForms processing:
very high precision control over the layout and rendition of the user interface
integration of XML Signatures with both XForms and a high precision user interface
comprehensive treatment for accessibility, localization and language support
dynamically computable presentation layer effects based on data value changes
multiple page support and ability to easily integrate web analytics within the form
ability to print or generate a PDF snapshot of the form or of just the agreement style pages
built-in ability to save the form locally and reload it later to continue the fill experience
a built-in file attachments folder system, including the ability of digital signatures to protect files attached into the form
a “public data” map for mapping XForms instance data to business process/workflow variables and case management properties
integrations to standard application server processing of form results and to run-time processing by JSR 168 and JSR 286 compliant portlets
the choice of zero-install operation within a web browser, using a server translator module
the choice of a client install to support both offline and online processing
In a larger sense, though, this only part of the benefit derived from XFDL. Still more benefit is derived from the availability of the XFDL forms visual design environment, which gives form authors integrated access to GUI features for XForms, for XML Signatures, for schema-driven design, for web sevice connections, and for the XFDL language benefits included in the list. The design environment even includes a converter that helps preserve the layout of PDF Forms that are brought over into XFDL. Finally, the XFDL forms visual design environment also provides features for maintaining a collection of forms, such as SVN and other team repository plugins and management of form parts.
Recently, I was experimenting with one of the features planned for the next version of XForms. The feature is the iterate attribute for XForms actions, which will perform a for-each loop operation based on a nodeset obtained from the xpath expression in the iterate attribute value. XForms 1.1 already has a while loop, but iterate makes many data processing loops easier to write and also more performant (subject of a future blog). There were lots of iteration use cases to choose from, but I decided to experiment with sorting because it is a well-known benchmark algorithm.
Before we go any further, let me say up front that XForms action scripting is intended for very lightweight data manipulation, like adding or deleting a data node corresponding to a table row or copying data results to or from the SOAP envelopes of a web service. By the time you get to nested iterations like those needed sorting, you should be considering alternatives expressed in full-blown imperative languages available in the information system within which the form is being used. For example, in the case of sorting, it is a better idea to request sorting in the database query whose results are returned from a web service into your form so that your form logic does not even have to do the sorting.
So, with the disclaimer out of the way, let's abuse the technology a bit to get a better sense of what is feasible in those customer-needs-it-yesterday circumstances. It turns out that XForms 1.1 does allow full nodeset processing in the insert action's origin attribute and the delete action's nodeset attribute. Without even needing the new iterate attribute, this is just enough iteration capability to perform efficient sorting -- so there are some kinds of iterations that can be done now without the iterate attribute.
We're going to do a divide-and-conquer "partition" sort that I personally created as a university freshman after my 1st semester instructor told our class that linked lists could only be sorted slowly. At the time, the usual computer languages only allowed static allocation for arrays, and even though I didn't know what a "quick sort" was, I had seen the light of dynamic allocation, and I was never going back! I later learned how great a merge sort is on a linked list, but the effort of turning an array quicksort into a linked list partition sort comes in handy now because a merge sort cannot be efficiently expressed in XForms until the iterate attribute is added.
The way a quicksort works on an array (or subarray) is that you pick a random element to be the 'pivot' value. Then you run two index variables at the same time, one from the start of the array upward and the other going from the end of the array downward. The 'up' index is advanced until it finds a value greater than the pivot, the 'down' index is decremented until a value less than the pivot is found, and then the values at the 'up' and 'down' locations are swapped. This keeps happening until 'up' and 'down' meet somewhere near the middle of the array. At this point you've partitioned the array into a subarray of values less than the pivot value and a subarray of values greater than the pivot value. The quicksort is then invoked recursively to sort both subarrays.
The main challenge with this approach is the 'down' index, which is a reverse iteration. In a singly linked list, you can only go forward. XForms insert and delete actions have a similar limitation: they can only identify a nodeset of nodes to insert or delete, but not really a direction of iteration. But the important bit is what the quicksort is doing, not how it is done. Think of the list content as being completely messy, and each partitioning stage must make it somewhat less messy by dividing the content into a partition of lesser elements and a partition of greater elements. Then, the next partitioning stage is invoked recursively to do a better job of cleaning up the mess within each partition.
Let's explore this concept by sorting a list of elements, such as sorting a list of <person> elements by a <lastname> child element. We begin by copying the list into an initial partition element of a temporary instance called 'sortdata', like this:
1) <xforms:insert context="instance('sortdata')" origin="instance('partition')"/>
2) <xforms:insert context="instance('sortdata')/partition" origin="instance('data')/list/person"/>
3) <xforms:delete nodeset="instance('data')/list/person"/>
Next, we initialize the random number generator so we can randomly select pivot values for all the partitioning stages:
4) <xforms:setvalue ref="instance('sortpivot')" value="random(true)"/>
Next, we start up a simple while loop that continues to process partitions until none are left.
5) <xforms:action while="instance('sortdata')/partition">
Within the loop, we grab the last partition from the sort data and determine whether it is non-trivial or trivial (only 1 or 2 elements). A non-trivial partition is subjected to further divide-and-conquer processing.
5.1) <xforms:action if="count(instance('sortdata')/partition[last()]/*) > 2">
The first part of dividing and conquering is to create a new empty <partition> element, which we obtain from an instance that expresses a template empty partition element.
5.1.1) <xforms:insert nodeset="instance('sortdata')/partition[last()]"
Step 5 and step 5.1.1 are more interesting than they seem at first. The list of partition elements in the sortdata actually implements the recursion stack, and we just pushed a new element into that stack at the second-to-last position. Because we have an explicit stack, we only need a loop in step 5 to implement recursion.
The next thing we do here is grab a random last name to serve as a pivot value for the partitioning. The first setvalue just picks a random location, and then the second step uses the location to get the value. Notice also that I use * rather than partition before the [last()] predicate because the sort data only contains partition elements, so there is no point in doing a name test for partition.
5.1.2) <xforms:setvalue ref="instance('sortpivot')"
value="1+floor(random(false) * count(instance('sortdata')/*[last()]/*))"/>
Now the magical part happens. All elements in the last partition whose key element (lastname) is less than or equal to the pivot value are moved to the newly created second-to-last partition. By combining the nodeset processing capability of XForms insert and delete actions with the predicate-based node selection capability of XPath, the matching nodes can be selected and moved using two single XForms actions, i.e. without using an XForms while loop.
5.1.3) <xforms:insert context="instance('sortdata')/*[last()-1]"
origin="instance('sortdata')/*[last()]/*[compare(lastname, instance('sortpivot')) <= 0]"/>
<xforms:delete nodeset="instance('sortdata')/*[last()]/*[compare(lastname, instance('sortpivot')) <= 0]"/>
If the last partition is now empty due to the move operation in step 5.1.3, then the new second-to-last partition received all its elements. If all the moved elements are equal to the pivot value, then we can output them back into the original data list and then remove the last two partitions. Note that the insert is configured to prepend the elements into the data list, and we're copying them from the last non-empty partition, which has the elements with the greatest key value.
5.1.4) <xforms:action if="count(instance('sortdata')/*[last()]/*) = 0">
<xforms:action if="not(instance('sortdata')/*[last()-1]/*[compare(lastname, instance('sortpivot')) != 0])">
<xforms:insert context="instance('data')/list" origin="instance('sortdata')/*[last()-1]/*"/>
<xforms:delete nodeset="instance('sortdata')/*[position() >= last()-1]"/>
</xforms:action> <!-- End of 5.1 non-trivial partition handler -->
Now, we've finished with the non-trivial partition handler, and we turn our attention to processing a trivial partition containing at most 2 elements. The content of the partition is moved to the original data list and the partition is removed. Again, note that we're processing the last partition, which has the greatest key values, and the insert prepends to the data list, so the sorted data list starts with the greatest values and grows as lesser and lesser values are prepended over time as all the partitions are processed.
5.2) <xforms:action if="count(instance('sortdata')/*[last()]/*) <= 2">
If the partition contains two elements and the greater one is first, then it is removed from the partition and put in the original list:
5.2.1) <xforms:action if="count(instance('sortdata')/*[last()]/*) = 2 and
instance('sortdata')/*[last()]/*/lastname) > 0">
<xforms:insert context="instance('data')/list" origin="instance('sortdata')/*[last()]/*"/>
Now the partition contains either no elements, one element, or two elements in sorted order, so we move the content to the data list and then delete the partition.
5.2.2) <xforms:insert if="count(instance('sortdata')/*[last()]/*) > 0"
</xforms:action> <!-- End of step 5.2 trivial partition handler -->
</xforms:action> <!-- End of step 5 recursion while loop -->
As a final note on all this algorithmic fun, the question arises whether this sort achieves optimal O(N log N) performance. The answer is no, not quite, due to hidden costs of data instance management and data node selection. However, the sort will be much faster than a "simple" sort because it does perform only O(N) XForms actions.
There are a number of new XPath extension functions available to XForms developers in the latest release of IBM Forms, and I'd like to draw your attention to two of them: eval() and eval-in-context(), and they are wicked cool!
The function eval(expr) evaluates expr in the context of the function call and returns the result. The eval-in-context(expr, contextExpr) function does a similar thing, except it first evaluates the contextExpr and uses the result as the context for the main expression. This is desparately needed for XPath 1.0 expressions to eliminate the infestation of pesky ".." operations that typically occurs. I've used it, rather than eval(), in the samples below.
One use of these functions is to enable the powerful capability to let XML data carry sophisticated dynamic metadata, which can then be implemented and enforced with singular XForms bind elements that attach the semantics of the metadata whereever it is found in the data.
It turns out that the xsi:type attribute from XML schema is already a rudimentary version of the metadata idea we're pursuing here, so the question becomes what if you could do it for all of the juicy metadata that XForms contains, like calculated data values, data validity constraints, and so forth? Let's look at what this "decorated" data might look like for a simple expense report:
<total value="quantity * price"/>
constraint="total < 10000"/>
It is really easy to use XPath capabilities to find all elements having a value attribute and then to bind an XForms calculate formula to those elements, and then eval-in-context() is used to determine the result according to whatever expression is given in the data, like this:
<xforms:bind nodeset="descendant::*[@value]" calculate="eval-in-context(@value, ..)"/>
The nodeset expression starts with descendant::* to explore all elements of the data, and then the predicate [@value] selects all elements that have a value attribute. For each such element node, a calculate formula is bound to it by the XForms bind. The eval-in-context() call uses ".." to go up a level so that the formulas in the value attributes can omit "../", e.g. so the expression can simply be "quantity * price" rather than "../quantity * ../price".
Similarly, a data validation constraint can be attached like this:
On the expense report data above, this binding evaluates the constraint expression attached to the total element, and then converts the result to a boolean. Due to the constraint, the expense form data cannot be submitted to a server for processing unless the total expense is less than 10000.
In other scenarios, you may want to control the other metadata properties like relevant, required and readonly. Here's an example of data where relevance and requiredness control is needed:
<parent relevant="age < 18" required="true">...</parent>
The xsi:type assignment for age already works in XForms without needing a bind. The required setting for name is statically true, not dynamically changeable, so it is very handy to be able to allow either a static boolean value or an expression, like this:
required="@required='true' or boolean-from-string(eval-in-context(@required, ..))"/>
Technically, the required property on the parent element is conditional on the age value, but that is an automatic feature of XForms, i.e. nodes marked required are only required if they are relevant. It should not be too surprising to see that the bind for relevance looks like this:
relevant="boolean-from-string( eval-in-context(@relevant, ..) )"/>
At this point, it's probably safe to leave the readonly bind as an exercise for the reader :-)
One of the New IBM Forms 4.0 Demonstrations
has just won the top-rated video award at Lotusphere
. This is a fantastic victory for the Websphere Portal segment of Lotus, which focuses on products like IBM Forms that create and provide exceptional web experiences. It is also a victory for the W3C XForms standard
since the Wizard Creator
featured in the video is, as far as I know, the first point-and-click design experience for an XForms switch. This feature exemplifies the principle that declarative markup languages result in more powerful application design environments because they express what the author wants. With an imperative languages, a design environment must either operate at a much lower level (less powerful) or do some wicked reverse engineering/pattern matching to discern what the author wanted from how he did it.
So, XForms has definitely been my friend in helping to create an award-winning feature in IBM Forms. Wanna meet my newest friend? Here we are, just me and Watson, celebrating our victories at the closing session of Lotusphere 2011:
To put it in Watson's terms:
Category: Lotusphere Best Demo Video Winners
Answer: An IBM Form that uses XForms to express Wizard interfaces for forms.
Question: What is a Smarter Web Application?
MC: Congratulations Watson, you're ready for prime-time! Wanna join my social network?
How would you like to be able to construct, deploy and get results from IT solutions using only your web browser?
Don't believe me? Well, how about coming to the IBM Forms wiki
, where you can watch a few short videos that show you.
You'll be intrigued and want to go the next step. One of the prominently available wiki pages is a community article that gives you a starter pack
of prebuilt solutions like the ones you see in the video. You can download any one or all of them because they're just single files that describe the forms, access control, workflow stages and other resources of each solution. You can import any of them into your own IBM Forms Experience Builder
server, and then deploy them, use them, get results from them, and of course edit them to see how they work or to change them and redeploy them. All from your web browser.
Don't have an IBM Forms Experience Builder server to try it out? Well, now we've gotten to the main topic of this blog article. You can get your own free public access to an IBM Forms Experience Builder server. You can try out any of these starter pack solutions as well as build and deploy any of your own solutions.
Since you will be a builder
of forms experience solutions, we will need to be able to present your solutions to you, distinguished from everyone else's solutions. So, you'll have to start by registering yourself with the system that hosts the IBM Forms Experience Builder server. The system is called Lotus Greenhouse
, so click the link and then choose "Sign up" to get your account.
Once you're able to log in to Greenhouse, you'll get access to a number of software products including IBM's social business software (Connections), IBM Websphere Portal Server, and of course IBM Forms Experience Builder. However, you don't really need to log in to Greenhouse then menu navigate to IBM Forms Experience Builder when you can just bookmark the direct link to IBM Forms Experience Builder on Greenhouse
Once you log in with your Greenhouse user id and password, you'll see the "Manage" solutions page, which lists all of the Forms Experience Builder (FEB) applications that you have designed. This is the page that gives you the ability to create a "New Application" or "Import" one of those starter pack applications, all at the press of a button.
So, now you can try out and evaluate IBM Forms Experience Builder now to see for yourself that there really is a smarter web where you can construct valuable solutions without coding now. If you are building IT solutions for your organization, you owe it to yourself to see how much more effective you'll be at satisfying your organization's IT solution demands. But even more importantly, if you're competing for IT solution services contracts, you owe it to yourself to become an IBM business partner or to expand your partnership to include IBM Forms Experience Builder. And finally, if you like to build industry-specific data management products, then you should consider becoming an IBM value-added reseller (VAR) so you can build your products more efficiently with IBM Forms Experience Builder and go to market with IBM to sell the bundle. In all these cases, you now have the access you need above so you can learn more and get started today.
The XForms team finished our face-to-face meeting in Amsterdam last week. A major focus of the work on XForms 1.2 is called modularization.
The rationale for this work is the observation that the set of XML data processing problems which may have first arisen in the electronic form space are really more generally applicable to the XML data processing needs of RIAs and web applications. On the other hand, those who hear the word "form" may think they do not have a form problem because they still think of a "form" as a simple or static application like those for ordering pizza or flowers. However, XForms has solved numerous problems that keep coming up again and again throughout the W3C standards stack as well as the web application stack. By modularizing the components of XForms, we believe we can increase adoption of the components in other technologies which may not have need of all aspects of XForms.
The current view of the XForms 1.2 modularization can be viewed here. As an example of the rationale above, consider an application that may want to use the submission capability from XForms in a regular web application. The application would import the instance data module, but it might have an application-specific way of populating a data instance with data. The submission module would reference a data instance for the upload data and another instance for the submission results. An application-specific method would then be invoked to consume the submission result into the application, but the means of invoking that method could be an event handler for the xforms-submit-done event. The key issue here is that submission could be consumed by a non-XForms application without needing to incorporate the XForms recalculation engine, user interface controls and so forth.
The full elaboration of this modularization will allow applications to consume pieces of XForms incrementally, including the notion of data validation, data relevance, declarative data calculations, event-drive action scripting, repeats, switches, groups, basic user interface controls, and of course submissions.
Well, my frieds, it seems a good time to draw your attention to a collection of available resources that can help you understand how you and your customers can address a full spectrum computing requirements, from the enterprise information application all the way down to the simple design-deploy-collect-analyze pattern for situational applications that arise throughout the enterprise
First of all, check out the Lotus Forms Enablement videos on YouTube. There are videos that show Lotus Forms in action in government, financial, and HR scenarios where enterprise-level functionality is a must. But you can also see videos on the new Lotus Forms Turbo product, which enables business users to design and deploy simpler survey style forms for themselves in a matter of minutes.
I want to make a pitstop here to plug the value of the XForms standard in creating this array of products. XForms as a language has a strong declarative component. This means that non-trivial relationships between data and sophisticated interactional and presentational objects is simply declared by the form developer, and it is up to the form run-time system to enforce the relationships. This maps very naturally to how we human beings think and work. We say things like "Whenever I get an email from domain X, put it in my super special high priority folder". We don't directly write batch jobs that run loops over our inbox to categorize our email, and even if we did, the time at which those jobs would run would be specified declaratively! In fact, I don't think it's hyperbole to say that the universe itself operates mostly on declarative laws. Gravity just happens. If you let go something, it drops. It's the law; get used to it ;-).
The point is that the declarative language constructs of XForms are what enable Lotus Forms to accelerate enterprise-level application design, to enable end-user design of simpler applications, and to ensure interoperability between these two types of applications so that the simpler situational applications can be scaled up to the enterprise level as their value becomes more broadly established. The declarative constructs of XForms accomplish all of this due to the simple fact that it is far easier to create software that maps point-and-click/drag-and-drop design experience gestures onto high-level declarative language constructs than it is to map the same things onto huge blobs of imperative scripted code from a general purpose programming language.
And don't just take my word for it. When you finish watching some of the videos at the YouTube link above, how about going for a test drive? You can get free trial downloads of the Lotus Forms, including the client-side runtime and design environments. If you like the enterprise-level capabilities, but you don't want to deploy a client-side runtime, you can also purchase the Lotus Forms Web Forms Server, which translates Lotus Forms into standard web pages. But once you're in test-drive mode with the client-side runtime viewer, check out the Lotus Forms catalog of sample forms. Finally, you can also test drive the new offering for non-technical users directly, without having to install a thing by visiting Lotus Forms Turbo on Greenhouse, or if you'd prefer to test drive it in-house, then just download it from the free trial downloads page.
Either way, once you've gotten Turbo, see a couple of the videos then think of your own simple survey app. Design and deploy it right onto Greenhouse and see how long it doesn't take!
A customer asked me recently how they could use XForms constructs to create a form that dynamically populates a table with available products that can be ordered based on selection of a product provider. In this case, the customer also wanted to have the product order list be editable once provided, allowing the user to delete rows or even to add more rows to the product order table initially provided for the selected store. Seemed like another good example for the blog.
So, suppose you have a main data instance for a form that looks something like this:
<xforms:instance id="data" xmlns="">
<product name="" code="" cost="0" qty="0" total="0"/>
Each product has a name, code, and cost, and the end-user will indicate the quantity of the product they desire. The above data corresponds to showing an empty table until the user chooses a "storeID". Here is the XForms user interface markup for showing a four column table having as many rows as there are "product" elements in the data.
<xforms:repeat nodeset="products/store/product" id="orderTable">
<xforms:label>Desired Quantity of Product</xforms:label>
<xforms:label>Calculated Line Total</xforms:label>
Initially, the table will have just one row of four user interface elements containing essentially empty or zero values. However, now lets hook up something that allows us to pick a store ID so we can fill the table with an initial order of products. Now, it would be reasonable in a full form application to obtain the list of stores from a web service and then get the starting list of products for the selected store from another web service. Getting data from web services is not an important but orthogonal point, so in this mock-up, I'm going shorten all of that down to just having the data available in a format that looks like this:
<xforms:instance id='storeLists' xmlns="">
<product name="Widget" code="W1" cost="3.50" qty="0" total="0"/>
<product name="Gadget" code="G1" cost="4.25" qty="0" total="0"/>
<product name="Trinket" code="T1" cost="2.75" qty="0" total="0"/>
<product name="Gadget" code="G1" cost="4.25" qty="0" total="0"/>
<product name="Gromet" code="G2" cost="3.50" qty="0" total="0"/>
<product name="Widget" code="W1" cost="3.50" qty="0" total="0"/>
<product name="Trinket" code="T1" cost="2.75" qty="0" total="0"/>
<product name="Gromet" code="G2" cost="3.50" qty="0" total="0"/>
<product name="Sprocket" code="S1" cost="1.99" qty="0" total="0"/>
<product name="Locket" code="L1" cost="9.50" qty="0" total="0"/>
<product name="Pocket" code="P1" cost="1.50" qty="0" total="0"/>
<product name="Rocket" code="R1" cost="7.50" qty="0" total="0"/>
<product name="Sprocket" code="S1" cost="1.99" qty="0" total="0"/>
<product name="Socket" code="S2" cost="2.49" qty="0" total="0"/>
The user is provide the ability to select a store using the "select1" control, and the list of stores can be easily picked up from the data using an "itemset". Once the user makes a choice, an "xforms-value-changed" event on the select1 could be used to run a web service to get the product list, but here we'll just mock that up with an "insert" because the data is already available:
<xforms:select1 ref="storeID" appearance="minimal">
<xforms:label>Choose a store</xforms:label>
<xforms:delete nodeset="instance('data')/products/store" at="1"/>
<xforms:insert context="instance('data')/products" origin="instance('storeLists')/store[@ID = instance('data')/storeID]"/>
The "ref" on the select1 tells where to store the resulting store selection. The "nodeset" on the itemset tells where to get the list of stores from. The "ref" attribute on the xforms:label in the itemset tells what to show for each item in the list of choices, and the "ref" xforms:value tells what to store in the data ("storeID" due to the ref on select1) when a particular list choice is selected.
The "xforms-value-changed" event handler recognizes when a selection has been made, since that results in a value change on the "storeID" data node. The delete action gets rid of any preceding list, and then the insert action copies the list of products for the selected store into the main data. In particular notice that the XPath predicate in the origin attribute selects a store element to copy based on the store element's ID attribute matching the selected store identity placed in the storeID element by the value change behavior of the select1.
Once this insert occurs, the xforms:repeat is automatically responsive to the change of the data. It generates a four column row of user interface controls for each of the inserted product elements. For example, if the user picks store A, then they get three rows for Widgets, Gadgets and Trinkets. If they then pick store D, the form automatically adjusts to five rows for Lockets, Pockets, Rockets, Sprockets and Sockets.
Once the table content has been set with the product list for a particular store, the user may choose to add or delete rows from the table. Here is an additional instance that would be used to store the data prototype for a product:
<xforms:instance id='proto' xmlns="">
<product name="" code="" cost="0" qty="0" total="0"/>
A button to add a row to the repeat table would trigger the addition using the following XForms markup:
<xforms:insert context="products/store" nodeset="product"
This simply inserts a product prototype, obtained via the origin attribute, into the location defined by the context and nodeset attributes at a position corresponding to the row of the table that current has the input focus. When the data is inserted, the repeat table automatically generates another four-column row of user interface elements to allow the user to interact with the new data.
Similarly, a button to delete a row from the repeat table would trigger the deletion using the following XForms markup:
<xforms:delete context="products/store" nodeset="product" at="index('orderTable')"/>
<xforms:insert context="products/store" at="1" position="before" origin="instance('proto')/product"
This simply deletes the product data element corresponding to the row of the table that currently contains the input focus. The row of user interface elements that presented this data is automatically deleted. As the next step of the action script, if the product list data becomes empty due to the prior delete, a new empty product prototype is inserted. The repeat table then presents one row of interface elements, so this extra insert ensures the user is never left with an unsightly empty table.
Last but not least, the actions scripts of both of the triggers above end with an xforms:setfocus action. This is because pressing a button, be it to add or delete an item from a table, transfers focus to the button. That's just how the web works. But the user's focus is not really on the buttons; those are just tools. The user's focus is on changing the table, so it is a better user experience to push the focus back to the repeat table.
In today's cognitive computing products and techniques, the perception of greater intelligent responsiveness comes not so much from having true explanatory power, but rather just having strong predictive power over increasingly chaotic and larger data sets.
To give us a third point of reference beside linear regression and neural nets, I'll use some other terms to bring the focus to natural language processing. In 2011, the IBM Watson system demonstrated greater intelligence than the best human opponents in the domain of linguistically challenging factual Q&A. This was based on the ability to quickly produce high confidence answers from a large corpus of unstructured information in response to challenging questions.
The linguistic product that is now based on that system is called the IBM Watson Engagement Advisor. As with other cognitive computing techniques, the product must first be trained to be an effective system in the target domain. The corpus of unstructured information often takes the form of documents, such as instruction manuals, technical reports, journal articles, and wiki pages. During training, the most important entities and relationships expressed in the documents are identified and stored in order to expedite later search and retrieval during Q&A interactions with users of the system. The identification process within a document is often called annotation, and the annotation and storage processes together are called ingestion.
The most important concept in understanding the training is to understand what really drives the identification, or annotation, of the documents. It's simple, really. It's a Q&A linguistic product, and the annotation and ingestion expedite the production of the A's in response to the Q's, so it is imperative to have a strong and large representative sampling of the potential questions in order to train and test the efficacy of the system. The questions encode the key concepts (e.g. entities, relationships and so forth) to which users of the system are interested in getting answers. Annotators for these key concepts are developed and, during ingestion, they are executed upon the documents.
This is the very basic level of explanation about how the linguistic product would learn, or be trained, to be an effective cognitive computing system for a domain, and future entries will dive further into this topic. For now, let it suffice to say that ingestion and training results in a system capable of producing answers from the corpus in response to questions like those used in training.
During a run-time Q&A session with the system, the user begins by posing a natural language question. The question is first analyzed to find the key concepts, and then a multiphased approach is used to dig up the best results from the ingested corpus content. As with training, there's a lot more to be said over time about how the run-time Q&A works, so more interesting future entries to come, and in fact, it's intrinsically related to the training anyway. To conclude and tee up these future entries, I'll say the high order bit here is that a trained Q&A linguistic product seems somehow more intelligent than a linear regression or even a typical neural net application. Why is that? To get a bit more background for that explanation, I'd encourage you to visit or revisit a few of my earlier blog entries about cognitive computing. Compare your perceptions of the intelligence of the minimax algorithm in  with the linear regression method in , and compare  with the neural net in . What's changing?
Ubiquity XForms version 0.7 is now available. In addition to contributing to the recent advancement of XForms 1.1 to Proposed Recommendation, this new version has many new features, fixes and highlights. Personally I'm happiest with the progress on implementing submissions, but that may be a bit of bias because I contributed some code to deal with aspects of submissions such as serialization, validation and relevance pruning. On the other hand, I contributed code in other areas, so maybe it's just that I'm biased toward any improvements to our ability to support the XRX architecture and connect the XForms client to server-side services.
And this blog entry would not be complete without my mentioning a special word of appreciation to the folks at webBackplane for all their contributions in general but especially for the Ubiquity XForms rollup system, which consolidates the many files of the processor into a single file to be deployed on your server. I have an internal project right now that uses the rollup, and it provides us with the high level of performance we expected/required in our applications.
Full details of the version 0.7 release can be found here: <http://code.google.com/p/ubiquity-xforms/wiki/Release0Point7>
Check out this developerWorks article for a step-by-step guide to deploying a DB2 web service and then consuming that web service using the Lotus Forms Designer.
Once the WSDL for a particular DB2 web service is pulled into your Lotus Forms Designer, you select and autogenerate a data instance and a specific service, drag-and-drop the data instance onto the design canvas to autocreate the user interface, and then generate the run-time XForms submission for the service. The article above shows exactly how to do each step.
A smart XForms client and a smart interface to the server database mean no custom code in the middle. Thus, computing power is made available to a broader class of IT knowledge workers who require only forms and database skills. Process democratization in action.
For a very long time, the XML language underlying Lotus Forms (called XFDL) has offered an option element called <format> to describe data type, validity constraints and presentational formatting for the (valid) values of form controls (called XFDL items, like text entry fields, selection popups, and so forth). Neither a whole Lotus Form document, nor the XForms instance data it contains, can be submitted to a server unless all relevant data type and validity constraints are satisfied.
However, the <format> elements are placed into the form controls, so the user interface layer must be articulated in order to create the run-time objects described by the <format> elements. This makes it harder to use Lotus Forms performance enhancing features like "on demand page loading" in a multipage XFDL document. It will also conflict with an increasing number of performance enhancing features over time. Suffice it to say, it would be better to express <format> data type and validity constraints as XForms binds in the form.
Sometimes this is easy to do. The following are a few examples. Suppose you have an XFDL <field> text entry element containing <xforms:input ref="data"> as the form control binding it to a data node named "data".
Change: <format> <datatype>integer</datatype> </format>
To: <xforms:bind nodeset="data" type="xforms:integer"/>
Change: <format> <datatype>float</datatype> </format>
To: <xforms:bind nodeset="data" type="xforms:double"/>
Change: <format> ... <constraints><length><max>10</max></constraints> </format>
To: <xforms:bind nodeset="data" constraint="string-length(.) < 10"/>
Change: <format> ... <constraints><range><min>18</min></constraints> </format>
To: <xforms:bind nodeset="data" constraint=". >= 18"/>
Change: <format> ... <constraints><mandatory>on</mandatory></constraints> </format>
To: <xforms:bind nodeset="data" required="true()"/>
These are easy mechanical changes to make that prepare a Lotus Form for use with performance enhancing features like "on demand page loading". The XForms binds are easy to create with the "Create Bind" or "Create Bind with Wizard" features in the Lotus Forms Designer's XForms Model Editor.
One aspect of <format>
option capability that is... somewhat less mechanical... to replace are constraints based on regex (regular expression) pattern matching. If you don't know what a regex is already, just do a web search for "regex tutorial" and spend an hour or two reading something that will save you days upon days of your life in the long run.
As an example, here is a simple XFDL <format> that applies a regex pattern limiting user input to zero or more alphanumeric characters:
<format> ... <constraints><patterns><pattern>[A-Za-z0-9]*</pattern></patterns></constraints> </format>
The * means zero or more characters, the brackets are used to express the range of characters, and subexpressions like A-Z and 0-9 provide subranges of the acceptable character range. Regex is a lot more powerful than this, so you really should read that tutorial if you needed this explanation. For example, you can express multiple alternative patterns directly in regex without exploiting the obvious ability to use multiple <pattern> elements (the availability of that option is simply because it is easier to read than really long single regex patterns).
Anyway, to replace a pattern-based format constraint with an XForms bind, you would use a type model item property that references an XML schema type library entry. In essence, you can build XML schema declarations in your form to declare types even if you don't use XML schema to define the element structure and the data types of the instance data. Here's what it looks like:
<xforms:model> <schema targetNamespace="http://test.org" xmlns="http://www.w3.org/2001/XMLSchema"> <simpleType name="alnum"> <restriction base="xsd:string"> <pattern value="[A-Za-z0-9]*"></pattern> </restriction> </simpleType> </schema> <xforms:instance xmlns="http://test.org"> <data> <node></node> </data> </xforms:instance> <xforms:bind nodeset="test:node" type="test:alnum"/> </xforms:model>
The inline schema declares a datatype named alnum (for alphanumeric) and indicates it is any string that can be restricted to the same pattern you saw in the XFDL format constraint. The XForms bind then associates that type with the data node. If you bind a field item's <xforms:input> to that data node, you'll see that the user input is indeed restricted to strings that match the pattern.
Check out this cool video that will help you understand how and why your business processes need to lose the paper and get on board with dynamic, interactive electronic forms... powered by the XForms standard of course!