The hit rate on my last blog announcement about IBM Forms 4.0 makes it clear that the release is pretty big news. I'm very pleased about that because it is a *major* release, both technically and in terms of the corporate branding. Corporate branding is not a decision lightly made, so it took the time it took, and meanwhile we didn't want to delay the technical features hitting the market, so you'll see some references to "Lotus Forms" under the covers of the main IBM branding. Not to fear, we're working on that for the next release.
Meanwhile, now that IBM Forms 4.0 is released, I've made a number of demonstration videos to show you key aspects of the IBM Forms Design and Run-time products. They're available on YouTube in HD, and they're broken into small segments, so you can really take them in pieces according to your interests.
To start, here's a 5-part series that takes you through run-time features with a demo form that ships with the product. These videos help you understand the value proposition of IBM Forms as well as new features of the release that will change your mind about what forms are and what they do.
- The New IBM Forms 4.0 - Introduction, Web 2.0 Look and Feel, AJAX Updates (LARGER YouTube HD)
- The New IBM Forms 4.0 - Dynamic Tables, Form Parts, Integrating Business Analytics (LARGER YouTube HD)
- The New IBM Forms 4.0 - Dojo Controls in Dialogs and on the Forms, Integrating Web Analytics (LARGER YouTube HD)
- The New IBM Forms 4.0 - High Precision Layout, Digital Signatures, Completing a Business Transaction (LARGER YouTube HD)
We also now have several videos that show off the IBM Forms Designer. This release really raises the bar on what level of data interactivity can be created through design GUI configuration rather than grunge coding, and many powerful XML data processing features of the IBM Forms Designer are directly attributable to W3C XForms. This includes Design by XML Schema, Dynamic Tables, Web Service Connections, Interaction Rules (e.g. constraints), and Wizard Creation:
Finally, here's a video series on the new FormParts feature (I didn't make these, but they're excellent and really belong in this post). FormParts allows you to create components that can be reused and kept up to date across a forms library.
If you want to know more about IBM Forms 4.0, then come to our wiki
or to our website
, where you can get more information, case studies and free trial downloads. Thanks for watching!
new release of our forms software, version 4.0, is now only a few weeks from shipping. And as of this release, the product line will be known as IBM Forms
! This is an incredibly important indicator of the strategic value IBM sees in the Forms business as a key component in building a Smarter Planet. The feature set coming in this new major release combines the best of Web 2.0 client application behaviors and design experience with the traditional strength of interactive XML data collection for which the prior releases our product line are well known. IBM Forms documents are interactive web application instances that have many traditional capabilities that we have been building into them since 1993.
- They can always be serialized to provide a user with their own saved copy of their web application experience, to archive, to email, or to pass to the next step of a business process or workflow. They can be reinstantiated at any time to continue the interactive fill experience.
- They can be instantiated directly into a web browser without using a plugin, and they can also be instantiated in a client application for an offline fill experience, dramatically simplifying IT maintenance and platform support, .
- They provide a multipage high precision visual layout with comprehensive accessibility support and localization
- They operate over XML payloads that directly conform to the needs of back-end business functions, enabling straight-through XML processing systems.
They enable rich web application interaction and behaviors based on the W3C XForms 1.1
- They allow users to attach supporting documents into the form in support of their business function, including spreadsheets, word processing files, images, videos, etc.
- They secure the business function implemented by the form with a comprehensive interactive document digital signature system that includes sectional signing, multiple signatures, and overlapping signatures, all of which protect not just the form data, but the form behaviors, the form appearance and the attached documents as well.
Now lets add to that what's coming in the new release:
- The ability to drive customer satisfaction and employee productivity through more efficient, compelling interfaces that can include
- gradients, border styling, thematic color control, combined text and images
- AJAX-driven XML web service updating without web page refreshing,
- formatted HTML text for bold, italic, underline, text colors, font control, lists, hyperlinks, etc.
- Embedded custom HTML extensions and popup dialogs that can show maps or movies or even interact with the running form.
- The ability to deliver integrated solutions faster and spend less time and money maintaining them through a comprehensive new set of Design Experience features including
- Form Parts that make it easy to reuse and maintain common components consisting of one or many user interface controls and corrresponding behavioral markup
- Data-driven form design with XML schemas that define back-end data formats; data type sensitive automatic user interface creation, including tables and label annotations
- Web service integration wizard that provides point-and-click, drag-and-drop web service consumption and user interface mapping.
- The "wizard of wizards" that enables rapid development or addition of a step-by-step wizard experience for a form
- SVN source code repository interaction, Integrated Design Time spell-checking, etc.
- The ability to use your Forms solutions how you want and where you want, including
- An automatic forms view portlet and support for Portal 7.0 and Mashup 2.0
- Updated browser and OS support and improved integrations
- Improved workflow system integrations with Filenet and Websphere BPM
- Webform Server support for the iPad
The real achievement here is not just these new features, but getting them to work well with all the traditional features. IBM Forms 4.0
, coming soon to a Smarter Web Application near you.
Forms exist to collect data from web users involved in business processes. Are you a business partner who wants to build solutions more quickly in order to make a higher margin? Then read on!
What if you could use a web browser to design not only the user interfaces of the multiple pages of a form, but also the whole solution for which it collects data?
Now, with IBM Forms Experience Builder, you finally can.
You can define the roles of users in the business process, and you can assign users and groups to those roles. You can even set up open roles whose users are defined dynamically during the business process once the right information is collected earlier in the process. For example, only once you take in a person's name can you access an LDAP service to look up his manager and then assign that person to the manager role for an approval step.
You can define the user interface of a Form, and have an automatic database created on the server side to store database records corresponding to completed instances of that Form. You can even define multiple Forms that work together within a solution that collects data according to different record schemas.
You can define the stages of a business process workflow that uses the Form or Forms to create and update database records. Stage transitions can branch forward, backward or even stay on the same stage to update a database record that still needs more work.
You can define access control for each workflow stage and determine which Forms, page, and UI elements are available in each stage.
You can even use the database records collected with one Form as a GUI configurable web service within the fill experience of a second Form. For example, you could have one Form of a solution that collects inventory data, and then use that data in a second Form that makes it possible to order from available inventory.
You can make the Form fill experience available within a portlet of an IBM Websphere Portal website.
You can create a solution with your web browser, you can save it to the server, you can hit Deploy in your web browser, and then your users can access the Forms of the solution from web links. If you later decide it is necessarily to add to or change the solution, you can edit the solution again using your web browser and hit Deploy again The data is retained for all the remaining form UI elements, and the database tables are altered as needed to make space to store data collected by any new form UI elements.
Via web links, users can access the list of database records collected by the solution. Only the records to which the user has access are presented. If you're the solution creator/administrator, you can get access to all the records. Whomever is given a link to view the records can also set up their own customized filters for the data, so a user can truly use the view as a business process task list, and even filter down to tasks of a particular type, from a particular person, having met or exceeded some value, etc.
Complete agile web solution creation.
Lose the custom coding, gain the market advantage, get IBM Forms Experience Builder now.
Industry solutions is an important area of endeavour for IBM. An industry solution is an IT asset that helps solve an industry-specific problem and is easily reconfigurable to meet specific needs of each client. A solution often helps a client to reach out to and interact with their own customers or users.
An important segment within industry solutions is called case management, which takes the view that a customer/user interaction pattern can be orchestrated by a case. The definition of a case includes data structure and data type definitions, metadata definitions, business process and user access rules, and other possible resources. Based on an initial request by a customer or user, the case management system instantiates the case definition, and the resultant case orchestrates the interaction to achieve the goal or goals implicit in the defined pattern. For example, a case management system could be used to orchestrate the means by which a customer makes and successfully completes a warranty claim for a defective product. The process would begin with collecting initial information about the defective product, about the defect, and about the purchase. The process would include determining the legitimacy of the warranty claim, providing basic support to qualify the defect and determine a course of action, and ultimately to effect a repair or replacement of the product.
A case management user interface is a collection of interactive components that collect data from a user and store it in the data structure of a case (an instance of a case definition). The presentation of the user interface is also affected by the metadata of the case. A common piece of metadata is an enumeration of the valid values that a data item may take. If such a list is available, then it would be presented in a dropdown menu or list box, and the input would be collected via list selection rather than by free-form typing in a text entry field. Other common metadata are boolean flags such as for indicating whether a data node is readonly or required to fill in a step of the case processing. The user interface components would be affected by enforcing the readonly property or providing a sensory indication of the required property. Still other metadata can define validity constraints, such as a numeric datatype or a minimum or maximum value or length. The user interface of a component associated with a data node would be affected by indicating whether the current data value is or is not valid according to the constraints. A case management asset would also typically forbid progress to the next step of the orchestrated pattern when the data associated with the current step contains invalid values.
The term “case management solution” has been used to describe a software solution that supports the design, deployment, execution and reconfiguration of case management assets. As the field of case management matures, the term “advanced case management” has emerged as a way to characterize case management assets that have advanced feature requirements, such as the requirement to collect many dozens, scores or hundreds of fields of data. Some examples of advanced case management include: home or car insurance claims, credit card charge dispute resolution, citizen-facing ombudsperson cases, contagious disease outbreak tracking, and management of complex medical or psychological treatment cases. IBM Case Manager (ICM) is IBM's advanced case management solutions.
A problem arises in advanced case management solutions with respect to the expected maturity of the user interface. The typical case management solution generates a user interface presentation layer for the data of a step using a simple linear columnar approach or a column of expandable stacks of related data values. The advantage of this approach is that it most easily adapts to a reconfiguration of the case management asset in which the data structure is amended to add or remove data nodes. However, there are a number of disadvantages to this approach. For one, it provides a one-size-fits-all approach to the user interface layout in which usability substantially degrades in quality as the size of the data set grows.
As well, larger data sets tend to correlate to more advanced requirements in an overall solution that a case management asset simply cannot begin to address using only a simple user interface approach. There are many such features, including creating multipage guided interview style wizards, creating mutiple print-style pages to reflect a “document of record” for the case, and of course adding the ability to digitally sign the “document of record” as a way to create a legally binding agreement or a record that can stand up to rigorous auditability requirements.
The new release of IBM Forms is the strategic IBM forms technology that now solves this problem for IBM Case Manager (ICM). An IBM Form combines an XML data structure with a template describing interaction behavior rules and a comprehensive user interface definition. The how-to for connecting an IBM Form to an ICM solution is as simple as going into the IBM Forms Designer, right-clicking on any number of XML data nodes, making them “public” and giving them public names equal to the ICM case property names they must map to, and then ataching the IBM Form into the ICM solution.
The IBM Form can then be used in the Case eForm widget anywhere in the ICM solution where the Case Data Widget would have been used. During execution of a case under the ICM solution, the case property values are automatically injected into the XML data of the IBM Form as it is rendered to the user, based on the public data mapping mentioned above. When the user completes or saves the form, the updated data values from the IBM Form are injected back to case properties so that the Form and the Case are in synch.
Oh, but the story is so much cooler than just adding high precision, multipage user interface control for the data. And it's cooler than having a “document of record” for the data that can be digitally signed. IBM Forms contains this fabulous technology for defining live interaction with XML. It's called... you guessed it... XForms. XForms manages not just XML data, but also metadata pertaining to each data node, such as the node's datatype, or whether the node is required, or whether it must be valid according to some constraint expression. XForms also allows a list of values to be associated with the input mechanism for a data node. Remember above where I said that case management assets define metadata just like this for case properties? Well, when you design a case solution that includes this metadata, and then map a XML node to a case property by assigning the public data name, the IBM Forms integration automatically injects not just the case property values, but also any lists as well as XForms binds for the metadata. During the form run-time, the XForms processor then automatically combines these XForms bind results with any metadata settings that might be defined within the form itself. In effect, there is a seamless bridge from ICM case processing to the user's interaction with the IBM Form.
Finally, this seamless bridge works in the other direction too, from the IBM Form back to the ICM case solution. In addition to synchronizing data updates from the form back to the case, the key lifecycle operations of saving or completing a form interaction are gated by a validation operation. IBM Case Manager delegates the validation operation to the IBM Form technology, which executes a validation operation based on the behavior of an XForms submission. This means that non-relevant nodes are automatically pruned from the validation, and the validation result is the sensible combination validation rules injected from the case solution and validation rules expressed directly in the form.
Netting it out, IBM Forms is a first-class citizen of IBM Case Manager solutions, and the case data and metadata of an IBM Case Manager solution are handled as first-class citizens of the user experience provided by an IBM Form.
Information system architects, and even vertical industry standardization or government IT standardization bodies, derive immense value from XML schema definitions that describe the data structures and data types expected in valid transactions of the information system. However, these assets are focused on defining the completed transaction. In many a presentation, I've talked to potential customers about how IBM Forms documents express far more value because they are about the human interaction that takes place during the fill experience needed to produce those completed transactions.
As a bit of an aside, it's important for the technically minded reader to be familiar with the "sell" side of this equation. It is important to be able to easily justify technology adoption decisions with business owners who need to be able to understand how you will be able to better server your customers, reduce development and maintenance costs, increase competitiveness, eliminate vendor lock-in, etc.
In this blog, I'd like enumerate various benefits that you get from the standards basis of IBM Forms documents and their implementations, but I'd also like to separate the enumerations into two lists: 1) benefits above using XML Schema that you get just from using the XForms markup within an IBM Form, and 2) the benefits you get from the XFDL processor (XFDL is the XML vocabulary that IBM Forms provides as a presentation layer for XForms).
Firstly, an XForm is a clearly a superset of XML Schema since an XForm can incorporate the XML Schema if it is available and provide its validation information set to the fill experience and the submission experience. But an XForm provides many additional benefits, including the ability to:
Express data validation constraints that are based on other XML data values entered during the fill experience.
Automatically compute data values based on other input entered during the fill experience, rather than requiring users to perform the error-prone task of calculating and inputting summative results manually.
Describe the inputs, outputs and triggering mechanisms of web services to invoke, such as to use entered XML data to obtain database query results that fill other parts of the form or to invoke server-side validation or calculation logic from a business rules engine.
Control whether a data value is readonly or whether the user can enter a value, for example based on conditions related to a business process step or an access rule.
Express user interface controls that indicate which XML data values will be available to the presentation layer for input or output.
Conditionally show or hide the user interface controls (and hence their presentation) in response to conditions, such as those that may relate to a business process step or access rule.
Define a step-wise fill experience to guide a user systematically through incremental successes in entering valid data.
Provide customized help and validation error messages to help users fix data input errors when they occur.
Provide prompting label text to be associated with the presentation of each input or output control, both visually and aurally (for accessibility)
Associate a selection list with any XML data node in a way that constrains user input capability to the provided list
Define labels, help messages, validation error messages and selection lists in more than one human language within the same form, thereby ensuring citizens receive the same form logic and interaction behaviors regardless of which official language they select to request government services
Associate repeated data with a logically tabular set of user interface controls and encode the means by logical rows of the table are added or removed in response to insertions or deletions of data
While the above benefits indicate what additional behaviors and features of XML forms can be expressed above those that can be expressed by an XML Schema alone, XForms is also more interesting for standardization of a forms repository due to what it does not express. An XForm does not rigorously bind its many behavioral benefit to a specific presentation layer of the form. The intention of this language architecture was to address multimodal requirements of forms applications, e.g. rendition on a desktop, tablet, smartphone, telephone call-in voice service or instant messaging interaction. Different presentation layer implementations can address these requirements, and such implementations can even be provided by different vendors. The XForms working group also anticipated that there would be a wide array of varied technical requirements for presentation layers, and this language architecture allows XForms to be used with fundamentally different XML presentation languages that address these disparate requirements. Examples range from ODF for flowing text with fill-in-the-blanks fields to XFDL with its high-precision contract-style layout capability.
Due to the above mentioned language architecture, XForms markup does not comprise a well-formed XML document until it is incorporated into a presentation layer XML document. XFDL in the XML vocabulary used in IBM Forms to provide a presentation layer for XForms.
Of course, in a "baseline standard" version of a form, the default presentation layer associated with the XForms markup can be minimal in nature so that the only benefit is to provide a well-formed XML document to host the XForms markup. Interestingly, once you have this from XFDL, then the result is in fact an XML document, and so it can be processed by readily available XML processing tools like XSLT. These XML tools can be usd to automate creation of different versions of the baseline form that may have a richer presentation description, alternative natural language usage, or even different presentation layer markup. In addition, various consumers could use simple XML tooling to rebrand the forms.
The Extensible Forms Description Language (XFDL) in an XML vocabulary describing the presentation layer and richly interactive behaviors of modern web-based electronic forms. This XML vocabulary was first introduced to the W3C in 1998 (http://www.w3.org/TR/NOTE-XFDL
) and over the years, the versions of the language have consumed XML data-processing components of the W3C XML technology stack as they have been standardized by the W3C, especially including XML Schema, XForms, and XML Signatures. XFDL is a royalty-free open format whose current version specification, which can be obtained from -
XFDL is a host XML language for the XForms standard, and so the many benefits of XForms described above are inherited by XFDL. The implementations of XFDL as a presentation layer language add the following benefits over the core XForms processing:
very high precision control over the layout and rendition of the user interface
integration of XML Signatures with both XForms and a high precision user interface
comprehensive treatment for accessibility, localization and language support
dynamically computable presentation layer effects based on data value changes
multiple page support and ability to easily integrate web analytics within the form
ability to print or generate a PDF snapshot of the form or of just the agreement style pages
built-in ability to save the form locally and reload it later to continue the fill experience
a built-in file attachments folder system, including the ability of digital signatures to protect files attached into the form
a “public data” map for mapping XForms instance data to business process/workflow variables and case management properties
integrations to standard application server processing of form results and to run-time processing by JSR 168 and JSR 286 compliant portlets
the choice of zero-install operation within a web browser, using a server translator module
the choice of a client install to support both offline and online processing
In a larger sense, though, this only part of the benefit derived from XFDL. Still more benefit is derived from the availability of the XFDL forms visual design environment, which gives form authors integrated access to GUI features for XForms, for XML Signatures, for schema-driven design, for web sevice connections, and for the XFDL language benefits included in the list. The design environment even includes a converter that helps preserve the layout of PDF Forms that are brought over into XFDL. Finally, the XFDL forms visual design environment also provides features for maintaining a collection of forms, such as SVN and other team repository plugins and management of form parts.
I've recently realized that there are a few milestones to celebrate all at once here: This blog turned 5, my W3C XForms group age is 10, and this is the 100th exciting episode of the IBM Forms dW blog. I took a look back at what you, the readers, seem to like best, and those 2000-3000 hit entries are at the intersection of strong technical content and the confluence of open standards. How serendipitous!
The power and value of the XFDL markup language underlying IBM Forms comes from the way it brings together all the features of open standards needed to design sophisticated solutions, including XML, XML Schema, XForms, XML Signatures, and XPath. In this entry, I'd like to continue the schema driven design story by talking to you about the information architect's half of the equation. Specifically, we'll cover the XML Schema constructs that the IBM Forms Designer converts into various user interface controls like XFDL popups skinning XForms select1 elements and XFDL tables skinning XForms repeat elements. If you haven't yet watched the short demo videos mentioned in the prior blog entry, take a moment to do so now because I'd like for us to dig into parts of the "Medical Preapproval" schema used in that demo. The top-level element definitions for the schema look like this:
<element name="medicalPreapproval" type="health:medicalPreapprovalType"/>
<element name="patient" type="health:patientType"/>
<element name="contactInfo" type="health:contactInfoType">
<element name="details" type="health:detailsType"/>
In this sample, the data needed fora medical insurance preapproval consists of some basic patient data like name and date of birth, contact information like email and address, and details of the medical procedures required. When the IBM Forms Designer is provided with this schema, it can generate an XML data conformant to the schema in an XForms instance element, and then the form author can drag and drop the XML elements in the instance view onto the Design canvas. For example, if the form author drags and drops the "patient" element, then the IBM Forms Designer creates an XFDL pane that skins an XForms group element. Within the group, various XFDL items like text input fields and calendar pickers are created and mapped to the child elements of the "patient" element so that the textual content can be collected by the form. So you can get an idea of how to write XML schema for structured content, here's a part of the patientType definition:
<element name="lastName" type="xsd:string"/>
<element name="firstName" type="xsd:string"/>
<element name="SSN" type="health:serialNumberType"/>
<element name="dateOfBirth" type="xsd:date"/>
At design time, text entry fields are created for these data items, and any specialized schema rules, such as the regex pattern definition for the SSN, are applied at XForms run-time. In case you're curious, here's how you'd set up a regex pattern, in this case a very simple one that takes 9 digits but also allows an empty string:
For data types like the xsd:date (above) and xsd:boolean, you will get an alternative form control, such as calendar picker or checkbox. The IBM Forms Designer will even generate a calendar picker if you have a type derived from xsd:date. For example, you may want your schema to allow a date or an empty value. So, if you change the type of the dateOfBirth element above to health:dateOrEmpty, then you can use the declaration below to achieve that effect:
<element name="dateOfBirth" type="health:dateOrEmpty"/>
By default, the label text generated for each element is based on the element name. Camel-casing is used to determine when spaces should be added to the label. For example, the default label generated for the "patient" group is "Patient", and the default label for the "lastName" element is "Last Name". However, the information architect is provided greater control over the labels in the schema. In the example above, an appinfo annotation is used to define the desired label of "Contact Information:" for the contactInfo element. The same appinfo annotation can be used to control the labels of the UI controls bound to leaf (text content) nodes. For example, below is a definition that could go into the patientType definition, but it provides an alternative label for an element that has a name that is perhaps medically accurate but perhaps less desirable:
<element name="sex" type="health:genders">
Continuing with this example, the obvious choices of Male and Female would be defined in a schema enumeration, like this:
An enumeration like this in the schema causes an XFDL "popup" item (a dropdown menu) to be generated, along with an xforms:select1 element. The dropdown menu entries are Male and Female, and the user's choice at run-time places the word Male or Female in the "sex" element. But maybe you'd prefer to have the data be based on codes like M and F, but still show menu entry labels like Male and Female. Again, the appinfo annotation comes to the rescue:
I've used Gender in these examples for brevity, but you can use enumerations to describe other lists, such as for States and Provinces. The IBM Forms Designer will generate an XFDL popup item by default, but the Designer also lets you right-click convert the popup to other user interface items that can legally skin an xforms:select1, such as a radiogroup, checkgroup or list box.
Also, sometimes what you need is a combobox, so that the user can access a dropdown menu but also have the option to type an open-ended answer. For example, the open-ended response for a gender might provide medically relevant information. The information architect can control this in the XML schema by creating an open-ended enumeration, i.e. an enumeration unioned with a string:
Last but not least, we have the mechanism that signifies the need to create an XFDL table and xforms:repeat. Here's an excerpt of the health:detailsType definition:
<element name="reason" type="xsd:string"/>
<element name="recommendedProcedures" type="health:recommendedProceduresType"/>
<element name="procedure" minOccurs="0" maxOccurs="unbounded">
<element name="name" type="xsd:string"/>
<element name="qty" type="health:positiveIntegerOrEmpty">
<element name="price" type="health:decimalOrEmpty"/>
<element name="lineValue" type="health:decimalOrEmpty"/>
<element name="total" type="health:decimalOrEmpty"/>
Within the recommendedProcedures element, the schema indicates that a "procedure" element can appear more than once (maxOccurs is unbounded in this example, but any value greater than 1 will also work). The IBM Forms Designer responds with the XFDL table editor. It reads the child elements in the sequence to determine the default set of table columns. Notice the use of the appinfo annotation to control the column header label text. When the form author drags and drops the details element, XFDL fields sking xforms:input elements are created for elements like reason and total, but the table editor also produces an XFDL table and xforms:repeat whose nodeset attribute contains the XPath needed to bind to however many procedure elements are present in the data at form run-time. The table editor also generates XFDL buttons containing xforms:trigger elements that provide the "add row" and "delete row" capabilities using xforms:insert, xforms:delete and xforms:setfocus actions.
Now hopefully you have a more complete picture of how IBM Forms concentrates the combined value of all these standards-- XML schema, XPath, XForms, XML-- into one solution creation machine. Well, that may have been a tad long, but we're celebrating three milestones here :-) Besides, the length of this entry is a simple reflection of how much cool stuff you can now exploit in your XML Schemas to help your form authors create the forms that feed the right XML to your backend systems.
One of the New IBM Forms 4.0 Demonstrations
has just won the top-rated video award at Lotusphere
. This is a fantastic victory for the Websphere Portal segment of Lotus, which focuses on products like IBM Forms that create and provide exceptional web experiences. It is also a victory for the W3C XForms standard
since the Wizard Creator
featured in the video is, as far as I know, the first point-and-click design experience for an XForms switch. This feature exemplifies the principle that declarative markup languages result in more powerful application design environments because they express what the author wants. With an imperative languages, a design environment must either operate at a much lower level (less powerful) or do some wicked reverse engineering/pattern matching to discern what the author wanted from how he did it.
So, XForms has definitely been my friend in helping to create an award-winning feature in IBM Forms. Wanna meet my newest friend? Here we are, just me and Watson, celebrating our victories at the closing session of Lotusphere 2011:
To put it in Watson's terms:
Category: Lotusphere Best Demo Video Winners
Answer: An IBM Form that uses XForms to express Wizard interfaces for forms.
Question: What is a Smarter Web Application?
MC: Congratulations Watson, you're ready for prime-time! Wanna join my social network?
There are a number of new XPath extension functions available to XForms developers in the latest release of IBM Forms, and I'd like to draw your attention to two of them: eval() and eval-in-context(), and they are wicked cool!
The function eval(expr) evaluates expr in the context of the function call and returns the result. The eval-in-context(expr, contextExpr) function does a similar thing, except it first evaluates the contextExpr and uses the result as the context for the main expression. This is desparately needed for XPath 1.0 expressions to eliminate the infestation of pesky ".." operations that typically occurs. I've used it, rather than eval(), in the samples below.
One use of these functions is to enable the powerful capability to let XML data carry sophisticated dynamic metadata, which can then be implemented and enforced with singular XForms bind elements that attach the semantics of the metadata whereever it is found in the data.
It turns out that the xsi:type attribute from XML schema is already a rudimentary version of the metadata idea we're pursuing here, so the question becomes what if you could do it for all of the juicy metadata that XForms contains, like calculated data values, data validity constraints, and so forth? Let's look at what this "decorated" data might look like for a simple expense report:
<total value="quantity * price"/>
constraint="total < 10000"/>
It is really easy to use XPath capabilities to find all elements having a value attribute and then to bind an XForms calculate formula to those elements, and then eval-in-context() is used to determine the result according to whatever expression is given in the data, like this:
<xforms:bind nodeset="descendant::*[@value]" calculate="eval-in-context(@value, ..)"/>
The nodeset expression starts with descendant::* to explore all elements of the data, and then the predicate [@value] selects all elements that have a value attribute. For each such element node, a calculate formula is bound to it by the XForms bind. The eval-in-context() call uses ".." to go up a level so that the formulas in the value attributes can omit "../", e.g. so the expression can simply be "quantity * price" rather than "../quantity * ../price".
Similarly, a data validation constraint can be attached like this:
On the expense report data above, this binding evaluates the constraint expression attached to the total element, and then converts the result to a boolean. Due to the constraint, the expense form data cannot be submitted to a server for processing unless the total expense is less than 10000.
In other scenarios, you may want to control the other metadata properties like relevant, required and readonly. Here's an example of data where relevance and requiredness control is needed:
<parent relevant="age < 18" required="true">...</parent>
The xsi:type assignment for age already works in XForms without needing a bind. The required setting for name is statically true, not dynamically changeable, so it is very handy to be able to allow either a static boolean value or an expression, like this:
required="@required='true' or boolean-from-string(eval-in-context(@required, ..))"/>
Technically, the required property on the parent element is conditional on the age value, but that is an automatic feature of XForms, i.e. nodes marked required are only required if they are relevant. It should not be too surprising to see that the bind for relevance looks like this:
relevant="boolean-from-string( eval-in-context(@relevant, ..) )"/>
At this point, it's probably safe to leave the readonly bind as an exercise for the reader :-)
Recently, I was experimenting with one of the features planned for the next version of XForms. The feature is the iterate attribute for XForms actions, which will perform a for-each loop operation based on a nodeset obtained from the xpath expression in the iterate attribute value. XForms 1.1 already has a while loop, but iterate makes many data processing loops easier to write and also more performant (subject of a future blog). There were lots of iteration use cases to choose from, but I decided to experiment with sorting because it is a well-known benchmark algorithm.
Before we go any further, let me say up front that XForms action scripting is intended for very lightweight data manipulation, like adding or deleting a data node corresponding to a table row or copying data results to or from the SOAP envelopes of a web service. By the time you get to nested iterations like those needed sorting, you should be considering alternatives expressed in full-blown imperative languages available in the information system within which the form is being used. For example, in the case of sorting, it is a better idea to request sorting in the database query whose results are returned from a web service into your form so that your form logic does not even have to do the sorting.
So, with the disclaimer out of the way, let's abuse the technology a bit to get a better sense of what is feasible in those customer-needs-it-yesterday circumstances. It turns out that XForms 1.1 does allow full nodeset processing in the insert action's origin attribute and the delete action's nodeset attribute. Without even needing the new iterate attribute, this is just enough iteration capability to perform efficient sorting -- so there are some kinds of iterations that can be done now without the iterate attribute.
We're going to do a divide-and-conquer "partition" sort that I personally created as a university freshman after my 1st semester instructor told our class that linked lists could only be sorted slowly. At the time, the usual computer languages only allowed static allocation for arrays, and even though I didn't know what a "quick sort" was, I had seen the light of dynamic allocation, and I was never going back! I later learned how great a merge sort is on a linked list, but the effort of turning an array quicksort into a linked list partition sort comes in handy now because a merge sort cannot be efficiently expressed in XForms until the iterate attribute is added.
The way a quicksort works on an array (or subarray) is that you pick a random element to be the 'pivot' value. Then you run two index variables at the same time, one from the start of the array upward and the other going from the end of the array downward. The 'up' index is advanced until it finds a value greater than the pivot, the 'down' index is decremented until a value less than the pivot is found, and then the values at the 'up' and 'down' locations are swapped. This keeps happening until 'up' and 'down' meet somewhere near the middle of the array. At this point you've partitioned the array into a subarray of values less than the pivot value and a subarray of values greater than the pivot value. The quicksort is then invoked recursively to sort both subarrays.
The main challenge with this approach is the 'down' index, which is a reverse iteration. In a singly linked list, you can only go forward. XForms insert and delete actions have a similar limitation: they can only identify a nodeset of nodes to insert or delete, but not really a direction of iteration. But the important bit is what the quicksort is doing, not how it is done. Think of the list content as being completely messy, and each partitioning stage must make it somewhat less messy by dividing the content into a partition of lesser elements and a partition of greater elements. Then, the next partitioning stage is invoked recursively to do a better job of cleaning up the mess within each partition.
Let's explore this concept by sorting a list of elements, such as sorting a list of <person> elements by a <lastname> child element. We begin by copying the list into an initial partition element of a temporary instance called 'sortdata', like this:
1) <xforms:insert context="instance('sortdata')" origin="instance('partition')"/>
2) <xforms:insert context="instance('sortdata')/partition" origin="instance('data')/list/person"/>
3) <xforms:delete nodeset="instance('data')/list/person"/>
Next, we initialize the random number generator so we can randomly select pivot values for all the partitioning stages:
4) <xforms:setvalue ref="instance('sortpivot')" value="random(true)"/>
Next, we start up a simple while loop that continues to process partitions until none are left.
5) <xforms:action while="instance('sortdata')/partition">
Within the loop, we grab the last partition from the sort data and determine whether it is non-trivial or trivial (only 1 or 2 elements). A non-trivial partition is subjected to further divide-and-conquer processing.
5.1) <xforms:action if="count(instance('sortdata')/partition[last()]/*) > 2">
The first part of dividing and conquering is to create a new empty <partition> element, which we obtain from an instance that expresses a template empty partition element.
5.1.1) <xforms:insert nodeset="instance('sortdata')/partition[last()]"
Step 5 and step 5.1.1 are more interesting than they seem at first. The list of partition elements in the sortdata actually implements the recursion stack, and we just pushed a new element into that stack at the second-to-last position. Because we have an explicit stack, we only need a loop in step 5 to implement recursion.
The next thing we do here is grab a random last name to serve as a pivot value for the partitioning. The first setvalue just picks a random location, and then the second step uses the location to get the value. Notice also that I use * rather than partition before the [last()] predicate because the sort data only contains partition elements, so there is no point in doing a name test for partition.
5.1.2) <xforms:setvalue ref="instance('sortpivot')"
value="1+floor(random(false) * count(instance('sortdata')/*[last()]/*))"/>
Now the magical part happens. All elements in the last partition whose key element (lastname) is less than or equal to the pivot value are moved to the newly created second-to-last partition. By combining the nodeset processing capability of XForms insert and delete actions with the predicate-based node selection capability of XPath, the matching nodes can be selected and moved using two single XForms actions, i.e. without using an XForms while loop.
5.1.3) <xforms:insert context="instance('sortdata')/*[last()-1]"
origin="instance('sortdata')/*[last()]/*[compare(lastname, instance('sortpivot')) <= 0]"/>
<xforms:delete nodeset="instance('sortdata')/*[last()]/*[compare(lastname, instance('sortpivot')) <= 0]"/>
If the last partition is now empty due to the move operation in step 5.1.3, then the new second-to-last partition received all its elements. If all the moved elements are equal to the pivot value, then we can output them back into the original data list and then remove the last two partitions. Note that the insert is configured to prepend the elements into the data list, and we're copying them from the last non-empty partition, which has the elements with the greatest key value.
5.1.4) <xforms:action if="count(instance('sortdata')/*[last()]/*) = 0">
<xforms:action if="not(instance('sortdata')/*[last()-1]/*[compare(lastname, instance('sortpivot')) != 0])">
<xforms:insert context="instance('data')/list" origin="instance('sortdata')/*[last()-1]/*"/>
<xforms:delete nodeset="instance('sortdata')/*[position() >= last()-1]"/>
</xforms:action> <!-- End of 5.1 non-trivial partition handler -->
Now, we've finished with the non-trivial partition handler, and we turn our attention to processing a trivial partition containing at most 2 elements. The content of the partition is moved to the original data list and the partition is removed. Again, note that we're processing the last partition, which has the greatest key values, and the insert prepends to the data list, so the sorted data list starts with the greatest values and grows as lesser and lesser values are prepended over time as all the partitions are processed.
5.2) <xforms:action if="count(instance('sortdata')/*[last()]/*) <= 2">
If the partition contains two elements and the greater one is first, then it is removed from the partition and put in the original list:
5.2.1) <xforms:action if="count(instance('sortdata')/*[last()]/*) = 2 and
instance('sortdata')/*[last()]/*/lastname) > 0">
<xforms:insert context="instance('data')/list" origin="instance('sortdata')/*[last()]/*"/>
Now the partition contains either no elements, one element, or two elements in sorted order, so we move the content to the data list and then delete the partition.
5.2.2) <xforms:insert if="count(instance('sortdata')/*[last()]/*) > 0"
</xforms:action> <!-- End of step 5.2 trivial partition handler -->
</xforms:action> <!-- End of step 5 recursion while loop -->
As a final note on all this algorithmic fun, the question arises whether this sort achieves optimal O(N log N) performance. The answer is no, not quite, due to hidden costs of data instance management and data node selection. However, the sort will be much faster than a "simple" sort because it does perform only O(N) XForms actions.
Continuing with the amazing stuff you can do with the eval() function: You can use it in a user interface binding to enable your form to programmatically control what the user sees.
As an exhibition of this capability, I'll give you the pertinent parts of an XML editor form that dynamically adjusts to XML structure and allows you to edit the content of any leaf nodes while giving you link buttons to allow you to drill deeper into element subtrees as well as a "back" button to go to the parent of any subtree whose leaves you may be editing. It starts with an XForms repeat, like this:
<xforms:repeat nodeset="eval(repeatexpr)" id="XMLEditor">
The repeat expression is computed by the form and is changed by user actions that drill deeper into the XML tree or go back to parent elements. The repeat expression will end with "/*" so that the controls in the repeat will show the children of whatever node the repeat expression selects before the "/*".
For simplicity, I've put the XML data to be edited as the first element of the instance that also manages the calculation of the repeat expression, but you could do this as two separate instances instead. Here's the instance structure I used in this example:
<xforms:instance id="data" xmlns="">
<Item-Cost type="currency" value="Price * Quantity"/>
<Subtotal type="currency" value="sum(Order-Details/Item/Item-Cost)"/>
<Tax type="currency" value="Subtotal * 0.05"/>
<Total type="currency" value="Subtotal + Tax"/>
The first element could be anything, but I used a "purchase order" data structure, so this form will magically morph into a purchase order editor. Further, it should now be clear why in the last blog I concentrated on data that carried its own formula calculations and data validation rules. If I replace the Purchase-Order element above with the loan calculation data below, then this same form will help calculate your monthly payment on a loan:
<Loan-Application label="ACME Used Car Loan Application">
<Name required="true">John Q. Public</Name>
<Address required="Name != ''"></Address>
<City required="Name != ''"></City>
<StatProv required="Name != ''" label="State/Province"/>
<PostalCode required="Name != ''"/>
<Employer required="../Principal > 20000"/>
<Principal required="true" constraint="Principal > 0 and Principal <= 50000"></Principal>
<Duration required="true" constraint="Duration > 0 and Duration <= 84"></Duration>
<Interest-Rate constraint1="Interest-Rate > 0"
constraint2="Interest-Rate <= 5+floor(Duration div 12)"></Interest-Rate>
<rate hidden="true" value="Interest-Rate div 1200.0"/>
relevant="Monthly-Payment > 0"
value="choose(rate > 0,
Principal * rate div (1.0 - power(1.0 + rate, -Duration)),
Principal div Duration)"
relevant="Monthly-Payment > 0"
value="Monthly-Payment * Duration"
Within the repeat, we can use different kinds of form controls to be responsive to the identified types of data and also to the issue of whether something is an input or an output based on whether it has a computed value. Here are two examples at the XFDL+XForms level:
<xforms:input ref="self::node()[not(*) and not(@value) and @type='currency']">
<xforms:label><xforms:output value="choose(./@label, ./@label, concat(translate(local-name(.), '-_', ' '), ': '))"/></xforms:label>
<xforms:output ref="self::*[not(*) and @value and @type='currency']">
<xforms:label><xforms:output value="choose(./@label, ./@label, concat(translate(local-name(.), '-_', ' '), ': '))"/></xforms:label>
<display compute="xformsenabled=='on' and value != 'NaN' ? 'on' : 'off'"/>
In the predicates of the form controls, "not (*)" ensures that these form controls are only relevant if the data node is a leaf that is to be filled with character content. The "value" attribute in the data provides a calculation formula, so that has been used to distinguish when to provide and input versus an output form control. The two examples above make relevant form controls for data elements annotated with a currency type attribute. Other form controls for checkboxes and dates can be created to bind to types like booleans and dates.
Next, let's look at how the repeat expression is computed:
<xforms:bind nodeset="firstelem" calculate="local-name(instance('data')/*)" />
<xforms:bind nodeset="path" calculate="concat(../firstelem, ../expr)" />
<xforms:bind nodeset="repeatexpr" calculate="concat(../path, '/*')" />
By design of this particular "magic morphing" (XML editor) form, the first element in the computed XPath expression is the first element of the instance. Then we compute the full path to the element whose children will become editable by the repeat and the form controls within the repeat. This is based on the "expr" data node that will be programatically manipulated. Because expr is initially empty in the instance data above, that means you will initially see the children of the first element of the instance, because the repeatexpr is calculated to the first element plus the initially empty "expr" plus "/*" to get the children.
OK, so how do we adjust what the XForms repeat presents to the user? Basically, we want to either add a child element name to drill down into a subtree or we want to subtract a child element name to go up a level. First, let's cover how to add an element, i.e. add a step to the location "path". Inside the repeat, each child element that is a subtree root (has children) gets an XForms trigger in a link style button. If you activate the trigger (press the button) then you drill down into the corresponding node. Here's what that looks like:
<xforms:label><xforms:output value="choose(./@label, concat(./@label, ' >>'), concat(translate(local-name(.), '-_', ' '), choose(*/* or *='', '', concat(' - ', *)), ' >>'))"/></xforms:label>
value="concat(., '/', local-name(context()),
choose(count(context()/../*[local-name(.)=local-name(context())]) = 1, '',
concat('[', 1 + count(context()/preceding-sibling::*[local-name(.)=local-name(context())]), ']')
<xforms:setindex repeat="XMLEditor" index="1"/>
The trigger ref binds to a node that has children, as tested by the predicate "[*]". The label shows the name of the child element whose subtree you will drill into if you activate the trigger. The action sequence simply chucks a slash plus that name onto the end of the "expr" as a new step in the location path. This adds to the "path" which adds to the "repeatexpr" which updates the XForms repeat to show the children of that subtree root.
The trigger to go back up to a parent from a child is something that would live outside of the repeat because you only need one "back" button. It's actually a bit trickier because you can't get the last slash in order to lop off the last location step in the path. Fortunately, XPath lets you find the first occurrence of a substring, and XForms actions include a loop. So, the way I did this was to construct a new expression out of all the location steps in the old one, except the last, which was detectable by there being no more slashes. Here's what that looks like:
<xforms:trigger id="GoBack" ref="expr">
<xforms:setvalue ref="../scratchexpr" value="substring(../expr, 2)"/>
<xforms:action while="contains(../scratchexpr, '/')">
<xforms:setvalue ref="." value="concat(., '/', substring-before(../scratchexpr, '/'))"/>
<xforms:setvalue ref="../scratchexpr" value="substring-after(., '/')"/>
<xforms:setindex repeat="XMLEditor" index="1"/>
The first setvalue copies the "expr" less the leading slash into the "scratchexpr". Then, we clear out the "expr" so we can build it up anew from the parts of the scratchexpr. Now, we execute while "scratchexpr" still contains a slash, so the loop stops short of copying the last location step from scratchexpr to expr. Once the processing is complete, then once again, the modifications made to expr, reverberate to "path" then to "repeatexpr" due to the XForms binds above, and so the XForms repeat updates to show and allow editing of the content of the parent element.
And that's it! Thanks to eval() used in combination with all other pre-existing features of XForms, you can make a form that edits any XML element data structure.
A number of times recently, I've been asked how to create a pair of list boxes with arrow buttons between them that move the selected item from one list to the other in the direction of the arrow (if something is selected in the source list).
As I did, you can easily use the IBM Forms Designer to create a simple form that has the two list boxes for the two lists (see here
if you want to get a copy of my sample form). I used icecream flavors as the list items to be moved around between a list of available flavors and a list of chosen flavors. The Designer automatically generates the XForms markup, such as the xforms:select1 elements and the instance data to store the results of selections and the lists to select from. Here is the XForms markup from the form I created with a few drag and drop operations:
The data instance where the user selections for each list are stored:
<xforms:instance id="Generated" designer:type="generated" xmlns="">
Data instances where the actual choices are stored (the available flavors and the chosen flavors):
<xforms:instance id="FlavorsAvailable" xmlns="">
<xforms:instance id="FlavorsChosen" xmlns="">
Normal XForms-level user interface markup for showing the two lists above and for storing the choice made in each list:
<xforms:select1 appearance="compact" ref="instance('Generated')/page1/list1">
<xforms:select1 appearance="compact" ref="instance('Generated')/page1/list2">
Now we get to the dynamic part. I added two buttons to the form, one labelled >> and the other labelled <<. Each contained an xforms:trigger whose DOMActivate action performed an insert/delete/setvalue sequence to copy a selected choice from one list to the other. Given the above markup, here's what that looks like:
A trigger to copy from the available flavors (positioned on the left) to the chosen flavors (positioned on the right):
<xforms:action if="instance('Generated')/page1/list1 != ''" ev:event="DOMActivate">
<xforms:insert context="instance('FlavorsChosen')" nodeset="choice" origin="instance('FlavorsAvailable')/choice[@value = instance('Generated')/page1/list1]"></xforms:insert>
<xforms:delete at="" nodeset="instance('FlavorsAvailable')/choice[@value = instance('Generated')/page1/list1]"></xforms:delete>
A trigger to copy from the chosen flavors back to the available flavors:
<xforms:action if="instance('Generated')/page1/list2 != ''" ev:event="DOMActivate">
<xforms:insert context="instance('FlavorsAvailable')" nodeset="choice" origin="instance('FlavorsChosen')/choice[@value = instance('Generated')/page1/list2]"></xforms:insert>
<xforms:delete at="" nodeset="instance('FlavorsChosen')/choice[@value = instance('Generated')/page1/list2]"></xforms:delete>
Each trigger's action uses 'if' to test whether the source list has a selected choice. If so, then we do an insert first to copy the selected flavor's choice element to the target list. Then, we delete the choice from the source list. Lastly, we erase the selection from the data value of the select1 so that it does not report an out of range error relative to the its updated list, from which we just deleted the item that was selected. That's really all there is to it.
As an alternative to the method above, you could maintain a single list of choice elements, and have an extra boolean attribute on each choice element in the data. Call it selected. It would look more like this:
<xforms:instance id="Flavors" xmlns="">
<choice value="v" selected="false">Vanilla</choice>
<choice value="c" selected="false">Chocolate</choice>
<choice value="s" selected="false">Strawberry</choice>
<choice value="b" selected="true">Butterscotch</choice>
Then, the itemsets of the two select1 lists could use predicates, like this:
<xforms:select1 appearance="compact" ref="instance('Generated')/page1/list1">
<xforms:select1 appearance="compact" ref="instance('Generated')/page1/list2">
The advantage of this method is that you can then avoid using inserts and deletes, which cause rebuilds. Instead, you just change the selected attribute to true or false (and then also set the value of the source list to empty again):
<xforms:action if="instance('Generated')/page1/list1 != ''" ev:event="DOMActivate">
<xforms:setvalue ref="instance('Flavors')/choice[@value = instance('Generated')/page1/list1]/@selected">true</xforms:setvalue>
<xforms:action if="instance('Generated')/page1/list2 != ''" ev:event="DOMActivate">
<xforms:setvalue ref="instance('Flavors')/choice[@value = instance('Generated')/page1/list2]/@selected">false</xforms:setvalue>
Engaging with customers (or employees) involves collecting information from them. To sell an insurance product, for example, you need to know what kind of insurance, how much, who's it for, their age, and a whole raft of other information. This is how we end up being able to provide a customer with an insurance quote or product, a financial preapproval, a medical appointment, and this is how we process IT requests, support requests, purchase orders, expense reports, etc.
If you're architecting a web-available XML-based information system, you will typically define the XML data required by the system. Specifically, what information does the organization need in order to process a transaction with a customer or an employee? A result of the design work will be an XML schema that describes the overall data structure and the data types for the back-end services that process the transactions. Then, a form is created to provide a web-based data collection interface for the data. Finally, there is the possibility of java servlet code in the middle to convert the data values collected by the form into the XML data structure required by your back-end services. Unless...
With IBM Forms 4.0 Designer, the XML schema describing the data structure can be consumed directly into the form design experience without the form author needing to understand anything about XML schemas. The "form author" skillset includes high precision GUI work to create either a UI that conforms to some regulations or a UI that captures and keeps the user's attention. The form author isn't and shouldn't be worried about the data structures. They want to drag and drop stuff onto a design canvas, and so much the better if the design environment makes sure that the GUI items are automatically mapped to the data structures needed on the back end to process the data collected by the form.
Here are two videos that give you an idea of what this design experience looks like: Schema-driven Forms Design
and Schema-driven Forms Design and Dynamic Tables
The form author has been provided with the data schema, and they just pull it into the design experience and ask for the data structure to be created based on the schema definition. Then, they drag and drop the GUI representations of the data elements to automatically create groups of user interface controls that are mapped onto those XML data elements. This leaves the form author free to focus on issues of colors, borders, GUI layout, and so on because the form automatically collects the data in the XML format defined by the XML schema
. The form author needs no knowledge of XML schema, nor XML markup for that matter
, to build up the form user interface.
The drag-and-drop operation is responsive to data types, so it creates different UI controls for various data types, such as calendar pickers for dates, check boxes for booleans, dropdown menus for closed enumerations, and comboboxes for open enumerations. The operation is also responsive to XForms label annotations on element and attribute definitions as well as enumeration items, so rather than using XML element and attribute names as the default values placed into the user interface, the data architect can assign friendly prompts or labels for element and attribute data to be collected as well as enumeration items appearing in lists.
The drag-and-drop operation is also responsive to structural definitions. If the form author drags and drops a piece of data for "Personal Details", then UI controls will be created for single child elements like "Name" and "Date of Birth" but whole subgroups of controls will be created for subtree elements like "Address", which may be comprised of child elements like "Street" and "City". Moreover, if an element being dropped includes a repeating subtree element, i.e. one bearing a schema maxOccurs value greater than 1, then a dynamic table element is created, including the insert and delete buttons that allow a user to insert or delete rows from the table.
So, have a look at the special demo videos above to see the work in action, then plug "IBM Forms" into your search engine, go to our site and download the Designer and Viewer trial software so you can give it a test drive, and stay tuned to this blog because in my next post I'll take you through a sample schema and show you exactly what bits of schema trigger the IBM Forms Designer to do something delightfully automatic for you.
I learned something interesting about survey research recently. Given a survey question with a choice list answer, such as a list of icecream flavors or political parties, apparently there is a non-trivial segment of the population who read only the first few possible choices before rendering an answer. This segment can introduce significant bias into survey results unless the list of choices is randomized for each survey.
My recommended approach is to use web application code to put the choice list in the desired order and then push the list into the form before sending the form to an end-user for interaction. However, we increasingly see that people want the server side web application code to be as simple as "send form to user, and when it comes back, extract the data". As a result, front-end form logic is increasingly used in lieu of writing middle tier Java code.
So, here's a sample XForms single choice selection control that, due to the appearance attribute, would be presented as a group of radio buttons:
<xforms:select1 appearance="full" ref="/survey/icecream">
<xforms:label>Pick your favorite flavor of icecream:</xforms:label>
The set of choice items come from XForms instance data at the location given by the XPath location in the nodeset attribute of the itemset element. The label element then indicates what is shown to the user, and the value element indicates what internal data value corresponds to that display label.
<xforms:instance id="Flavors" xmlns="">
<choice value="CC">Cotton candy</choice>
Now, since we don't want to bias the results toward butterscotch and chocolate flavors, we want to randomize this list each time the question is presented to an end-user. The best solution would be if XForms had additional appearance keywords as custom namespaced extensions, something like this:
<xforms:select1 appearance="full xfdl:randomized" ref="/survey/icecream">
In current XForms, the appearance attribute only allows one keyword, and only the words minimal, compact and full are defined. The XForms working group agreed today to change the appearance attribute to a space-separated list to allow extension features like this ("xfdl:sorted" would be nice too, though we also recommend sorting a list before putting it in the form).
Supposing you do want to randomize a list of choices within an XForm, let's start by focusing on just creating a permutation when a form starts. Start with a list of any number of data values, such as 5 values, like this:
<xforms:instance id="Permutation" xmlns="">
<permutation count="" position="" random="" size="" swapvalue="">
Now, on form load, we'll create a permutation of them, like this:
<xforms:action id="GenPermutation" ev:event="xforms-model-construct-done">
<xforms:setvalue ref="instance('Permutation')/@random" value="random(true())"/>
<xforms:action while="instance('Permutation')/@count < instance('Permutation')/@size">
value="floor(random(false()) * (../@size - ../@count + 1))"/>
<xforms:setvalue ref="instance('Permutation')/@position" value="../@random + ../@count"/>
<xforms:action if="instance('Permutation')/@count != instance('Permutation')/@position">
<xforms:setvalue ref="instance('Permutation')/@count" value=". + 1"/>
The first line hooks the start of form processing, after the data model has been initialized. Next, we seed the random number generator and a couple of variables used in the main loop. The loop iterates through positions 1 to n-1 of an n-length permutation. For each position k, the value is swapped with a randomly determined position between k and n, inclusive.
Because XPath automatically concatenates the text nodes of a given XML tree or subtree, you can look at the whole permutation using a single XForms output, like this:
More importantly, you can use the permutation as a lens on the set of items shown by a selection control, such as an XForms select1, like this:
<xforms:select1 appearance="full" ref="/survey/icecream">
<xforms:label>Pick Favorite (random lens)</xforms:label>
Compared to the first code sample above, the only differences are the ref attribute expressions in the label and value elements, but they both do something interesting with the nodes produced by the itemset element. Typically, the itemset is used as a "for each" construct, and the label and value ref expressions drill *into* each node to obtain the label for an item and the data value corresponding to choosing that item. In this example, the itemset is really only used to obtain the correct number of items to generate and to set an initial context node within the set of items. The label and value ref expressions then pop up to the parent of that context node, and then drill down into a different choice element based on the order given in the permutation.
Below is a screen snap of the two select1 controls (and the XForms output for the permutation). On the left, you can see the radio button group with the items appearing in the original order of the data. At the bottom you can see the permutation that was generated when the form was rendered. Every time the form runs, a different order is selected, but this time, the order generated was 3, 1, 5, 2, 4. So, you can see on the left that "Cotton candy" is the 3rd choice, and so it appears first in the permuted choice list on the right. Similarly, "Butterscotch" is the first choice in the data, so it appears second in the permuted list on the right, followed by Vanilla, Chocolate and then Strawberry is last in the permuted list since it is 4th in the original list.
Finally, note that you only have to generate the one permutation of a particular size, which can be used as a lens on all selection controls that have the same number of elements as the permutation. If you have a survey in which different questions have different numbers of answers, then you'll need to generate a permutation for each different number of answers. However, once you have all the permutations you need, you can dynamically pick the permutation to use in the label and value ref expressions based on using the last() function to obtain the size of the itemset's node set.
Gaining customer insight has very high business value. In the last blog, I showed you how you can gain customer insight without introducing bias by randomizing the answers to each question you ask. In a prior blog, I showed you how you can make the presentation layer be dynamically responsive to the type of question you are asking. In this blog, we'll cover how to randomly select the subset of questions to be asked of a particular user. This is important in avoiding survey fatigue and low response rates that can result from surveys that take too long to complete.
As in the prior blog entry that presented survey items of different types, suppose you have an XForms instance that includes any number of items of interest, where each <item> includes a prompting question, a type of answer, storage space for an answer, and so forth. The goal is to have the form template initialize itself with a small subset of these <item> elements, and once again to do so randomly, i.e. without bias, so that over a large customer population, we will get a close-to-equal response rate on all items of interest.
Let the following two XForms instances exist:
<xforms:instance id="AllItems" xmlns="">
<!-- All survey item elements -->
<xforms:instance id="ChosenItems" xmlns="">
<items random="" i="" number="3">
<!-- Initially empty, will contain "number" item elements chosen from AllItems -->
Data initialization in XForms can be done in the xforms-ready event but is more typically done in the xforms-model-construct-done event because it is more efficient to manipulate the data before the user interface is created. So, you'd randomly choose some survey items from all the items like this:
<xforms:action if="count(instance('ChosenItems')/item) = 0"
<xforms:setvalue ref="instance('ChosenItems')/@random" value="random(true())"/>
<xforms:action while="instance('ChosenItems')/@i <= instance('ChosenItems')/@number">
<xforms:setvalue ref="instance('ChosenItems')/@random" value="random(false())"/>
The if statement on the main action ensures that the random selection only happens once, when a copy of the form template is used for the first time, thereby becoming a form. The first setvalue action seeds the random number generator, and the second action initializes the loop variable. The main loop randomly chooses "number" items from AllItems and places them in the ChosenItems. Each selected item is then deleted from AllItems to avoid duplicate items.
The "number" of questionnaire items to select is shown to be 3 in the example, but it is really up to the designer(s) of the survey to set that number into the form template based on how many questions they feel their customers will answer without impeding the return rate due to survey fatigue.
Finally, rather than binding a user interface to the 'AllItems' data instance, you simply bind it to the 'ChosenItems instance instead. An XForms repeat iterates the user inteface controls within it for each of a set of nodes identified by the XPath in its nodeset attribute, like this:
<!-- XForms user interface controls, like input and select1, go here -->
Inside the repeat is where you could use simple XForms input controls for typed answers or select1 controls for choice-based answers, or this is where you could use dynamic user interface techniques to show different kinds of user interface controls based on a type of survey item.
Modified by John M. Boyer
When I started on Java Server Pages (JSP) as a topic, I had intended it to be a blog topic. But it grew quite beyond blog size, so now that the technical work is finished, I can give you the meta-level on using JSP with Enterprise IBM Forms.
The work I'm telling you about here is intended to make it easy for you to exploit the powerful, simplifying JSP technique within the XFDL+XForms markup of IBM Forms documents. It took a some work to sort it all out, but with that done, it is easy for you to replicate what I did and gain the benefits. I wrote this wiki page on the IBM Forms product wiki to help you get set up, and the page references the developerWorks article I put together to show how to use JSP in your XFDL+XForms forms.
It was pretty challenging to get the JSP to talk to the Webform Server Translator module, so I was pretty happy when that started to work for me. It's one of those cases of only needing a line or two of code, but it being really hard to get exactly the right line or two. As Mark Twain once said, it's like the difference between lightning and the lightning bug. Anyway now that we know the smidge of code, it's easy for you to copy and use in your XFDL-based JSPs.
At first I thought, OK I have a good blog topic, but then I realized we weren't covering the full Forms information lifecycle. Put simply, a form is possibly prepopulated and then served, it collects data, but then it comes back and you have to do something with the data collected. So, back for more work sorting out how to receive a completed form into a JSP and use its values in JSP scriptlet code that helps prepopulate the next outbound form. This was a fair bit less challenging, as it maps very closely to how you start up the IBM Forms API in a regular Java servlet. Remember, JSP is just a convenient notation that the web application server knows how to turn into a Java servlet. JSP just makes it easier for you to focus on your special sauce application code.
Well, now that I could handle the whole Forms information lifecycle, I realized I hadn't covered the software development lifecycle. Back to the salt mines again. The problem was that JSP annotations are incompatible with XML. Although there is an alternative XML syntax for JSP, I devote a section in the article to explaining why it's a bit of a train wreck, and I focus instead on the normal JSP annotations. By representing them as XML processing instructions, we're able to maintain the XFDL and the JSP logic together using the IBM Forms Designer, and then use an XSLT to convert to actual JSP when it's time to deploy the IBM Form. This was really important to me because, quite frankly, if a new feature does not work in the Design environment for a language, then the feature essentially does not exist in the language.
Now, that's a wrap! I hope you like the article and get accelerated development benefit from it. JSP is really for building quick prototypes and demos, and also for solving simpler problems much more simply than using straight Java servlet coding. It's even a really nice complement to using Java servlet coding within a larger project. So don't delay, get ready to use JSP with XFDL today.