Within IBM, there is an "Extreme Blue for Employees" program called Biztech, which promotes innovative development within IBM by funding special projects developed by teams that can come from all across IBM. Today, the 6th Biztech Showcase is being held at the IBM T.J. Watson Research Lab, Yorktown. It consists of 12 projects including ours.
Our project, which will be presented this afternoon, is called Dual Forms. To understand the basic business problem, suppose you have a complex office document such as a contract, a patent filing or a new supplier agreement. It will have many sections to fill out, some with precise information like names, supply codes, dates, milestones, deadlines, and payment amounts. These are things that need to be validated in order to prevent errors, or which may need to match with data stored elsewhere. However, other parts of the document will contain free flowing text, such as special terms and conditions. The document may also contain pie charts, bar graphs, or other rich content that serves as visual aids. The key here is that the document is complicated and hard to complete, which frustrates those who must collaborate to complete and results in costly delays in completion.
Now imagine you had a technology that could augment the office document completion experience with:
- wizard behaviors that take the user step-by-step through the fill process and help correct errors immediately
- automatically invoke web services of an SOA to obtain data during the fill experience, rather than making the user responsible to type in matching data.
- digital signature security so that the legally binding agreement can be made electronically rather than having to print, sign, fax, scan and store document images.
- automatic submission of completed documents to the back-end workflow system or the repository of executed legal agreements.
This is the domain of Dual Forms. The Lotus Forms technology provides the above benefits as well as the ability to enclose any other files within a Lotus Form. One such enclosure can be an ODF office document. Lotus Symphony is an implementation that runs under the Lotus Notes application platform as well as the underlying Lotus Expeditor application platform. In this project, we've created a Lotus Notes composite application that allows the Lotus Forms Viewer to run under Notes, including read/write access to the running form. The composite application also hosts the Lotus Symphony ODF editors at the same time.
Using this project, the end-user can get access within a single application to the capabilities of office documents AND the capabilities of Lotus Forms (wizards, help, web services, signatures, document submission to the business process). The underlying technology that allows the two rich document formats to interoperate is... XForms.
Well, my frieds, it seems a good time to draw your attention to a collection of available resources that can help you understand how you and your customers can address a full spectrum computing requirements, from the enterprise information application all the way down to the simple design-deploy-collect-analyze pattern for situational applications that arise throughout the enterprise
First of all, check out the Lotus Forms Enablement videos on YouTube. There are videos that show Lotus Forms in action in government, financial, and HR scenarios where enterprise-level functionality is a must. But you can also see videos on the new Lotus Forms Turbo product, which enables business users to design and deploy simpler survey style forms for themselves in a matter of minutes.
I want to make a pitstop here to plug the value of the XForms standard in creating this array of products. XForms as a language has a strong declarative component. This means that non-trivial relationships between data and sophisticated interactional and presentational objects is simply declared by the form developer, and it is up to the form run-time system to enforce the relationships. This maps very naturally to how we human beings think and work. We say things like "Whenever I get an email from domain X, put it in my super special high priority folder". We don't directly write batch jobs that run loops over our inbox to categorize our email, and even if we did, the time at which those jobs would run would be specified declaratively! In fact, I don't think it's hyperbole to say that the universe itself operates mostly on declarative laws. Gravity just happens. If you let go something, it drops. It's the law; get used to it ;-).
The point is that the declarative language constructs of XForms are what enable Lotus Forms to accelerate enterprise-level application design, to enable end-user design of simpler applications, and to ensure interoperability between these two types of applications so that the simpler situational applications can be scaled up to the enterprise level as their value becomes more broadly established. The declarative constructs of XForms accomplish all of this due to the simple fact that it is far easier to create software that maps point-and-click/drag-and-drop design experience gestures onto high-level declarative language constructs than it is to map the same things onto huge blobs of imperative scripted code from a general purpose programming language.
And don't just take my word for it. When you finish watching some of the videos at the YouTube link above, how about going for a test drive? You can get free trial downloads of the Lotus Forms, including the client-side runtime and design environments. If you like the enterprise-level capabilities, but you don't want to deploy a client-side runtime, you can also purchase the Lotus Forms Web Forms Server, which translates Lotus Forms into standard web pages. But once you're in test-drive mode with the client-side runtime viewer, check out the Lotus Forms catalog of sample forms. Finally, you can also test drive the new offering for non-technical users directly, without having to install a thing by visiting Lotus Forms Turbo on Greenhouse, or if you'd prefer to test drive it in-house, then just download it from the free trial downloads page.
Either way, once you've gotten Turbo, see a couple of the videos then think of your own simple survey app. Design and deploy it right onto Greenhouse and see how long it doesn't take!
Recently, I was experimenting with one of the features planned for the next version of XForms. The feature is the iterate attribute for XForms actions, which will perform a for-each loop operation based on a nodeset obtained from the xpath expression in the iterate attribute value. XForms 1.1 already has a while loop, but iterate makes many data processing loops easier to write and also more performant (subject of a future blog). There were lots of iteration use cases to choose from, but I decided to experiment with sorting because it is a well-known benchmark algorithm.
Before we go any further, let me say up front that XForms action scripting is intended for very lightweight data manipulation, like adding or deleting a data node corresponding to a table row or copying data results to or from the SOAP envelopes of a web service. By the time you get to nested iterations like those needed sorting, you should be considering alternatives expressed in full-blown imperative languages available in the information system within which the form is being used. For example, in the case of sorting, it is a better idea to request sorting in the database query whose results are returned from a web service into your form so that your form logic does not even have to do the sorting.
So, with the disclaimer out of the way, let's abuse the technology a bit to get a better sense of what is feasible in those customer-needs-it-yesterday circumstances. It turns out that XForms 1.1 does allow full nodeset processing in the insert action's origin attribute and the delete action's nodeset attribute. Without even needing the new iterate attribute, this is just enough iteration capability to perform efficient sorting -- so there are some kinds of iterations that can be done now without the iterate attribute.
We're going to do a divide-and-conquer "partition" sort that I personally created as a university freshman after my 1st semester instructor told our class that linked lists could only be sorted slowly. At the time, the usual computer languages only allowed static allocation for arrays, and even though I didn't know what a "quick sort" was, I had seen the light of dynamic allocation, and I was never going back! I later learned how great a merge sort is on a linked list, but the effort of turning an array quicksort into a linked list partition sort comes in handy now because a merge sort cannot be efficiently expressed in XForms until the iterate attribute is added.
The way a quicksort works on an array (or subarray) is that you pick a random element to be the 'pivot' value. Then you run two index variables at the same time, one from the start of the array upward and the other going from the end of the array downward. The 'up' index is advanced until it finds a value greater than the pivot, the 'down' index is decremented until a value less than the pivot is found, and then the values at the 'up' and 'down' locations are swapped. This keeps happening until 'up' and 'down' meet somewhere near the middle of the array. At this point you've partitioned the array into a subarray of values less than the pivot value and a subarray of values greater than the pivot value. The quicksort is then invoked recursively to sort both subarrays.
The main challenge with this approach is the 'down' index, which is a reverse iteration. In a singly linked list, you can only go forward. XForms insert and delete actions have a similar limitation: they can only identify a nodeset of nodes to insert or delete, but not really a direction of iteration. But the important bit is what the quicksort is doing, not how it is done. Think of the list content as being completely messy, and each partitioning stage must make it somewhat less messy by dividing the content into a partition of lesser elements and a partition of greater elements. Then, the next partitioning stage is invoked recursively to do a better job of cleaning up the mess within each partition.
Let's explore this concept by sorting a list of elements, such as sorting a list of <person> elements by a <lastname> child element. We begin by copying the list into an initial partition element of a temporary instance called 'sortdata', like this:
1) <xforms:insert context="instance('sortdata')" origin="instance('partition')"/>
2) <xforms:insert context="instance('sortdata')/partition" origin="instance('data')/list/person"/>
3) <xforms:delete nodeset="instance('data')/list/person"/>
Next, we initialize the random number generator so we can randomly select pivot values for all the partitioning stages:
4) <xforms:setvalue ref="instance('sortpivot')" value="random(true)"/>
Next, we start up a simple while loop that continues to process partitions until none are left.
5) <xforms:action while="instance('sortdata')/partition">
Within the loop, we grab the last partition from the sort data and determine whether it is non-trivial or trivial (only 1 or 2 elements). A non-trivial partition is subjected to further divide-and-conquer processing.
5.1) <xforms:action if="count(instance('sortdata')/partition[last()]/*) > 2">
The first part of dividing and conquering is to create a new empty <partition> element, which we obtain from an instance that expresses a template empty partition element.
5.1.1) <xforms:insert nodeset="instance('sortdata')/partition[last()]"
Step 5 and step 5.1.1 are more interesting than they seem at first. The list of partition elements in the sortdata actually implements the recursion stack, and we just pushed a new element into that stack at the second-to-last position. Because we have an explicit stack, we only need a loop in step 5 to implement recursion.
The next thing we do here is grab a random last name to serve as a pivot value for the partitioning. The first setvalue just picks a random location, and then the second step uses the location to get the value. Notice also that I use * rather than partition before the [last()] predicate because the sort data only contains partition elements, so there is no point in doing a name test for partition.
5.1.2) <xforms:setvalue ref="instance('sortpivot')"
value="1+floor(random(false) * count(instance('sortdata')/*[last()]/*))"/>
Now the magical part happens. All elements in the last partition whose key element (lastname) is less than or equal to the pivot value are moved to the newly created second-to-last partition. By combining the nodeset processing capability of XForms insert and delete actions with the predicate-based node selection capability of XPath, the matching nodes can be selected and moved using two single XForms actions, i.e. without using an XForms while loop.
5.1.3) <xforms:insert context="instance('sortdata')/*[last()-1]"
origin="instance('sortdata')/*[last()]/*[compare(lastname, instance('sortpivot')) <= 0]"/>
<xforms:delete nodeset="instance('sortdata')/*[last()]/*[compare(lastname, instance('sortpivot')) <= 0]"/>
If the last partition is now empty due to the move operation in step 5.1.3, then the new second-to-last partition received all its elements. If all the moved elements are equal to the pivot value, then we can output them back into the original data list and then remove the last two partitions. Note that the insert is configured to prepend the elements into the data list, and we're copying them from the last non-empty partition, which has the elements with the greatest key value.
5.1.4) <xforms:action if="count(instance('sortdata')/*[last()]/*) = 0">
<xforms:action if="not(instance('sortdata')/*[last()-1]/*[compare(lastname, instance('sortpivot')) != 0])">
<xforms:insert context="instance('data')/list" origin="instance('sortdata')/*[last()-1]/*"/>
<xforms:delete nodeset="instance('sortdata')/*[position() >= last()-1]"/>
</xforms:action> <!-- End of 5.1 non-trivial partition handler -->
Now, we've finished with the non-trivial partition handler, and we turn our attention to processing a trivial partition containing at most 2 elements. The content of the partition is moved to the original data list and the partition is removed. Again, note that we're processing the last partition, which has the greatest key values, and the insert prepends to the data list, so the sorted data list starts with the greatest values and grows as lesser and lesser values are prepended over time as all the partitions are processed.
5.2) <xforms:action if="count(instance('sortdata')/*[last()]/*) <= 2">
If the partition contains two elements and the greater one is first, then it is removed from the partition and put in the original list:
5.2.1) <xforms:action if="count(instance('sortdata')/*[last()]/*) = 2 and
instance('sortdata')/*[last()]/*/lastname) > 0">
<xforms:insert context="instance('data')/list" origin="instance('sortdata')/*[last()]/*"/>
Now the partition contains either no elements, one element, or two elements in sorted order, so we move the content to the data list and then delete the partition.
5.2.2) <xforms:insert if="count(instance('sortdata')/*[last()]/*) > 0"
</xforms:action> <!-- End of step 5.2 trivial partition handler -->
</xforms:action> <!-- End of step 5 recursion while loop -->
As a final note on all this algorithmic fun, the question arises whether this sort achieves optimal O(N log N) performance. The answer is no, not quite, due to hidden costs of data instance management and data node selection. However, the sort will be much faster than a "simple" sort because it does perform only O(N) XForms actions.
Something happened today that seemed like perfect fodder for this blog because it combines a humanitarian story, social business, and a good use case for IBM Forms Experience Builder in a social business context.
An esteemed colleague at my Canada software lab site is the organizer for our site's donations and efforts for a local charitable cause. Turns out, IBM has a special charitable grant program in which IBM makes a substantial additional donation to the charity if a set number of IBMers sign up as volunteers. I really like this model because it essentially gives IBMers a way to vote, not only on how much IBM corporately donates but also where the donation goes. It's the same reason I like charitable tax incentives, as it is essentially the same as the federal government allowing the people to vote on the charitable causes they support and how much tax money goes to them.
Anyway, IBM internally runs an enterprise social business platform called Connections, which includes profile pages for all IBMers. These are like Facebook or LinkedIn pages, only inside the corporate firewall. Still, there is information that would not normally show up on one's profile page, but which my colleague needs in order to complete our internal IBM charitable grant application. So, my colleague has found it necessary to reach out to a large number of people at our site to get this information; she's under a tight timeline, and she needs responses from everyone.
This is a great example use case for IBM Forms Experience Builder. One reason is that my colleague needs to quickly stand up a lightweight custom data collection solution, where we'd miss the grant deadline for our charity if we had to wait for a normal IT project to be completed. So, this is a great example of workforce agility to be gained. Yet, on a technical front, this is interesting because it is so easy to see that the solution needs two forms, and their corresponding backend DB storage, as well as a service to connect one to the other. The fact that you can make all this stuff and hook it up in your web browser without coding is truly a step forward in IT democratization.
In this solution, my colleague can create a first LIST form that simply allows her to enter the names of all the people that she needs to get information from. Each person's email address can then be automatically looked up by hooking up into the form the LDAP lookup service that is added to our installation of Forms Experience Builder.
Then, my colleague can create a second COLLECT form that allows each person to provide the information she needs.
Most importantly, my colleague really needs to be able to re-enter her LIST form at any time, not to add more names, but rather to see which people have submitted the COLLECT form-- and which haven't. To do this, she can automatically hook up the database for the COLLECT form as a service to her LIST form. As each person fills the COLLECT form, their entry on the LIST form can show their required information.
At any point, my colleague can look at her LIST form for blank entries to see those people with whom she has to take "secondary measures" to ensure 100% response before the deadline. Most importantly, her LIST form would already then have all the information she requires collected into one place.
Can you imagine how much more work she'd have to do if she did all this by email? She'd have a flood of emails mixed with her normal email, and she'd have to fish out the information from each email and put it into a list manually. Tedious, error prone, and lots more ways to miss the deadline. My colleague is be able to collaborate much more efficiently and effectively using a 2-form IBM Forms Experience Builder solution, and this is why lightweight data management solutions should be added as an essential ingredient of an enterprise grade social business platform. Specifically, if you're going to purchase IBM Connections, then add IBM Forms Experience Builder.
Information system architects, and even vertical industry standardization or government IT standardization bodies, derive immense value from XML schema definitions that describe the data structures and data types expected in valid transactions of the information system. However, these assets are focused on defining the completed transaction. In many a presentation, I've talked to potential customers about how IBM Forms documents express far more value because they are about the human interaction that takes place during the fill experience needed to produce those completed transactions.
As a bit of an aside, it's important for the technically minded reader to be familiar with the "sell" side of this equation. It is important to be able to easily justify technology adoption decisions with business owners who need to be able to understand how you will be able to better server your customers, reduce development and maintenance costs, increase competitiveness, eliminate vendor lock-in, etc.
In this blog, I'd like enumerate various benefits that you get from the standards basis of IBM Forms documents and their implementations, but I'd also like to separate the enumerations into two lists: 1) benefits above using XML Schema that you get just from using the XForms markup within an IBM Form, and 2) the benefits you get from the XFDL processor (XFDL is the XML vocabulary that IBM Forms provides as a presentation layer for XForms).
Firstly, an XForm is a clearly a superset of XML Schema since an XForm can incorporate the XML Schema if it is available and provide its validation information set to the fill experience and the submission experience. But an XForm provides many additional benefits, including the ability to:
Express data validation constraints that are based on other XML data values entered during the fill experience.
Automatically compute data values based on other input entered during the fill experience, rather than requiring users to perform the error-prone task of calculating and inputting summative results manually.
Describe the inputs, outputs and triggering mechanisms of web services to invoke, such as to use entered XML data to obtain database query results that fill other parts of the form or to invoke server-side validation or calculation logic from a business rules engine.
Control whether a data value is readonly or whether the user can enter a value, for example based on conditions related to a business process step or an access rule.
Express user interface controls that indicate which XML data values will be available to the presentation layer for input or output.
Conditionally show or hide the user interface controls (and hence their presentation) in response to conditions, such as those that may relate to a business process step or access rule.
Define a step-wise fill experience to guide a user systematically through incremental successes in entering valid data.
Provide customized help and validation error messages to help users fix data input errors when they occur.
Provide prompting label text to be associated with the presentation of each input or output control, both visually and aurally (for accessibility)
Associate a selection list with any XML data node in a way that constrains user input capability to the provided list
Define labels, help messages, validation error messages and selection lists in more than one human language within the same form, thereby ensuring citizens receive the same form logic and interaction behaviors regardless of which official language they select to request government services
Associate repeated data with a logically tabular set of user interface controls and encode the means by logical rows of the table are added or removed in response to insertions or deletions of data
While the above benefits indicate what additional behaviors and features of XML forms can be expressed above those that can be expressed by an XML Schema alone, XForms is also more interesting for standardization of a forms repository due to what it does not express. An XForm does not rigorously bind its many behavioral benefit to a specific presentation layer of the form. The intention of this language architecture was to address multimodal requirements of forms applications, e.g. rendition on a desktop, tablet, smartphone, telephone call-in voice service or instant messaging interaction. Different presentation layer implementations can address these requirements, and such implementations can even be provided by different vendors. The XForms working group also anticipated that there would be a wide array of varied technical requirements for presentation layers, and this language architecture allows XForms to be used with fundamentally different XML presentation languages that address these disparate requirements. Examples range from ODF for flowing text with fill-in-the-blanks fields to XFDL with its high-precision contract-style layout capability.
Due to the above mentioned language architecture, XForms markup does not comprise a well-formed XML document until it is incorporated into a presentation layer XML document. XFDL in the XML vocabulary used in IBM Forms to provide a presentation layer for XForms.
Of course, in a "baseline standard" version of a form, the default presentation layer associated with the XForms markup can be minimal in nature so that the only benefit is to provide a well-formed XML document to host the XForms markup. Interestingly, once you have this from XFDL, then the result is in fact an XML document, and so it can be processed by readily available XML processing tools like XSLT. These XML tools can be usd to automate creation of different versions of the baseline form that may have a richer presentation description, alternative natural language usage, or even different presentation layer markup. In addition, various consumers could use simple XML tooling to rebrand the forms.
The Extensible Forms Description Language (XFDL) in an XML vocabulary describing the presentation layer and richly interactive behaviors of modern web-based electronic forms. This XML vocabulary was first introduced to the W3C in 1998 (http://www.w3.org/TR/NOTE-XFDL
) and over the years, the versions of the language have consumed XML data-processing components of the W3C XML technology stack as they have been standardized by the W3C, especially including XML Schema, XForms, and XML Signatures. XFDL is a royalty-free open format whose current version specification, which can be obtained from -
XFDL is a host XML language for the XForms standard, and so the many benefits of XForms described above are inherited by XFDL. The implementations of XFDL as a presentation layer language add the following benefits over the core XForms processing:
very high precision control over the layout and rendition of the user interface
integration of XML Signatures with both XForms and a high precision user interface
comprehensive treatment for accessibility, localization and language support
dynamically computable presentation layer effects based on data value changes
multiple page support and ability to easily integrate web analytics within the form
ability to print or generate a PDF snapshot of the form or of just the agreement style pages
built-in ability to save the form locally and reload it later to continue the fill experience
a built-in file attachments folder system, including the ability of digital signatures to protect files attached into the form
a “public data” map for mapping XForms instance data to business process/workflow variables and case management properties
integrations to standard application server processing of form results and to run-time processing by JSR 168 and JSR 286 compliant portlets
the choice of zero-install operation within a web browser, using a server translator module
the choice of a client install to support both offline and online processing
In a larger sense, though, this only part of the benefit derived from XFDL. Still more benefit is derived from the availability of the XFDL forms visual design environment, which gives form authors integrated access to GUI features for XForms, for XML Signatures, for schema-driven design, for web sevice connections, and for the XFDL language benefits included in the list. The design environment even includes a converter that helps preserve the layout of PDF Forms that are brought over into XFDL. Finally, the XFDL forms visual design environment also provides features for maintaining a collection of forms, such as SVN and other team repository plugins and management of form parts.
One of the New IBM Forms 4.0 Demonstrations
has just won the top-rated video award at Lotusphere
. This is a fantastic victory for the Websphere Portal segment of Lotus, which focuses on products like IBM Forms that create and provide exceptional web experiences. It is also a victory for the W3C XForms standard
since the Wizard Creator
featured in the video is, as far as I know, the first point-and-click design experience for an XForms switch. This feature exemplifies the principle that declarative markup languages result in more powerful application design environments because they express what the author wants. With an imperative languages, a design environment must either operate at a much lower level (less powerful) or do some wicked reverse engineering/pattern matching to discern what the author wanted from how he did it.
So, XForms has definitely been my friend in helping to create an award-winning feature in IBM Forms. Wanna meet my newest friend? Here we are, just me and Watson, celebrating our victories at the closing session of Lotusphere 2011:
To put it in Watson's terms:
Category: Lotusphere Best Demo Video Winners
Answer: An IBM Form that uses XForms to express Wizard interfaces for forms.
Question: What is a Smarter Web Application?
MC: Congratulations Watson, you're ready for prime-time! Wanna join my social network?
The XForms team finished our face-to-face meeting in Amsterdam last week. A major focus of the work on XForms 1.2 is called modularization.
The rationale for this work is the observation that the set of XML data processing problems which may have first arisen in the electronic form space are really more generally applicable to the XML data processing needs of RIAs and web applications. On the other hand, those who hear the word "form" may think they do not have a form problem because they still think of a "form" as a simple or static application like those for ordering pizza or flowers. However, XForms has solved numerous problems that keep coming up again and again throughout the W3C standards stack as well as the web application stack. By modularizing the components of XForms, we believe we can increase adoption of the components in other technologies which may not have need of all aspects of XForms.
The current view of the XForms 1.2 modularization can be viewed here. As an example of the rationale above, consider an application that may want to use the submission capability from XForms in a regular web application. The application would import the instance data module, but it might have an application-specific way of populating a data instance with data. The submission module would reference a data instance for the upload data and another instance for the submission results. An application-specific method would then be invoked to consume the submission result into the application, but the means of invoking that method could be an event handler for the xforms-submit-done event. The key issue here is that submission could be consumed by a non-XForms application without needing to incorporate the XForms recalculation engine, user interface controls and so forth.
The full elaboration of this modularization will allow applications to consume pieces of XForms incrementally, including the notion of data validation, data relevance, declarative data calculations, event-drive action scripting, repeats, switches, groups, basic user interface controls, and of course submissions.
Last blog I told you how you could get a "Greenhouse" account that gives you access to a cloud-hosted build-without-coding environment for constructing and deploying data-centric IT solutions. The product is called IBM Forms Experience Builder (FEB), and I want to spend some time on this blog giving you a better sense of the solutions you can create.
The solution I'd like to cover is a bit more advanced than a "Hello, World" solution; maybe I'll do one of those in the next blog. But the first thing I wanted to be able to do was use my account to enable myself to share solution files with you over time. Every FEB solution has a body of code that represents it. You can export the serialization of any solution as a file so that it can be imported into another account or onto another FEB server. So, I wanted to create a FEB solution that allowed me to share FEB solution files with anyone. I created that "Simple File Sharer" solution and then used it to share the "Simple File Sharer" solution file. You can get the file here (requires Greenhouse account login).
Once you have the solution file, you can import it into your own FEB account within Greenhouse and then use it to share files, especially any sample FEB solutions.
The Simple File Sharer solution uses ordinary features in the form (user interface). I just dragged and dropped a table item, and then put a name, description and upload component into the table. I also dragged and dropped a textual label so I could provide some basic instructions for users of the solution.
The Simple File Sharer solution uses some more interesting FEB capabilities to implement behavioral features around the form user interface.
- I used the Access panel to indicate that only one user, yours truly, could create a new database record or update a database record.
- Often, once a user has used a Form to create a database record, subsequent visits to that Form should allow the user to update the record rather than creating a new record in the database. An example would be a vote or a survey response. You don't want the same person voting or answering the survey more than once. In my case, I wanted to restrict myself to one database record because I always want to update a single list of shared files, not create a new list of shared files each time I fill out the form. Regardless of your use case, you can do this by going to the Forms tab, clicking the properties of the form, going to the Advanced tab and then checking "Limit to single submission per Authenticated user".
- I used the Stages panel to add an Update stage to the solution so that database records can be updated after their initial creation.
- I used the Access panel to add a "Reader" role, and I assigned "all authenticated users" to that role. Then, in the Update stage, I gave only Read access to the Reader role. So, as soon as you do a Greenhouse account login, you can use the link I provided to read from the database record I created that includes the table of files that I've shared.
In conclusion, the Simple File Sharer solution may only allow me to create a record, but it still ends up demonstrating a number of FEB features that you would commonly use in solutions that allow multiple users to create and edit records. In the future, I'll share more solutions with you that highlight various FEB features that contribute to creating interesting solutions. Remember, it's not about just forms; it's really about the whole solution you can wrap around forms, including automatic database storage, access control, lightweight workflow stages, and even configuring web services-- all without coding. And that's what makes IBM Forms Experience Builder a platform-as-a-service capability that everyone from a line-of-business users up to an IT professional can appreciate!
Ubiquity XForms version 0.7 is now available. In addition to contributing to the recent advancement of XForms 1.1 to Proposed Recommendation, this new version has many new features, fixes and highlights. Personally I'm happiest with the progress on implementing submissions, but that may be a bit of bias because I contributed some code to deal with aspects of submissions such as serialization, validation and relevance pruning. On the other hand, I contributed code in other areas, so maybe it's just that I'm biased toward any improvements to our ability to support the XRX architecture and connect the XForms client to server-side services.
And this blog entry would not be complete without my mentioning a special word of appreciation to the folks at webBackplane for all their contributions in general but especially for the Ubiquity XForms rollup system, which consolidates the many files of the processor into a single file to be deployed on your server. I have an internal project right now that uses the rollup, and it provides us with the high level of performance we expected/required in our applications.
Full details of the version 0.7 release can be found here: <http://code.google.com/p/ubiquity-xforms/wiki/Release0Point7>
There are a number of new XPath extension functions available to XForms developers in the latest release of IBM Forms, and I'd like to draw your attention to two of them: eval() and eval-in-context(), and they are wicked cool!
The function eval(expr) evaluates expr in the context of the function call and returns the result. The eval-in-context(expr, contextExpr) function does a similar thing, except it first evaluates the contextExpr and uses the result as the context for the main expression. This is desparately needed for XPath 1.0 expressions to eliminate the infestation of pesky ".." operations that typically occurs. I've used it, rather than eval(), in the samples below.
One use of these functions is to enable the powerful capability to let XML data carry sophisticated dynamic metadata, which can then be implemented and enforced with singular XForms bind elements that attach the semantics of the metadata whereever it is found in the data.
It turns out that the xsi:type attribute from XML schema is already a rudimentary version of the metadata idea we're pursuing here, so the question becomes what if you could do it for all of the juicy metadata that XForms contains, like calculated data values, data validity constraints, and so forth? Let's look at what this "decorated" data might look like for a simple expense report:
<total value="quantity * price"/>
constraint="total < 10000"/>
It is really easy to use XPath capabilities to find all elements having a value attribute and then to bind an XForms calculate formula to those elements, and then eval-in-context() is used to determine the result according to whatever expression is given in the data, like this:
<xforms:bind nodeset="descendant::*[@value]" calculate="eval-in-context(@value, ..)"/>
The nodeset expression starts with descendant::* to explore all elements of the data, and then the predicate [@value] selects all elements that have a value attribute. For each such element node, a calculate formula is bound to it by the XForms bind. The eval-in-context() call uses ".." to go up a level so that the formulas in the value attributes can omit "../", e.g. so the expression can simply be "quantity * price" rather than "../quantity * ../price".
Similarly, a data validation constraint can be attached like this:
On the expense report data above, this binding evaluates the constraint expression attached to the total element, and then converts the result to a boolean. Due to the constraint, the expense form data cannot be submitted to a server for processing unless the total expense is less than 10000.
In other scenarios, you may want to control the other metadata properties like relevant, required and readonly. Here's an example of data where relevance and requiredness control is needed:
<parent relevant="age < 18" required="true">...</parent>
The xsi:type assignment for age already works in XForms without needing a bind. The required setting for name is statically true, not dynamically changeable, so it is very handy to be able to allow either a static boolean value or an expression, like this:
required="@required='true' or boolean-from-string(eval-in-context(@required, ..))"/>
Technically, the required property on the parent element is conditional on the age value, but that is an automatic feature of XForms, i.e. nodes marked required are only required if they are relevant. It should not be too surprising to see that the bind for relevance looks like this:
relevant="boolean-from-string( eval-in-context(@relevant, ..) )"/>
At this point, it's probably safe to leave the readonly bind as an exercise for the reader :-)
As we start off the business of a new decade, I already find myself looking forward to the improvements in XForms 1.2.
I know, I know, we just released the XForms 1.1 Recommendation, which contained a huge bundle of new features and architectural improvements. Still, the working group has been energetically advancing on still more features and improvements even as we closed the loop on the W3C process in the latter half of last year. I am very pleased with the completion of XForms 1.1, but the only constant is change, and we have an excellent array of new technical results to achieve in XForms.
The working group has put a fair bit of effort into the triage of features to separate nearer term (and simpler) objectives from those features of a more architecturally significant nature, which tend to get put in the "XForms 2.0" bucket.
My personal favorite "simple" feature is model-based UI switching with the switch element. This may seem like a bit of a nerdy thing to perseverate on, but the switch element is the only UI control we have that is not directly controllable by a UI binding to data. So, if a form author wants to switch UIs based on a user input, such as changing the payment details UI based on the selection of a payment, then the form author typically resorts to using group elements and the relevant model item property. But this is more than just the sense of completion we'll get from being able to say *all* UI controls can be declaratively linked to data. And it's more than just the satisfaction we'll get from providing the right tool for the job. The ability to declaratively link the state of a switch to data is critical to the save/reload web application pattern and the transaction digital signature web application pattern.
Other web application standardization efforts are focused mainly on what happens during execution of a single web page. XForms-based technogies are a step ahead because they can more easily address the user patterns related to collaboration on the web. Whether it is one user saving an application context and reloading it again in the future, or whether one user provides data to be loaded by a second user, the simple fact remains that data can be in any state along a continuum between empty template and fully completed, so web applications need to be declaratively responsive to that continuum. And nowhere is this clearer than when a web-based transaction is of high enough value to warrant a digital signature. A digital signature needs to be able to capture the full current state of the web application, and having a declarative binding to the data is the easiest way to achieve that goal since it means one can simultaneously sign the web application page template(s) and the data as separate resources.
For the record, Lotus Forms has for a few years now used a legitimate XForms extension mechanism, the xfdl:state attribute, to support model-based switching. We deemed this extension critical to our ability to address our customers' requirements for the above two web application patterns. Standardized integration of XForms with XML Signatures and other advanced security features like XAdES is, so far, in the XForms 2.0 bucket.
In conclusion, I'm not necessarily saying this is the "biggest" feature of XForms 1.2, only that it is my fave. I'll be giving some coverage to the other XForms 1.2 features in future blog posts, so please click the 'follower' button above (top right) and stay tuned.
Modified by John M. Boyer
As an interesting possible counterexample to my last blog about MLR models not understanding the knowledge they learn, consider the neural network. Our brains are neural networks, and we are capable of learning at all levels of Bloom's Taxonomy, not just the knowledge level. Shouldn't artificial neural networks be able to achieve the same things?
The answer is no, not really. Our brains biologically, chemically and physically perform in ways that we scarcely understand, so our name for the thing we call "artificial neural network" is no less anthropomorphizing than when we say that a computer program of today "understands" anything.
Still (again), this is not to say that they aren't incredibly useful and effective. It's just that they are based on straightfoward and well-understood mechanical methods such as feed forward activation of neural outputs via sigmoidal threshold functions applied to inputs and back propagation of synaptic weight adjustments based on easily quantified classification errors. Before going any further, let's have a quick look at a diagram of an artificial neural network (ANN):
The ANN has an output layer on the right that is a classifier for input patterns received on the left. For example, an ANN for optical character recognition could have an input layer of an 8x8 matrix of bits, and the output layer could be an 8-bit code that indicates an ASCII character. The hidden layer(s) of neurons help the ANN to represent more sophisticated phenomena, though there is seldom need for more than one hidden layer. The "synaptic" connections between the neurons in the layers are weighted numbers, and the neurons apply the weights to the inputs and then feed the results into a Sigmoid function that essentially decides, like a transistor or switch, whether or not to fire the output.
An ANN is "trained" by giving it a sequence of input patterns for which the correct output pattern is known. The input pattern feeds forward through the ANN to produce an output. If there is a difference between the ANN output and the correct output, then the differential error is back propagated through the ANN to adjust the weights so that future occurrences of that input pattern are more likely to produce the correct output.
The synaptic weights, then, essentially represent the knowledge that the ANN "learns" from the input patterns. This is analogous to the constants that are "learned" by an MLR model. In fact, all elements of the ANN and MLR model architectures are analogous. The ANN input layer maps to the the independent X variables, the ANN output layer maps to the dependent Y variable, and the transition from input X values and the Y value that is achieved in MLR by multiplication and addition is achieved by a feed forward through synaptic connections, hidden layer neurons and Sigmoid functions in an ANN.
With such a one-to-one architecture mapping between ANNs and MLR models, it is easier to see them as having similar intellectual power. That's not to say they're equivalent, as ANNs are far more powerful. It's just that they're roughly the same (low) order of magnitude with respect to human intellect, and in terms of Bloom's Taxonomy, we call that order of magnitude "knowledge storage/retrieval".
Despite being in the lowest order of magnitude of intellect, the realm of today's artificial intelligence includes many interesting knowledge storage/retrieval techniques that are worth comparing and contrasting to see the range and limits of their power and the use cases they address. Stay tuned!
How would you like to be able to construct, deploy and get results from IT solutions using only your web browser?
Don't believe me? Well, how about coming to the IBM Forms wiki
, where you can watch a few short videos that show you.
You'll be intrigued and want to go the next step. One of the prominently available wiki pages is a community article that gives you a starter pack
of prebuilt solutions like the ones you see in the video. You can download any one or all of them because they're just single files that describe the forms, access control, workflow stages and other resources of each solution. You can import any of them into your own IBM Forms Experience Builder
server, and then deploy them, use them, get results from them, and of course edit them to see how they work or to change them and redeploy them. All from your web browser.
Don't have an IBM Forms Experience Builder server to try it out? Well, now we've gotten to the main topic of this blog article. You can get your own free public access to an IBM Forms Experience Builder server. You can try out any of these starter pack solutions as well as build and deploy any of your own solutions.
Since you will be a builder
of forms experience solutions, we will need to be able to present your solutions to you, distinguished from everyone else's solutions. So, you'll have to start by registering yourself with the system that hosts the IBM Forms Experience Builder server. The system is called Lotus Greenhouse
, so click the link and then choose "Sign up" to get your account.
Once you're able to log in to Greenhouse, you'll get access to a number of software products including IBM's social business software (Connections), IBM Websphere Portal Server, and of course IBM Forms Experience Builder. However, you don't really need to log in to Greenhouse then menu navigate to IBM Forms Experience Builder when you can just bookmark the direct link to IBM Forms Experience Builder on Greenhouse
Once you log in with your Greenhouse user id and password, you'll see the "Manage" solutions page, which lists all of the Forms Experience Builder (FEB) applications that you have designed. This is the page that gives you the ability to create a "New Application" or "Import" one of those starter pack applications, all at the press of a button.
So, now you can try out and evaluate IBM Forms Experience Builder now to see for yourself that there really is a smarter web where you can construct valuable solutions without coding now. If you are building IT solutions for your organization, you owe it to yourself to see how much more effective you'll be at satisfying your organization's IT solution demands. But even more importantly, if you're competing for IT solution services contracts, you owe it to yourself to become an IBM business partner or to expand your partnership to include IBM Forms Experience Builder. And finally, if you like to build industry-specific data management products, then you should consider becoming an IBM value-added reseller (VAR) so you can build your products more efficiently with IBM Forms Experience Builder and go to market with IBM to sell the bundle. In all these cases, you now have the access you need above so you can learn more and get started today.
For a very long time, the XML language underlying Lotus Forms (called XFDL) has offered an option element called <format> to describe data type, validity constraints and presentational formatting for the (valid) values of form controls (called XFDL items, like text entry fields, selection popups, and so forth). Neither a whole Lotus Form document, nor the XForms instance data it contains, can be submitted to a server unless all relevant data type and validity constraints are satisfied.
However, the <format> elements are placed into the form controls, so the user interface layer must be articulated in order to create the run-time objects described by the <format> elements. This makes it harder to use Lotus Forms performance enhancing features like "on demand page loading" in a multipage XFDL document. It will also conflict with an increasing number of performance enhancing features over time. Suffice it to say, it would be better to express <format> data type and validity constraints as XForms binds in the form.
Sometimes this is easy to do. The following are a few examples. Suppose you have an XFDL <field> text entry element containing <xforms:input ref="data"> as the form control binding it to a data node named "data".
Change: <format> <datatype>integer</datatype> </format>
To: <xforms:bind nodeset="data" type="xforms:integer"/>
Change: <format> <datatype>float</datatype> </format>
To: <xforms:bind nodeset="data" type="xforms:double"/>
Change: <format> ... <constraints><length><max>10</max></constraints> </format>
To: <xforms:bind nodeset="data" constraint="string-length(.) < 10"/>
Change: <format> ... <constraints><range><min>18</min></constraints> </format>
To: <xforms:bind nodeset="data" constraint=". >= 18"/>
Change: <format> ... <constraints><mandatory>on</mandatory></constraints> </format>
To: <xforms:bind nodeset="data" required="true()"/>
These are easy mechanical changes to make that prepare a Lotus Form for use with performance enhancing features like "on demand page loading". The XForms binds are easy to create with the "Create Bind" or "Create Bind with Wizard" features in the Lotus Forms Designer's XForms Model Editor.
One aspect of <format>
option capability that is... somewhat less mechanical... to replace are constraints based on regex (regular expression) pattern matching. If you don't know what a regex is already, just do a web search for "regex tutorial" and spend an hour or two reading something that will save you days upon days of your life in the long run.
As an example, here is a simple XFDL <format> that applies a regex pattern limiting user input to zero or more alphanumeric characters:
<format> ... <constraints><patterns><pattern>[A-Za-z0-9]*</pattern></patterns></constraints> </format>
The * means zero or more characters, the brackets are used to express the range of characters, and subexpressions like A-Z and 0-9 provide subranges of the acceptable character range. Regex is a lot more powerful than this, so you really should read that tutorial if you needed this explanation. For example, you can express multiple alternative patterns directly in regex without exploiting the obvious ability to use multiple <pattern> elements (the availability of that option is simply because it is easier to read than really long single regex patterns).
Anyway, to replace a pattern-based format constraint with an XForms bind, you would use a type model item property that references an XML schema type library entry. In essence, you can build XML schema declarations in your form to declare types even if you don't use XML schema to define the element structure and the data types of the instance data. Here's what it looks like:
<xforms:model> <schema targetNamespace="http://test.org" xmlns="http://www.w3.org/2001/XMLSchema"> <simpleType name="alnum"> <restriction base="xsd:string"> <pattern value="[A-Za-z0-9]*"></pattern> </restriction> </simpleType> </schema> <xforms:instance xmlns="http://test.org"> <data> <node></node> </data> </xforms:instance> <xforms:bind nodeset="test:node" type="test:alnum"/> </xforms:model>
The inline schema declares a datatype named alnum (for alphanumeric) and indicates it is any string that can be restricted to the same pattern you saw in the XFDL format constraint. The XForms bind then associates that type with the data node. If you bind a field item's <xforms:input> to that data node, you'll see that the user input is indeed restricted to strings that match the pattern.
I learned something interesting about survey research recently. Given a survey question with a choice list answer, such as a list of icecream flavors or political parties, apparently there is a non-trivial segment of the population who read only the first few possible choices before rendering an answer. This segment can introduce significant bias into survey results unless the list of choices is randomized for each survey.
My recommended approach is to use web application code to put the choice list in the desired order and then push the list into the form before sending the form to an end-user for interaction. However, we increasingly see that people want the server side web application code to be as simple as "send form to user, and when it comes back, extract the data". As a result, front-end form logic is increasingly used in lieu of writing middle tier Java code.
So, here's a sample XForms single choice selection control that, due to the appearance attribute, would be presented as a group of radio buttons:
<xforms:select1 appearance="full" ref="/survey/icecream">
<xforms:label>Pick your favorite flavor of icecream:</xforms:label>
The set of choice items come from XForms instance data at the location given by the XPath location in the nodeset attribute of the itemset element. The label element then indicates what is shown to the user, and the value element indicates what internal data value corresponds to that display label.
<xforms:instance id="Flavors" xmlns="">
<choice value="CC">Cotton candy</choice>
Now, since we don't want to bias the results toward butterscotch and chocolate flavors, we want to randomize this list each time the question is presented to an end-user. The best solution would be if XForms had additional appearance keywords as custom namespaced extensions, something like this:
<xforms:select1 appearance="full xfdl:randomized" ref="/survey/icecream">
In current XForms, the appearance attribute only allows one keyword, and only the words minimal, compact and full are defined. The XForms working group agreed today to change the appearance attribute to a space-separated list to allow extension features like this ("xfdl:sorted" would be nice too, though we also recommend sorting a list before putting it in the form).
Supposing you do want to randomize a list of choices within an XForm, let's start by focusing on just creating a permutation when a form starts. Start with a list of any number of data values, such as 5 values, like this:
<xforms:instance id="Permutation" xmlns="">
<permutation count="" position="" random="" size="" swapvalue="">
Now, on form load, we'll create a permutation of them, like this:
<xforms:action id="GenPermutation" ev:event="xforms-model-construct-done">
<xforms:setvalue ref="instance('Permutation')/@random" value="random(true())"/>
<xforms:action while="instance('Permutation')/@count < instance('Permutation')/@size">
value="floor(random(false()) * (../@size - ../@count + 1))"/>
<xforms:setvalue ref="instance('Permutation')/@position" value="../@random + ../@count"/>
<xforms:action if="instance('Permutation')/@count != instance('Permutation')/@position">
<xforms:setvalue ref="instance('Permutation')/@count" value=". + 1"/>
The first line hooks the start of form processing, after the data model has been initialized. Next, we seed the random number generator and a couple of variables used in the main loop. The loop iterates through positions 1 to n-1 of an n-length permutation. For each position k, the value is swapped with a randomly determined position between k and n, inclusive.
Because XPath automatically concatenates the text nodes of a given XML tree or subtree, you can look at the whole permutation using a single XForms output, like this:
More importantly, you can use the permutation as a lens on the set of items shown by a selection control, such as an XForms select1, like this:
<xforms:select1 appearance="full" ref="/survey/icecream">
<xforms:label>Pick Favorite (random lens)</xforms:label>
Compared to the first code sample above, the only differences are the ref attribute expressions in the label and value elements, but they both do something interesting with the nodes produced by the itemset element. Typically, the itemset is used as a "for each" construct, and the label and value ref expressions drill *into* each node to obtain the label for an item and the data value corresponding to choosing that item. In this example, the itemset is really only used to obtain the correct number of items to generate and to set an initial context node within the set of items. The label and value ref expressions then pop up to the parent of that context node, and then drill down into a different choice element based on the order given in the permutation.
Below is a screen snap of the two select1 controls (and the XForms output for the permutation). On the left, you can see the radio button group with the items appearing in the original order of the data. At the bottom you can see the permutation that was generated when the form was rendered. Every time the form runs, a different order is selected, but this time, the order generated was 3, 1, 5, 2, 4. So, you can see on the left that "Cotton candy" is the 3rd choice, and so it appears first in the permuted choice list on the right. Similarly, "Butterscotch" is the first choice in the data, so it appears second in the permuted list on the right, followed by Vanilla, Chocolate and then Strawberry is last in the permuted list since it is 4th in the original list.
Finally, note that you only have to generate the one permutation of a particular size, which can be used as a lens on all selection controls that have the same number of elements as the permutation. If you have a survey in which different questions have different numbers of answers, then you'll need to generate a permutation for each different number of answers. However, once you have all the permutations you need, you can dynamically pick the permutation to use in the label and value ref expressions based on using the last() function to obtain the size of the itemset's node set.