Yesterday I presented a full paper on a new method and research prototype that describes our thinking on the strategic technical direction of the Lotus Forms document format and processing model. The paper is now available from the ACM Digital Library and is entitled "A REST Protocol and Composite Format for Interactive Web Documents". If you have one, use your institutional access to the ACM Digital Library. For IBMers, you can get access using KnowledgeGate.
As you may know, the document format for Lotus Forms is called XFDL, and it has grown and changed many times in the last 15 years to become more and more comformant to open standards. Once upon a time, back in the 1990's, XFDL was not even an XML format. Currently, the XFDL format is an XML vocabulary that describes a high precision multipage layout for an interactive Web 2.0 application, and XFDL incorporates other standards including XForms, XML Schema and XML Signatures to express dynamic behaviors, business rules, client-side data validation, web service access, and digital signature security.
This open standards transition has represented an important and substantial evolution of the Lotus Forms document format and processing model over time, and this new paper on "Interactive Web Documents" articulates the next step in the evolution of Lotus Forms. As with any strategic operation, it is the kind of thing that plays out over years in the product line, but I am used to that, and I am very excited by our progress.
The initially empty interactive web document template would describe the web application. To initiate a business process, the template is instantiated, becoming an interactive web document capable of holding XML data and file attachments provided by end-user participants and machine agents in the business process. The instantiation sets up a REST service keyed by a unique ID to the instantiated document, and simple GETs, PUTs and DELETEs are used to access and update the resources in the composite document. In particular, XForms submissions are used to synchronize data and change pages within the interactive web document. When a user completes their work on the document, it is the edit token, not the whole document that is passed along through the business process. In this way, the interactive web document and its REST interaction session become the central artifacts over which the entire business process operates, with document serialization only being required to transmit the interactive web document (i.e. the web application combined with its current state) across enterprise domain boundaries.
This is the general idea, and the paper contains numerous additional technical explanations and markup examples. It should be clear from this description, though, that this next evolutionary step will enable Lotus Forms to more easily adapt to and incorporate emerging Web 2.0 technologies, via its page basis in HTML and XHTML, and it will provide a quantum leap in performance of the Webform Server by avoiding a single large XML DOM for large forms applications, by offloading XForms processing to the client-side, and by avoiding serialize/parse operations during the course of the business process.
Happy birthday to me!
Here's to XForms ubiquity;
In FireFox and IE.
Happy birthday to me!
But seriously folks, IBM is a proud sponsor and contributor to the Ubiquity XForms open source project, and the Lotus Forms team has exciting plans for capitalizing on the ability to have XForms support directly within a web page.
Gone is the distorting lens of limitations imposed by the fraction of language features that the dominating browser makers are prepared to support today. The Ubiquity movement is all about freeing us, as consumers of the interactive web, to express open standards for software that meets our customers' technical requirements and deploy our solutions directly to the web.
Gee, with all this birthday fun, the next thing I really ought to do is go on a Safari!
My wife got me one of those GPS devices for Christmas. I really like it. Actually, I work closely with Steven Pemberton on the W3C Forms Working Group, and every face to face meeting we manage not to get lost because Steven has one of these devices. His talks a bit much and sounds just a little pouty when you don't follow her directions properly, though I think that's just a skin.
Perhaps needless to say, I've wanted one of my own for some time now because I'm pretty good at getting lost. I once took 90 minutes to complete a 7 minute journey-- well, it was 7 minutes according to Google Maps, but my version of "take the first left out of the San Jose airport" occured earlier than their version, and next thing you know, I was headed for LA rather than San Fran.
Anyway, so I visited Research Triangle Park recently for the Forms face to face, and although it is an easy trip from the airport to the hotel, I took my new GPS device along so I could get used to using it before I have to truly rely on it. Well, my plan went slightly wrong in that I ended up relying on it after all.
You see, I had dutifully printed off and memorized the Google Map instructions before leaving the rental car lot, so I knew exactly what I was supposed to do. I knew exactly what script to run. And sure enough, my handy GPS device was rattling off the correct command "Turn right onto Exit 280" when an unfortunate arrangement of traffic prevented me from executing the command. Then something wonderful happened. Without a pout, my GPS device said "recalculating" and next thing you know the device was in control of getting me to my destination.
It was then, when "recalculating" happened to me alone as I hurtled down the interstate somewhere near the speed limit, that I realized what an excellent example this was of the power and efficiency of declarative programming. I had declared my intent to arrive at a location. The rule was: get me from where I am to this location. The device computed the best way to get there. And when a change event occured, then the device recalculated a new sequence of instructions for me to run in order to achieve my objective. I suffered so much less than I did that time in California. It's just so much more effective. It's just so like XForms.
As promised, here are the links to my W3C Track talk at WWW2007 on the Rich Web Application Backplane (html, click here for the ppt file). Also, note that I heavily annotated my slides so you can read more about what I said than just the points on the slides. Let me know your thoughts on the content...
As we start off the business of a new decade, I already find myself looking forward to the improvements in XForms 1.2.
I know, I know, we just released the XForms 1.1 Recommendation, which contained a huge bundle of new features and architectural improvements. Still, the working group has been energetically advancing on still more features and improvements even as we closed the loop on the W3C process in the latter half of last year. I am very pleased with the completion of XForms 1.1, but the only constant is change, and we have an excellent array of new technical results to achieve in XForms.
The working group has put a fair bit of effort into the triage of features to separate nearer term (and simpler) objectives from those features of a more architecturally significant nature, which tend to get put in the "XForms 2.0" bucket.
My personal favorite "simple" feature is model-based UI switching with the switch element. This may seem like a bit of a nerdy thing to perseverate on, but the switch element is the only UI control we have that is not directly controllable by a UI binding to data. So, if a form author wants to switch UIs based on a user input, such as changing the payment details UI based on the selection of a payment, then the form author typically resorts to using group elements and the relevant model item property. But this is more than just the sense of completion we'll get from being able to say *all* UI controls can be declaratively linked to data. And it's more than just the satisfaction we'll get from providing the right tool for the job. The ability to declaratively link the state of a switch to data is critical to the save/reload web application pattern and the transaction digital signature web application pattern.
Other web application standardization efforts are focused mainly on what happens during execution of a single web page. XForms-based technogies are a step ahead because they can more easily address the user patterns related to collaboration on the web. Whether it is one user saving an application context and reloading it again in the future, or whether one user provides data to be loaded by a second user, the simple fact remains that data can be in any state along a continuum between empty template and fully completed, so web applications need to be declaratively responsive to that continuum. And nowhere is this clearer than when a web-based transaction is of high enough value to warrant a digital signature. A digital signature needs to be able to capture the full current state of the web application, and having a declarative binding to the data is the easiest way to achieve that goal since it means one can simultaneously sign the web application page template(s) and the data as separate resources.
For the record, Lotus Forms has for a few years now used a legitimate XForms extension mechanism, the xfdl:state attribute, to support model-based switching. We deemed this extension critical to our ability to address our customers' requirements for the above two web application patterns. Standardized integration of XForms with XML Signatures and other advanced security features like XAdES is, so far, in the XForms 2.0 bucket.
In conclusion, I'm not necessarily saying this is the "biggest" feature of XForms 1.2, only that it is my fave. I'll be giving some coverage to the other XForms 1.2 features in future blog posts, so please click the 'follower' button above (top right) and stay tuned.
Check out this cool video that will help you understand how and why your business processes need to lose the paper and get on board with dynamic, interactive electronic forms... powered by the XForms standard of course!
Lotus Forms 3.0 is now available. Click here for details and trial downloads.
Perhaps the biggest news overall is the performance enhancements to Web Form Server, which is a "Rich Internet Application" platform that provides the capabilities of Lotus Forms directly to web browsers without the need to install "rich client" software.
We have also added digital signature capability to Web Form Server. This is, of course, one case where a small amount of native code must be deployed by the server in order to perform the core cryptographic functions allowed by the end-user with their private key material. But this is a small and seamless download needed for digital signatures only. Again, no "rich client" deployment to maintain.
I am excited by the ease-of-authoring improvements in the Forms Designer, such as the Form Designer's enhanced datatype awareness. I am also very pleased with enhancements made to the many ease-of-authoring wizards, including the XForms-savvy Digital Signature wizard, the table calculation wizards, and the compute wizard. And the performance improvements made for large forms rival the improvements made to the run-times. No blog entry could do justice to the improvements to the designer; you just have to download the trial version and test drive it.
Perhaps the biggest overall cross-product change has been the performance improvement made available via a new language feature called on demand page loading. This feature is available in both the Web Form Server and the rich client Forms Viewer plugin (the latter of which is available for trial download). If you have multipage forms, this feature greatly reduces the time spent bringing up a form, and it avoids loading data structures for pages not visited by the end-user's navigation experience. This feature is combined with a new capability to do page-level validity checking on page-change operations so a user cannot progress through a wizard experience unless the validity criteria for that page are met.
The biggest overall language change goes to the support of bidirectional text and components for Hebrew and Arabic locales. This is a deep integration with the XFDL language that encompasses nested component orientations, text direction options, and even support for visual versus logical data representations.
Finally, on the XForms front, there are several XForms 1.1 features now supported in the product line, including having Lotus Forms that communicate with the web services of an SOA. This release also supports half a dozen new functions that facilitate everyday programming, such as date math and better string comparisons for sorting and searching of data. The release also supports the new XForms datatypes that allow empty content and simplify authoring.
To be sure, no blog entry could cover all the news of a major release, but these are some highlights I have found to be meaningful, and I hope you will download the trial versions and start finding out what you like best about Lotus Forms.
Within IBM, there is an "Extreme Blue for Employees" program called Biztech, which promotes innovative development within IBM by funding special projects developed by teams that can come from all across IBM. Today, the 6th Biztech Showcase is being held at the IBM T.J. Watson Research Lab, Yorktown. It consists of 12 projects including ours.
Our project, which will be presented this afternoon, is called Dual Forms. To understand the basic business problem, suppose you have a complex office document such as a contract, a patent filing or a new supplier agreement. It will have many sections to fill out, some with precise information like names, supply codes, dates, milestones, deadlines, and payment amounts. These are things that need to be validated in order to prevent errors, or which may need to match with data stored elsewhere. However, other parts of the document will contain free flowing text, such as special terms and conditions. The document may also contain pie charts, bar graphs, or other rich content that serves as visual aids. The key here is that the document is complicated and hard to complete, which frustrates those who must collaborate to complete and results in costly delays in completion.
Now imagine you had a technology that could augment the office document completion experience with:
- wizard behaviors that take the user step-by-step through the fill process and help correct errors immediately
- automatically invoke web services of an SOA to obtain data during the fill experience, rather than making the user responsible to type in matching data.
- digital signature security so that the legally binding agreement can be made electronically rather than having to print, sign, fax, scan and store document images.
- automatic submission of completed documents to the back-end workflow system or the repository of executed legal agreements.
This is the domain of Dual Forms. The Lotus Forms technology provides the above benefits as well as the ability to enclose any other files within a Lotus Form. One such enclosure can be an ODF office document. Lotus Symphony is an implementation that runs under the Lotus Notes application platform as well as the underlying Lotus Expeditor application platform. In this project, we've created a Lotus Notes composite application that allows the Lotus Forms Viewer to run under Notes, including read/write access to the running form. The composite application also hosts the Lotus Symphony ODF editors at the same time.
Using this project, the end-user can get access within a single application to the capabilities of office documents AND the capabilities of Lotus Forms (wizards, help, web services, signatures, document submission to the business process). The underlying technology that allows the two rich document formats to interoperate is... XForms.
I'm about to start packing my bags to head out for Lotusphere 2009. Sunday is Business Development Day (BDD), and I'll be presenting at the Futurists Panel on Dual Forms: An open standard office document mashup for document-centric business processes, and the broader context of Interactive Documents as Web 2.0 Applications. I'm very pleased to be part of such an exciting panel, which also includes Mary Beth Raven (of Notes fame, speaking on task management) and Mathew Flaherty (from the Lotus CTO's Office, speaking on the value of open source and open standards).I've seen their talks in rehearsal of course, and they're well worth your time.
In addition to the talk above, the Lotus Forms team has quite a showing at the conference again this year. Here's a list of where you can find us and when:
Lotus Forms Exhibits (in Oceana, Mon. 10:30AM-8:00PM, Tue. 9:00AM-6:00PM, Wed. 9:00AM-5:00PM)
Meet the Lotus Forms Developers in the Labs (on the Showcase Floor in the Dolphin Exhibit Hall)
- Electronic Forms Ped #5
- DB2 and Lotus Forms at Ped #6
Business Development Day: Sunday, January 18
BDD 204, 8:45-9:45amNew Lotus Industry Solution-based Marketing & Sales Initiatives (will include Lotus Forms)Click here to see full abstractSpeaker: Gary DolsenLocation: Toucan 1 and 2, Swan Hotel
BDD102, 2:45-3:45pmWebSphere Portal Product Family Applied to Real Business (will include Lotus Forms)Click here to see full abstractSpeakers: Gary Dolsen & Jeffrey SeifertLocation: Pelican 1 and 2, Swan Hotel
Meet the Speakers/Product Experts Session 5:00-6:00pmLocation: Fantasia Garden Pavilion, Swan Hotel
Sessions & Other Related Activities:
The Client Reference Lounge: The IBM Client Reference team is hosting ahigh-end hospitality suite as a way to say "Thank You!" to our client reference customers and to invite potential reference customers to stop by and learn more about the program. Open all week, this suite is available to you to hold customer meetings or to use as a respite.
Monday, January 19
1:00-2:00pmAD 507: BPM Made Easy: Looking forward with Lotus Forms and the Business Process AcceleratorClick here to see full abstractLocation: SW 1-2Presenters: Dave Manning, Tony Fiorot
2:15-3:15pmID604: What’s New in Lotus FormsClick here to see full abstractLocation: SW 1-2Presenters: Bob Levy, Dave Manning
3:45-4:45pmIDR604R: What’s New in Lotus FormsClick here to see full abstractLocation: SW 1-2Presenters: Bob Levy, Dave Manning
Tuesday, January 20
7:15-8:15amPartner Round Table: WebSphere Portal & Lotus FormsLocation: Swan, Dove hospitality suite (breakfast is included)
10:00-11:00amCustomer Panel: Succeeding with IBM WebSphere PortalSpeakers: Pat Kozak of C-Lock and Matt Garst (EIM) Attend this panel session to hear directly from customers who have realized business advantages through their WebSphere Portal and Business Process Acclerator implementations. Learn how each approached their business requirements, organized investment value assessments and technical strategies, and are now offering leading edge portal-based services. Hear their insights on approaches that work well, lessons learned, and their plans for portal platform expansions supporting future growth.
1:30-2:30pmPortal and Related Products Networking SessionLocation: Dolphin, Europe 5During this time customers can mingle with subject matter experts in a relaxed atmosphere. It is a great opportunity to thank the customers who contribute to our success. Invite your customers to come to the reference room, meet Portal and Forms experts encourage their participation in the reference program.
4:15-5:15pmAD514: Composing a Mashup Application with IBM Lotus FormsClick here to see full abstractLocation: SW 1-2Presenters; Dave Manning, Dave Shepherd
Thursday, January 22
7:00-8:00amBOF 107: Getting Started with IBM Lotus Forms TurboLocation: SW Parrot 1Presenter: Ricardo Rossi
For more information and free trial software for XML powered electronic forms web applications using XForms, visit the Lotus Forms website today.
I have to say it was no small perk for Lotusphere participants that the guest speaker for the opening general session was Neil Armstrong. Putting a man on the moon is probably the single greatest technical achievement of humankind, and it is a matter of great pride to be working for a company like IBM that had a hand in that achievement. Mr. Armstrong's description of the IBM computer with the 4K of RAM, no mouse or keyboard, and a 7-digit readout for a monitor was also a pertinent reminder of just how far we've come in the last 40 years.
Yet, no matter how things change, some things remain the same, which Mr. Armstrong proved with a wonderful story that included a priceless punchline. He described a scientific achievement that was being sought at the time: to determine the distance to the moon with an accuracy of less than one foot. To do this, scientists would measure the time it took for a laser pulse to travel from Mt. Lick Observatory to the moon and back. Mr. Armstrong said he did not serve as a scientist for this mission but rather as a field technician: "My job," he said, "was to install the mirror." Nice. Everyone in the computer industry can relate to a site visit, but going to the moon and back for the customer is usually more a figure of speech in our profession. Anyway, he then proceeded to describe the difficulties that were had with the experiment because the pulses were not able to be detected by the observatory on their return. It turned out that the latitude and longitude for the Mt. Lick observatory, established in 1889, were incorrect! Seems that if you want to achieve something great, you have to know exactly where you stand.
It was with that message in mind that the Forms team set out to meet customers, business partners and net new interested parties. For a number of customers interested in what will be new in our upcoming 2.7 release, we had a great time dazzling them with the full range of things they didn't know were in the current release. This was understandable in many cases since the current release came out after Lotusphere 2006 and after they had adopted our products. But it really made the show a lot more exciting for us since we were able to show interesting things they could get their hands on today as well as how we're making those features even better in 2.7.
In terms of feature requests and responses, we got lots of results. People really seem to like the attention we have been paying to simplifying the forms design experience in the Forms Designer, including the reorganized eclipse perspective, the snap-to geometry guidelines, the click and drag of data subtrees to create grouped controls, the point-and-click compute wizard and the three wizards for creating repeat tables and row and column calculations over those tables. We got lots of requests to make multipage forms faster in designer, viewer and webform serveer, and we got lots of requests for digital signature functionality. Most notable there was the desire to have an interface for the viewer to allow business partners to redefine the signing ceremony, e.g. to comply with German signature statutes. We definitely got a lot of good feedback to take to the lab.
In terms of net new offerings, perhaps the biggest additions to our 2.7 forms product line are the Streaming Forms API and the Services Platform. The Streaming Forms API is a pure Java module that understands our forms language, including XForms, and allows highly streamlined functions for extracting the data from completed forms to drive back-end business processes as well as writing data into the form to accomplish functions such as pre-population or mid-population of forms at the start of or during a business process. The services platform is designed to radically simplify the server-side development of forms processing modules. Each logical step needed to process a form is called a pipe, and the processing of a form can be viewed as a pipeline. This allows you to organize tasks such as form template prepopulation, consumption of web services during execution of a form, and the complex set of tasks that often occur once a form is completed to archive the form, and shred data from it to kick off various business processes. The services platform will include the ability to leverage IBM Websphere Datastage TX as a pipe, so you can get access to any data source that TX can touch!
Over the next month or two, I'll be blogging about these additions and all the important pieces of the upcoming 2.7 release. Stay tuned...
The Lotus Forms team will be out in force next week at Lotusphere 2008. To me, the most exciting presentations will be on a new product in development currently codenamed Total Forms. You can expect it to be shown in the Portal vignette during the OGS (opening general session), but there is also a focused session at 11:15 on the 23rd in SW Pelican. I don't want to spoil any surprises, so I won't say too much more, except it's a revolutionary new way to build XForms-based applications that really does put the computing power into the hands of the people. So go see the next killer app of Web 2.0.
Turns out the 23rd is a busy day for the forms team. At 10, there's a session on server-side mashups. Basically, we're talking new tooling that makes the Web 2.0 server accessible to non-coders. Again, putting computing power into the hands of the people, it's for people who want the next step up from Total Forms in application creation capability.
OK, so now I should take a step back because I started out talking about the 23rd because of Total Forms, but we actually also have a fine session coming on the 22nd at 4:15pm to tell you about all the cool new stuff in our latest 3.0 release.
And on the 24th at 8:30 sharp, there's a session devoted to the integration of Lotus Forms, Portal and Process Server. This session gives you what it takes for the full meal deal of forms applications, those involving human centered workflows that allow people to collaborate on the business processes they are involved in every day.
And rounding out the Lotus Forms development team's showing at Lotusphere is the "Meet the Developer" pedestal, where you can talk one on one with some of our best software architects, development team leads, and even a red-headed senior manager whose technical gifts are the very antithesis of Dilbert's pointy-haired boss!
It's an exciting time to be in the Lotus division of IBM! The Workplace Forms team is busily preparing for numerous presentations and labs to be held at the annual Lotusphere conference next week. This is the time to see the best of what Lotus has to offer, so of course the Workplace Forms team has lots of content to contribute.
Look for Mike Rhodin, the chief of chiefs in Lotus, to include Workplace Forms in his demonstration of social networking software.
Look for strong presentations, round tables and BOFs throughout the week.
- Monday: What's new in IBM Workplace Forms.
- Integrated Business Processes by Extending IBM Lotus Domino Applications with IBM Workplace Forms
- Bruce Lyman of EIM/US Air Force on Achieving New Business Horizons with Expanded Portal Solutions
- IBM Workplace Forms and Inter-Portlet Communication for Next-Generation Portals
- BOF: Forms Customer Experiences and Best Practices featuring Ali Hosseini
- Wednesday: IBM Workplace Forms Designer 101
- Thursday: Hands-on Session with IBM Workplace Forms Designer: Advanced Forms Design
All week you can Meet the Developers at two labs. You'll meet Mike Mansell and Chris Phillips to talk about the Workplaces Forms Services Platform, a new offering that streamlines the development of the server-side processing within Forms Applications. And you'll meet Veronika Irvine and me, who are there to show you anything you want to see on the Workplace Forms Designer, Viewer and Web Form Server products, including what's coming in the upcoming 2.7 release.
And finally, take some time to visit the User Experience lab, where you can meet Jeff Sokolov and test drive the Workplace Forms products and provide your own feedback on what you like and what you'd like to see in the future of these products!
Modified by John M. Boyer
Machine learning today is every bit as calculated, as simulated, as is machine intelligence. It is easier to use machine intelligence to highlight how much greater human cognition is, which is why I've been using a machine intelligence algorithm over the last several entries. However, the conclusion drawn so far is that, while machine intelligence is only simulated, it is still quite effective and valuable as an aid to human insight and decision making. Machine learning offers another leap forward in the effectiveness and hence value of machine intelligence, so let's see what that is.
Machine learning occurs when the machine intelligence is developed or adapted in response to data from the domain in which the machine intelligence operates. The James Blog entry only does this degenerately, at a very coarse grain level, so it doesn't really count except as a way to begin giving you the idea. The James Blog entry plays a game with you, and if he loses, he adapts by increasing his lookahead level so that his minimax method will play more effectively against you next time. In some sense, he learned that you were a better player. However, this is only a single integer of configurability with only a few settings of adjustment that controls only one aspect of the machine intelligence algorithm's operation. To be considered machine learning, a method must typically have a more profound impact on the operation of the algorithm, with much more adaptation and configurability based on many instances of input data. An example will clarify the more fine grain nature of machine learning.
The easiest example of which I can think is a predictive analytic algorithm called linear regression. Let's say you'd like to be able to predict or approximate the purchase price of a person's new car based on their age. Perhaps you want to do this so that you can figure out what automobile advertisements are most appropriate to show the person. Now, as soon as you hear this example, your human cognition kicks in and you rattle off several other likely variables that would impact the most likely amount of money a person is willing to spend on a car, such as their income level, debt level, nuclear familial factors, etc. This analytic technique is typically called multiple linear regression (MLR) exactly because we humans most often dream up many more than two variables that we want to simultaneously consider. Like most machine learning techniques, MLR does not learn of new factors to consider by itself. It only considers those factors that a human has programmed it to consider. When they are well chosen, additional variables typically do make an MLR model more effective, but for the purpose of discussing the concept of machine learning, the simple two-variable example suffices since your mind will have no problem generalizing the concept.
Suppose you have records of many prior car purchases, including a wide and nicely distributed selection of prices of the cars and ages of their buyers. This is referred to as "training data". If you plotted the training data, it might look something like the blue points in the image below. Let purchase price be on the vertical Y axis since it is the "dependent" variable that we want to predict, and let age be on the X-axis since it is a predictor, or "independent" variable. MLR uses a standard formula to compute a "line of best fit" through the given data points, again like the one shown in red in the picture.
A line has a formula that looks like this: Y=C1X1+C0, where C1 is a constant that governs the slant (slope) of the line, and C0 is a constant that governs how high or low the line is (C0 happens to be the point where the line meets the Y-axis, and the line slopes up or down from there). If we had more variables, then MLR would just compute more constants to go with each of them. For example, if we wanted to use two variable predictors of a dependent variable, then we'd be using MLR to create a line of the form Y=C2X2+C1X1+C0.
Technically, MLR computes the constants like C1 and C0 of the line Y=C1X1+C0 in such a way that the line minimizes the sum of the squares of the vertical (Y) distances between each data point and the line. For each point, we take its distance from the line as an amount of "error" in the prediction. We square it because that gets rid of the negative sign (and, less importantly, magnifies the error resulting from being further from the line). We sum the squares of the errors to get a total measure of the error produced by the line, and the line is computed so as to minimize that total error.
Once the constants have been computed, it is a trivial matter to use the MLR model as a predictor. You simply plug the known values of the predictor variables into the formula to compute the predicted Y-value. In the car buying example, X1 is the age of a potential buyer, and so you multiply that by the C1 constant, then add C0 to obtain the Y-value, which is the predicted value of the car.
In this way, hopefully you can see that the MLR "learns" the values of the constants like C1 and C0 from the given data points. Furthermore, the actual algorithm that produces the machine intelligence only computes the result of a simple linear equation, so hopefully you can also see that the predictive power comes mainly from the constants, which were "learned" from the data. In the case of the minimax method, most of the machine intelligence came from the algorithm, but with MLR-- as with most machine learning-- the machine intelligence is for the most part an emergent property of the training data.
Lastly, it's worth noting that there are a lot of "best practices" around using MLR. However, these are orthogonal to topic of this post. Suffice it to say that just like the minimax method has a very limited domain in which it is effective as a machine intelligence, MLR also has a limited domain. For example, the predictor variables (the X's) do need to be linearly related to the dependent variable in reality. However, within the limited domain of its linearly related data, MLR is quite effective and an excellent example of a simple machine learning technique that produces machine intelligence within that domain.
Continuing with the amazing stuff you can do with the eval() function: You can use it in a user interface binding to enable your form to programmatically control what the user sees.
As an exhibition of this capability, I'll give you the pertinent parts of an XML editor form that dynamically adjusts to XML structure and allows you to edit the content of any leaf nodes while giving you link buttons to allow you to drill deeper into element subtrees as well as a "back" button to go to the parent of any subtree whose leaves you may be editing. It starts with an XForms repeat, like this:
<xforms:repeat nodeset="eval(repeatexpr)" id="XMLEditor">
The repeat expression is computed by the form and is changed by user actions that drill deeper into the XML tree or go back to parent elements. The repeat expression will end with "/*" so that the controls in the repeat will show the children of whatever node the repeat expression selects before the "/*".
For simplicity, I've put the XML data to be edited as the first element of the instance that also manages the calculation of the repeat expression, but you could do this as two separate instances instead. Here's the instance structure I used in this example:
<xforms:instance id="data" xmlns="">
<Item-Cost type="currency" value="Price * Quantity"/>
<Subtotal type="currency" value="sum(Order-Details/Item/Item-Cost)"/>
<Tax type="currency" value="Subtotal * 0.05"/>
<Total type="currency" value="Subtotal + Tax"/>
The first element could be anything, but I used a "purchase order" data structure, so this form will magically morph into a purchase order editor. Further, it should now be clear why in the last blog I concentrated on data that carried its own formula calculations and data validation rules. If I replace the Purchase-Order element above with the loan calculation data below, then this same form will help calculate your monthly payment on a loan:
<Loan-Application label="ACME Used Car Loan Application">
<Name required="true">John Q. Public</Name>
<Address required="Name != ''"></Address>
<City required="Name != ''"></City>
<StatProv required="Name != ''" label="State/Province"/>
<PostalCode required="Name != ''"/>
<Employer required="../Principal > 20000"/>
<Principal required="true" constraint="Principal > 0 and Principal <= 50000"></Principal>
<Duration required="true" constraint="Duration > 0 and Duration <= 84"></Duration>
<Interest-Rate constraint1="Interest-Rate > 0"
constraint2="Interest-Rate <= 5+floor(Duration div 12)"></Interest-Rate>
<rate hidden="true" value="Interest-Rate div 1200.0"/>
relevant="Monthly-Payment > 0"
value="choose(rate > 0,
Principal * rate div (1.0 - power(1.0 + rate, -Duration)),
Principal div Duration)"
relevant="Monthly-Payment > 0"
value="Monthly-Payment * Duration"
Within the repeat, we can use different kinds of form controls to be responsive to the identified types of data and also to the issue of whether something is an input or an output based on whether it has a computed value. Here are two examples at the XFDL+XForms level:
<xforms:input ref="self::node()[not(*) and not(@value) and @type='currency']">
<xforms:label><xforms:output value="choose(./@label, ./@label, concat(translate(local-name(.), '-_', ' '), ': '))"/></xforms:label>
<xforms:output ref="self::*[not(*) and @value and @type='currency']">
<xforms:label><xforms:output value="choose(./@label, ./@label, concat(translate(local-name(.), '-_', ' '), ': '))"/></xforms:label>
<display compute="xformsenabled=='on' and value != 'NaN' ? 'on' : 'off'"/>
In the predicates of the form controls, "not (*)" ensures that these form controls are only relevant if the data node is a leaf that is to be filled with character content. The "value" attribute in the data provides a calculation formula, so that has been used to distinguish when to provide and input versus an output form control. The two examples above make relevant form controls for data elements annotated with a currency type attribute. Other form controls for checkboxes and dates can be created to bind to types like booleans and dates.
Next, let's look at how the repeat expression is computed:
<xforms:bind nodeset="firstelem" calculate="local-name(instance('data')/*)" />
<xforms:bind nodeset="path" calculate="concat(../firstelem, ../expr)" />
<xforms:bind nodeset="repeatexpr" calculate="concat(../path, '/*')" />
By design of this particular "magic morphing" (XML editor) form, the first element in the computed XPath expression is the first element of the instance. Then we compute the full path to the element whose children will become editable by the repeat and the form controls within the repeat. This is based on the "expr" data node that will be programatically manipulated. Because expr is initially empty in the instance data above, that means you will initially see the children of the first element of the instance, because the repeatexpr is calculated to the first element plus the initially empty "expr" plus "/*" to get the children.
OK, so how do we adjust what the XForms repeat presents to the user? Basically, we want to either add a child element name to drill down into a subtree or we want to subtract a child element name to go up a level. First, let's cover how to add an element, i.e. add a step to the location "path". Inside the repeat, each child element that is a subtree root (has children) gets an XForms trigger in a link style button. If you activate the trigger (press the button) then you drill down into the corresponding node. Here's what that looks like:
<xforms:label><xforms:output value="choose(./@label, concat(./@label, ' >>'), concat(translate(local-name(.), '-_', ' '), choose(*/* or *='', '', concat(' - ', *)), ' >>'))"/></xforms:label>
value="concat(., '/', local-name(context()),
choose(count(context()/../*[local-name(.)=local-name(context())]) = 1, '',
concat('[', 1 + count(context()/preceding-sibling::*[local-name(.)=local-name(context())]), ']')
<xforms:setindex repeat="XMLEditor" index="1"/>
The trigger ref binds to a node that has children, as tested by the predicate "[*]". The label shows the name of the child element whose subtree you will drill into if you activate the trigger. The action sequence simply chucks a slash plus that name onto the end of the "expr" as a new step in the location path. This adds to the "path" which adds to the "repeatexpr" which updates the XForms repeat to show the children of that subtree root.
The trigger to go back up to a parent from a child is something that would live outside of the repeat because you only need one "back" button. It's actually a bit trickier because you can't get the last slash in order to lop off the last location step in the path. Fortunately, XPath lets you find the first occurrence of a substring, and XForms actions include a loop. So, the way I did this was to construct a new expression out of all the location steps in the old one, except the last, which was detectable by there being no more slashes. Here's what that looks like:
<xforms:trigger id="GoBack" ref="expr">
<xforms:setvalue ref="../scratchexpr" value="substring(../expr, 2)"/>
<xforms:action while="contains(../scratchexpr, '/')">
<xforms:setvalue ref="." value="concat(., '/', substring-before(../scratchexpr, '/'))"/>
<xforms:setvalue ref="../scratchexpr" value="substring-after(., '/')"/>
<xforms:setindex repeat="XMLEditor" index="1"/>
The first setvalue copies the "expr" less the leading slash into the "scratchexpr". Then, we clear out the "expr" so we can build it up anew from the parts of the scratchexpr. Now, we execute while "scratchexpr" still contains a slash, so the loop stops short of copying the last location step from scratchexpr to expr. Once the processing is complete, then once again, the modifications made to expr, reverberate to "path" then to "repeatexpr" due to the XForms binds above, and so the XForms repeat updates to show and allow editing of the content of the parent element.
And that's it! Thanks to eval() used in combination with all other pre-existing features of XForms, you can make a form that edits any XML element data structure.
For a very long time, the XML language underlying Lotus Forms (called XFDL) has offered an option element called <format> to describe data type, validity constraints and presentational formatting for the (valid) values of form controls (called XFDL items, like text entry fields, selection popups, and so forth). Neither a whole Lotus Form document, nor the XForms instance data it contains, can be submitted to a server unless all relevant data type and validity constraints are satisfied.
However, the <format> elements are placed into the form controls, so the user interface layer must be articulated in order to create the run-time objects described by the <format> elements. This makes it harder to use Lotus Forms performance enhancing features like "on demand page loading" in a multipage XFDL document. It will also conflict with an increasing number of performance enhancing features over time. Suffice it to say, it would be better to express <format> data type and validity constraints as XForms binds in the form.
Sometimes this is easy to do. The following are a few examples. Suppose you have an XFDL <field> text entry element containing <xforms:input ref="data"> as the form control binding it to a data node named "data".
Change: <format> <datatype>integer</datatype> </format>
To: <xforms:bind nodeset="data" type="xforms:integer"/>
Change: <format> <datatype>float</datatype> </format>
To: <xforms:bind nodeset="data" type="xforms:double"/>
Change: <format> ... <constraints><length><max>10</max></constraints> </format>
To: <xforms:bind nodeset="data" constraint="string-length(.) < 10"/>
Change: <format> ... <constraints><range><min>18</min></constraints> </format>
To: <xforms:bind nodeset="data" constraint=". >= 18"/>
Change: <format> ... <constraints><mandatory>on</mandatory></constraints> </format>
To: <xforms:bind nodeset="data" required="true()"/>
These are easy mechanical changes to make that prepare a Lotus Form for use with performance enhancing features like "on demand page loading". The XForms binds are easy to create with the "Create Bind" or "Create Bind with Wizard" features in the Lotus Forms Designer's XForms Model Editor.
One aspect of <format>
option capability that is... somewhat less mechanical... to replace are constraints based on regex (regular expression) pattern matching. If you don't know what a regex is already, just do a web search for "regex tutorial" and spend an hour or two reading something that will save you days upon days of your life in the long run.
As an example, here is a simple XFDL <format> that applies a regex pattern limiting user input to zero or more alphanumeric characters:
<format> ... <constraints><patterns><pattern>[A-Za-z0-9]*</pattern></patterns></constraints> </format>
The * means zero or more characters, the brackets are used to express the range of characters, and subexpressions like A-Z and 0-9 provide subranges of the acceptable character range. Regex is a lot more powerful than this, so you really should read that tutorial if you needed this explanation. For example, you can express multiple alternative patterns directly in regex without exploiting the obvious ability to use multiple <pattern> elements (the availability of that option is simply because it is easier to read than really long single regex patterns).
Anyway, to replace a pattern-based format constraint with an XForms bind, you would use a type model item property that references an XML schema type library entry. In essence, you can build XML schema declarations in your form to declare types even if you don't use XML schema to define the element structure and the data types of the instance data. Here's what it looks like:
<xforms:model> <schema targetNamespace="http://test.org" xmlns="http://www.w3.org/2001/XMLSchema"> <simpleType name="alnum"> <restriction base="xsd:string"> <pattern value="[A-Za-z0-9]*"></pattern> </restriction> </simpleType> </schema> <xforms:instance xmlns="http://test.org"> <data> <node></node> </data> </xforms:instance> <xforms:bind nodeset="test:node" type="test:alnum"/> </xforms:model>
The inline schema declares a datatype named alnum (for alphanumeric) and indicates it is any string that can be restricted to the same pattern you saw in the XFDL format constraint. The XForms bind then associates that type with the data node. If you bind a field item's <xforms:input> to that data node, you'll see that the user input is indeed restricted to strings that match the pattern.