A draft of the Building Composite Applications in Lotus Expeditor V6.1 redpaper has been posted to the IBM Redbooks site
Bring information and collaboration into the context of your daily business processes using composite applications
Matching: calow X
As we start talking about the possibilities for composition of "reusable" services (even UI ones), it's important to take a look around the industry and understand what some barriers to this could be.
I just happened to come across this article by David Chappell who was pessimistic about the reality of reuse with SOA. This article talks about the problem being one of larger context demands for business services that are not present for the very successful object level reuse of rich frameworks like Java and .Net.
In the chain of other responses that I ended up following, was this excellent IEEE article talking about The Social Side of Services that reflects the challenges that any reuse effort faces due to non-technical social reasons and approaches you can take from any level of the organization.
Points to ponder,
Today I want to talk about the different types of composite application frameworks that are available with WebSphere Portal 6. For some background: here is a "What's New in WebSphere Portal 6" developerWorks article and here is the Portal 6.0 InfoCenter main topic for Composite Applications (which is really talking about Instance Based Composite Apps, but I'm getting ahead of myself...)
Disclaimer: These are NOT official names, and I'm just experimenting on you brave readers to see how they resonate. Please provide us feedback on whether you love them or hate them.
Page Based Composite Applications
The traditional way that WebSphere Portal applications are created is by assembling portlets on a collection of related pages that are integrated into the primary MyPortal navigation hierarchy. These applications are either created by using the Portal based web composition tools or manually created using XML Access. Once they are integrated into the topology there is no real way to tell them apart from any of the other other pages, although there is usually a clear distinction at development and UX design time. Of course this style of creating pages is still fully supported in Portal 6. It's very important to note that there is no reason to stop developing these kinds of applications if your needs are being met (i.e. the new capabilities are not a replacement for what you are doing now, but additional tools to solve additional problems). Access control, page level properties and portlet entity level properties can be used to provide a little bit of local context, but all these applications are essentially running in the global Portal context and leveraging global user roles (e.g. user profile attributes for someone being a manager and/or global group membership). I'd like to call this type of application a "Page Oriented Composite Application" to reflect their focus on portal pages as a fundamental unit of aggregation. It doesn't mean that there can't be multi-pages though. I assume that readers are familiar with these from earlier releases of WebSphere Portal and will not go into more detail here. (For those who what a high level description, have a look at these articles written about Portal 5.1 by the Lotus software CTO Doug Wilson Part 1: Overview, Part 2: Cooperative Portlets, Part 3: Advanced Patterns)
Instance Based Composite Applications
In WebSphere Portal 6 we have added a new context level, an application instance. It augments and extends the capabilities of Page Based Composite Applications, and still supports everything you could do before. More precisely, I like to say that an "Instance Based Composite Application" ISA "Page Based Composite Application" in the conceptual OO inheritance sense. Instance based applications can do everything that page based ones can do and more. The difference is more significant than just tracking a new set of preference data though. An application instance is managed as a single unified entity (e.g. created, deleted, backed up, restored, access controlled, customized). Instance lives within the application instance catalog and are grouped by category. As part of implementing the ISA relationship, each application instance also manages it's own unique branch of the portal topology, with portlet entity and page level properties settable on a per instance basis. Due to configuration split (a major new feature of Portal 6), the instance specific branches are not in the MyPortal topology branch, but can be referenced from there via page links. There are also programming model capabilities that are associated only with the instance based model . These are application level roles with component role mapping, creation of new instances from templates (including component instance lifecycle), creating templates from existing instances, template parameters and policy driven administration.
Application level Roles
Application level roles can be defined using the application domain language (e.g. for a club: President, Treasurer, Member, etc). Application instance access control is done using these application level role names, and membership is instance specific. Thus, I can be the President of one club, and just a Member of three others. My user experience and allowed capabilities depends on my role in the specific instance (even if they were all created from the exact same template). Transitivity of access control can be accomplished by having these application level roles be mapped to roles defined by the component, using the component's domain language. This allows the template assembler to hide the details of the application domain to component domain mapping from the application instance owner and allows component's to be reused in many contexts.
Instances from Templates
Application instances are created from templates. A Template is a single XML file serialization of the important aspects of an application instance (and transitively all the contained component instances). Each component gets to decide what "important" means to it. The Portal component automatically serializes the entire portal page topology of the application instance along with all portlet entity properties. Thus, most standard portlets that use portlet entity properties to store customization state are already template enabled "out of the box". That means that you don't need to rewrite your existing portlets to leverage them in instance based composite apps. I'll go into the patterns for component creation ("By Reference" and "By Value") in a different blog entry, but, before then, here is a developerWorks article by Thomas Stober, Peter Fischer and Hendrik Haddorp. It walks through an example of creating a "By Value" component.
Templates are important to enable "self-service IT". They allow the ability to capture a common pattern of component composition and then allow the creation and management of new instances to be delegated to end users. The creator becomes the owner/administrator of these new application instances. The owner is in control of the membership of the application and can optionally be given rights to modify the instances that they own. (adding co-owners is also permitted)
Templates from Instances
Templates can also be used to capture subsequent organizational learning. As an owner customizes their instance to more closely match their particular business problem, it may evolve a long way from the original template. This new instance can then be "Saved as a Template" to capture that learning for reuse in other contexts. Templates can even be used as a unit of deployment by importing and exporting them from the template catalog.
Template parameters are a useful tool to allow for instantiation time customization of a new application instance by the eventual owner. When a new instance is created, a dialog is presented that asks the creator to specify one or more values (maybe a customer id, project id or other information applicable to the new context being created). Components use these instantiation time parameters to affect how they create their own instances and can automatically adapt their configuration so they are up and running right away (e.g. use the customer id to automatically filter a dashboard portlet view for just that customer). Expanding the richness of capabilities in this area is one of the things we are looking to do in future Portal releases.
Policy Driven Management
Policy driven application instance management is being added in Portal 6.0.1 to help to enforce instance based quota's and relevance (e.g. by archiving instances that have not been used in a long time), without manual IT intervention. This is done by sending email to owners and automatically locking the application instance if they go over quota and automatically archiving instances based on last used time. This helps an IT organization manage the infrastructure impact of proliferation of instances. It was developed based on administration experiences with the viral adoption of early versions of Lotus QuickPlace in big customers. (Note: Similar solutions have also been developed for more recent QuickPlace releases). More details of this capability will be available when 6.0.1 is released next year.
Conceptual Model for Instance Based Composite Applications:
Note: that the "Portal Topology Branch" ISA "Component Instance". That is because we have implemented a Business Component Adapter for the Portal itself and drive the lifecycle and templating of the portal topology in the exact same way as any component BC adapter that you could write. If you look in the resulting template.xml you will see that the portal topology XML is just component template instance data like any other.
Now that I've covered the high level conceptual architecture, I'd like to describe a taxonomy for use cases for this new instance based model.
Use Case Categories for "Instance from Templates"
The primary use for instance based applications is where there are going to be multiple instances of some entity, concept or process that share some common structure. (singleton instances are a special case of this) Furthermore, an instance can be further customized to better fit it's exact context (which over time can lead to singleton instances...). In truth, these three primary classifications categories blend together at the edges, but they have some differences that it makes sense to talk about them separately. Let's look at each in turn.
1) Multiple Concrete Things (need a flashy name here, any suggestions? :-)
Examples in this category are stores, departments, business partners, customers, clubs, promotional offerings. What they have in common is an association with some concrete "thing" that provides a external reason for the creation of a context. These "thing" domains also generally provides a natural terminology for the application level roles. (e.g. the roles in a club, the roles when interacting with a store, business partner, etc). Each kind of domain would likely be represented by a separate application instance catalog category and define one or more templates (e.g. there may be different levels of relationships with business partners that determines what the default set of components and capabilities would be for a new instance). The instances usually are usually oriented to interact with multiple existing systems of record (e.g. eCommerce, CRM) and the portlets are filtered for the specific context (e.g. showing just the orders of the customer associated with the instance). They could also mix in some of the team collaboration components like Wiki's, Discussions, Document Libraries (see below), but these are usually not the primary purpose of creating the instance. In a future post I'll explore the "offering" scenario in more detail. The primary components are of a "By Reference" type.
2) Team Collaboration
Collaboration in teams is similar to the above, but a team or project is a somewhat less concrete thing. I use this category to reflect uses where the primary purpose of the instance is to facilitate the collaboration of the team members towards a shared purpose (a project or other defining feature that creates a team). Another term for this could be community based collaboration. The instances are usually oriented around collaboration components (e.g. document library, discussion, Wiki, team calendar, etc) that have their data scoped to the instance. They could also bring in contextually relevant data from external systems of record (e.g. a defect dashboard summary from the bug tracking system). Thus the majority of components are the "By Value" types. In order to facilitate optimal collaboration in the team, individual instances may be customized by adding instance specific components (e.g. tying into to a team specific build system) or customizing specific components (e.g. specifying instance specific categories for a document library). After the holiday's I'll blog about a concrete example in this category.
I also split this category out because Lotus' collaboration heritage is in facilitating team collaboration using Lotus Notes and Lotus QuickPlace. The patterns we have learned in creating those products have been applied to the instance based infrastructure we have created for WebSphere Portal 6.
3) Business Process Centric
In this case, an instance is created to explicitly help in the execution of a complex business process. "Request For Proposal", "Employee Onboarding" "Statistical Instrument" are good examples of this. What makes this category different from the Team Collaboration category is the use of instance specific state (or state maintained in an associated Process Server workflow instance) to affect what is seen within the composition. This is based on the current process state and also allows the components to adapt themselves to the state via a "workflow state" component and cooperative portlets. You could even imagine peoples roles automatically changing as the process progresses. Many people use Lotus Notes to manually create these kinds of applications today and our vision is to allow business users to safely customize the processes without needing to be a developer. Preliminary steps towards this goal has been made generally available in Portal 220.127.116.11 this past Monday. Here is the online documentation for this feature. We expect to greatly enhance the capability in this space in future releases. I have a great scenario that covers this case that the Australian Bureau of Statistics has allowed me to share with you. I'll talk about that in another post (probably in the new year, given the backlog I've been accumulating as I write each post :-).
In summary, we've talked about the two kinds of user facing composite applications supported in Portal 6. They are 1) Page Based Composite Applications and 2) Instance Based Composite Applications. I talked about how Instance Based adds another level of context around an instance and a unified lifecycle and management story. I also talked about how instance based is a specialization of a page based composite application.
I finished up with a rough categorization of use cases for instance based composite apps. They are 1) Multiple Concrete Things, 2) Team Collaboration and 3) Business Process Centric.
In future posts I'll talk about patterns for component creation (e.g. the "By Reference" and "By Value" terms mentioned above) as well as giving some concrete examples for each of the three instance based use case categories.
Questions to the readers:
We had a wide ranging internal debate on the names for the two types of composite applications supported in Portal 6.0. In the end I picked the names that had pretty good internal consensus and resonated with me (since I was writing this blog entry :-).
What is your opinion on these names? Do they help clear up why there are two models and what some of the use cases are for the instance based one? Do they just confuse you more? Is there too fine a distinction being made? (and thus only of interest to us philosophically oriented architects)?
Do you like the use of the UML conceptual model? I really like the UML for it's information density and precision.
What about the categorization of the use cases? Are the names meaningful? Helpful? Anyone have a better name for the "Multiple Concrete Things" category?
Thanks for reading,
Now that I'm back from Thanksgiving vacation and caught up on all my email, I figured I'd start by defining what I mean when I talk about "Composite Applications". (and yes you can see my philosophical side being exposed here :-)
Let's start with a dictionary definition of composite. The only part of the definition even related to computing is "1: made up of distinct parts". So then a composite application is an application made up of distinct parts. That's pretty broad. The definition of distinct is "distinguishable to the eye or mind as discrete". Thus the questions and confusion in this space comes down to:
1) What are the parts that are distinct?
2) And to whose eye or mind are they distinct?
Computer Architecture Oriented Definitions:
The current general industry trend for the use of the term "composite application" is that it is related to application integration and Service Oriented Architectures (SOA). The parts are existing applications or whole systems of applications that are integrated together within some kind of composition framework.
Doing a quick Goggle search and just looking at some of the first page of results:
The IBM SOA Definition:
...composite applications, which are applications that draw upon functionality from multiple sources within and beyond the enterprise to support horizontal business processesFrom Loosely Coupled.com
An application built by combining multiple services. A composite application consists of functionality drawn from several different sources within a service oriented architecture (SOA). The components may be individual web services, selected functions from within other applications, or entire systems whose outputs have been packaged as web services (often legacy systems).Oasis has a Web Services Composite Application Framework TC
...define a generic and open framework for applications that contain multiple services used in combination (composite applications).Here is the Wikipedia entry (sorry Brian, they consider Mashups as composite apps and so do I...)
...expresses a perspective of software engineering that defines an application built by combining multiple servicesWhat do all these definitions have in common? The parts that are distinct are services and they are distinct in the eyes of the composite application assembler.
In general, the perspective is that the assembler is bringing together back end services (generally exposed as web services) to perform some kind of automated application integration. That is an important goal and great for automating those processes that can be automated. But this blog is about "Composite Applications for People" and by definition, people are not very involved in an automated process (except maybe to kick them off or solve "exception problems" that the computers can't)...
Relating this back to people:
To relate this back to people, we think about creating compositions by bringing the user interface of distinct systems together into a unified visible context. This is done to increase the productivity of the end user in executing the tasks that are part of the actual processes of doing business (vs just those processes that can be automated). It does it by providing the user access to relevant and appropriate information based on the current context (e.g. the kind of process or the step in the process). We'll talk much more about the importance of context in another post.
Developer created compositions (aka Mashups):
The assembler in all these case is still a developer and there is tight coupling between the user interface and the custom control. Each control generally has it's own custom API even if they are conceptually doing similar things (e.g. the different Map controls) This is also the level that most Mashups work today.
To facilitate development, there have been many "tools" created that allow for the rapid, visual assembly of controls into a user experience and binding the data between controls using gestures or scripting. Some Web 2.0 trends in this area are the "application wiki's". However, I still class these all as development tools and development created compositions.
Non-developer assembly (aka Portals):
In contrast to that we have the Portal pattern. This is were a "mini-application" component owns a piece of real estate and many of these components can be assembled together on a single visible page. This mini-application is one of our units of distinction and can be considered a UI service. I'll generically call it a Portlet . (this also happens to be what the Java world calls the standard component. In Eclipse it would be a ViewPart and Microsoft calls them Web Parts). This is the unit of aggregation, not necessarily the unit of deployment (e.g. in Java the unit of deployment is a web application archive or WAR and in Eclipse the unit of deployment is an OSGi bundle)
The key distinction with this pattern is the strong separation between developing the Portlet and it's deployment unit (which is still a developer task), deploying the Portlet's deployment unit (which is an administrator task) and the creating a composition of various Portlets, sourced from different, independent providers, on sets of pages to create a portal or a section of a portal (which is an administrator or end user task). We call this "UI level aggregation" or "integration on the glass". Portlets provide a fragment of the total UI for a specific back end application or service and are generally coarser grained than the above mentioned UI controls (but the Map controls above are a great example of the intended granularity).
In WebSphere Portal this distinction is supported by a specific framework with a standard component interface for plugging in the UI blocks and mediating loosely coupled interaction between them. A design goal was to allow for a variety of different programming models to create components (e.g. hand coded Java, JSP, Java Server Faces, Portlet Factory models, Lotus Component Designer, Lotus Domino, remote web service name your favorite web template system) unified by a common interface to the framework. This way, the composer sees all components as the same "kind" and picks based on the capability, not the underlying implementation language. Administration, roles and access control can also be handled at the high level component level.
We also use this same, centrally manged, composition definition to "project" to our rich client based "portal" in Lotus Expeditor (more about that in another post).
Now many developers out there will say "That's just bait and switch. You just created a component model at another layer of abstraction that everyone now needs to conform with. I can do the same thing with my favorite XYZ tool in my favorite GHI programming language". In one sense that is true (and it's what various 4GL's have done many times in the past). On the other hand, this interface level is intended to be coarser grained than a custom control, it's based on industry standards for both Java (Portlet 1.0 JSR 168, Portlet 2.0 JSR 286) and for interaction based web services in any implementation language (WSRP).
This has gone much longer than I intended and I need to save some words for other posts :) In summary I'd just like to reiterate that "Composite Applications for People" involves UI integration "on the glass" and a separation of the roles and skills between component creation and component assembly.
In upcoming posts we'll talk about some new features in WebSphere Portal 6.0 that add templates and application instances to the previous composition model, what we mean when we talk about business components and some business component patterns for "by-reference" and "by-value" component instances. We'll also talk more about the importance of the different levels of context, remote portlets, WSRP and rich client projection in Lotus Expeditor.
Thanks for reading,
[UPDATED 11/28/06 by Jeff Calow to remove references to the internal division name that was causing confusion and added links to product home pages]
Welcome to our new group blog. It brings together all the key technical architects and user experience design professionals for the major products from the Lotus software division that provide support for user facing composite applications. These products include IBM Websphere Portal 6.0, IBM Lotus Notes 8.0, IBM Lotus Domino 8.0, IBM Lotus Expeditor 6.1 and future products to be named when they are announced. The folks that will be helping with this blog on a regular basis are:
Art Thomas -> Lead architect for Domino administration
Bob Balfe -> Lead architect for Portal projection in Expeditor
Brian O'Donnovan -> Overall development manager for user facing composite applications
Craig Wolpert -> Lead architect for Notes/Domino ISV enablement
Eric Masselle -> User experience designer for composite application editing and portal document workflow
Jeff Calow (me) -> Chief architect for user facing composite applications
Kristina Beckley -> Lead user experience designer for composite applications in Notes/Domino
Niklas Heidloff -> Lead architect for composite applications in Notes
Peter Fischer -> Lead architect for Application Infrastructure in Portal
Stanley Dunne -> Lead architect for composite application composition tools
Thomas Stober -> Release architect for the next version of Portal
The impact of composite applications is so broad that we will also be having many guest speakers from across our product lines. Our goal with this blog is provide a personal, direct line of communication between experts working in this exciting and often confusing technology area and our customers and partners. Each poster brings their own diverse perspective and ideas to the conversation, so don't expect a marketing pitch or a single voice or style :-) We'll also be "trying out" various ways of explaining what we are doing and why, so we value your feedback in helping refine our technical story to be crisp and understandable. We'll also be posting direct questions to gather your feedback in time to affect ongoing product plans.
It's important to note that none of us speak officially for IBM, nor can we make product commitments or discuss products that have not been announced.
As a way of starting the conversation, each of the regulars will be posting a short bio of themselves and then a short description of how the product they are associated with fits into the Lotus software composite application strategy. We'll then be posting about topics that we find personally interesting as well as looking out into the industry. If you have a topic that you're burning to get information about, let us know.
The Lotus software Composite Application Team