UI automation in the single sign-on world, Part 1: Bridging the ISAM ESSO integration gap

A new solution

This two-part series provides a solution for some unsupported single sign-on (SSO) scenarios when deploying IBM® Security Access Manager for Enterprise Single Sign-On (ISAM ESSO). Learn about the current SSO and operating system alliance model and why, for some software technologies, ISAM ESSO can't provide full integration. See how you can apply the powerful UI Automation and Active Accessibility architecture mechanism in the SSO world. Part 2 of this series will provide application examples with useful ISAM ESSO profiles based on the latest UIAutomation helper plug-in.


Oktawian Powązka (Oktawian.Powazka@pl.ibm.com), Software Services Consultant, IBM China

Photo of Oktawian PowązkaOktawian Powązka has worked as a services consultant and software engineer for a couple of internal TAM E-SSO IBM projects, mainly in Europe. He is currently working on the TAM E-SSO GIAMA project. Oktawian has four years of experience handling complex solutions in the TAM E-SSO arena.

28 May 2013

Single sign-on integration architecture

IBM Security Access Manager for Enterprise Single Sign-On (ISAM ESSO) is a solution that supports a huge variety of software applications on the Windows™-based platform. By design, ISAM ESSO takes advantage of different single sign-on (SSO) techniques depending on the application types. The differentiation is based on the wide spectrum of existing software technologies whereby each technology requires a separate approach. For example, consider web-based software solutions. Each of the widely known browsers requires a unique software plug-in to integrate itself with the outside world. ISAM ESSO delivers two such plug-ins: one for Internet Explorer and another for Firefox.

Figure 1 shows the schema of plug-ins used by ISAM ESSO to integrate with the corresponding software component. It is a somewhat simplified model; in reality, WinSSOAgent is loaded into every process participating in some other aspects of the SSO procedure. Read "ISAM ESSO under the hood" to learn how this ISAM ESSO loading mechanism really works (see Resources for a link).

Figure 1. ISAM ESSO plug-ins to integrate with other software components
ISAM ESSO Plug-ins to integrate itself with other software components

For the sake of simplicity, I've intentionally omitted any annotations concerning support for Java-based applications.

Now for the tricky part. ISAM ESSO works well as long as a given website is entirely based on Document Object Model (DOM) architecture, where each object can be easily addressed using the XPath query language. XPath facilitates HTML or XML document navigation and allows an easy selection process of all the respective elements and attributes that constitute the web page. During Internet Explorer or Firefox initialization, an ISAM ESSO plug-in loads itself into the browser and continuously monitors the web content of the profiled websites by inspecting the DOM tree (such as HTML or XML structure).

A problem arises when a website embeds a component with internal structure that is not compatible with DOM architecture. Two of the most famous examples of this problem are Flash/Flex from Adobe and Silverlight from Microsoft™. Components written using such technologies cannot be parsed by the XPath language and would require supplemental solutions to unveil their hidden structures.

A similar problem occurs when a software component is written to be used as a stand-alone application that runs on the Windows platform. As with websites and DOM structure, applications on the Windows platform can be written using a native Win32 API interface (usually with MFC or Windows Forms wrappers around it) or can take advantage of more advanced graphical user interface (GUI) frameworks such as Windows Presentation Foundation (WPF). The next section looks at those architectural differences.

Windows GUI model

When an application's user interface (UI) design is based on the Win32 API, it essentially means the application takes advantage of basic UI controls available within User32 or COMCTL32 Windows libraries. In the Win32 paradigm, all UI controls are really child windows that an application uses in conjunction with the main application window to enable smooth user interaction. Almost every UI control encapsulates the so-called Window Procedure, which establishes the main point of contact with the parent window and the whole operating system. This Window Procedure bonds every UI control with the Windows message-handling mechanism constituting a sort of unified message-driven architecture. This is also true for a wide range of UI component wrappers such as Microsoft Foundation Class, Windows Forms, and Windows Template Library.

In each case, ISAM ESSO takes advantage of an API hooking mechanism to monitor the message-handling procedures. By using Win32 and the hooking API it's possible to monitor and also to intercept and simulate Windows messages directly. If you have an interest in getting this message-driven architecture it's strongly recommended you become familiar with WinSpector, which is a tool that lets you easily keep track of most of the Windows messages flowing between operating system components.

The more advanced Windows GUI interfaces can have SSO problems. WPF, one of the leading frameworks, provides a consistent model for building applications. It provides a separation between the UI and the business logic. For our context, the most important aspect of the WPF application is that it uses DirectX to render most of its UI controls and other graphic elements. The WPF framework is no longer based on the native Win32 API to create and manipulate windows, and there is no place for any message-handling procedures. WPF treats every GUI object in the same manner, whether it's a control, an image, or a video—everything can be mixed together and displayed in unified fashion.

Figure 2 depicts a somewhat simplified architecture model showing the conceptual relationship between the different UI frameworks and operating system elements. Clearly, there are a lot of mutual interactions within this whole structure.

Figure 2. UI frameworks relationship
UI frameworks relationship

As seen in Figure 2, the DirectX rendering aspect of UI controls is not exclusive to WPF. Silverlight and Flash take direct advantage of the same concept. This also explains why, during ISAM ESSO profile creation, neither Flash nor Silverlight controls can be recognized as Win32 native controls when embedded into a website.

Figure 2 also shows more elements of the overall Windows GUI architecture. Graphics Device Interface (GDI) exports some of the primitive drawing functions and encapsulates many hardware complexities. Most of the basic UI controls from Win32 API (User32 and COMCTL32) are built upon GDI functions that provide the actual UI rendering. Some types of programs will also call GDI functions directly to perform lower-level UI controls, drawing within a window previously created using Win32 API functions. If this is the case, ISAM ESSO support is also out of scope.

The idea of a UI design shift from pure Win32 into more advanced GUI solutions actually makes a lot of sense, considering the initial Win32/GDI interface was designed about 20 years ago when CPU accelerated graphic cards were in the infant stage. Contrary to popular belief, GDI is to some extent hardware accelerated (depending on the OS version). But, because it keeps a lot of legacy implementations alive, it can't compete with fully accelerated frameworks like DirectX.

The next section switches back to some ISAM ESSO profiling scenarios.

SSO unsupported scenarios

This section discusses four application examples that, by design, are not supported by currently available ISAM ESSO plug-ins. Each example will introduce a different type of UI framework. The examples include (see Resources for links):

  • The whole operating system accessed as a web application based on Silverlight technology: SilveOS
  • A web application written using the Flex framework: Flex sample application
  • A stand-alone application (part of SDK) based on the WPF framework: Logon Screen Demo
  • A website entirely based on the Flash framework: Just AdFlavor

Part 2 of this series will use these same examples to create four ISAM ESSO profiles by taking advantage of the UIAutomation helper plug-in.

Flex and Flash are separated because in reality they are two different frameworks that Adobe uses for generating multimedia-based content.

Let's see what profile developers will get when they try to unveil the document structure of the examples listed above using ISAM ESSO AccessStudio. Each case is numbered separately in Figure 3.

Figure 3. Four signatures generated using ISAM ESSO AccessStudio
Four signatures generated using ISAM ESSO AccessStudio

There is a very significant issue related to embedding Flash and Silverlight multimedia-based content into the website. Components written in Flash and in Silverlight can be blended with the website using one of the following methods:

  • Using the windowless, or "opaque," mode.
  • Using a dedicated window integrated with the web page

You can see these two methods in Figure 3. The first (Silverlight) and the second case (Flex) both use the windowless mode, which is why AccessStudio generated somewhat undefined HTML content signatures. AccessStudio was not able to distinguish the difference between HTML structure and the embedded multimedia object. The main reason to use windowless, or opaque, mode is to blend multimedia content with a given HTML document.

The third case in Figure 3 involves a stand-alone WPF application so the window/windowless aspect is not an issue here. In the fourth case, the browser has assigned a dedicated window instance for the Flash application, subsequently rendering its content independently from the underlying web page. Thus, AccessStudio was able to easily spot the difference and construct the proper signature for the embedded object. For this specific website, using window or windowless mode would not make any visual difference from the end user point of view.

Admittedly, using this windowless mode in cases with more sophisticated websites might seem advantageous from a UI perspective, but it includes some serious limitations. The most crucial limitation is that animation and video performance are somewhat affected by this mode.

The window/windowless aspect is emphasized here because UI automation technology doesn't support windowless mode.

The next section explores how to cope with this limitation with some nifty ways to disable the windowless mode by using VBScript directly from the ISAM ESSO profile. You will have a chance to see for yourself if there is any visual difference when the windowless mode is disabled during the process.

The good news is that the windowless mode problem needs to be addressed in only about 10-20% of all the existing Silverlight and Flash websites.

The windowless mode

Listing 1 and Listing 2 show two examples of HTML code where windowless (opaque) mode has been enabled.

Listing 1. Flex multimedia content embedded using opaque mode
var props = new Object();
props.swf = "http://wwwimages.adobe.com/www.adobe.com/devnet-apps/flex/
props.id = "Main";
props.w = "1024";
props.h = "768";
props.ver = "5";
props.wmode= "opaque";
var swfo = new SWFObject( props );
Listing 2. Silverlight content embedded using windowless mode
<object data="data:application/x-silverlight-2," type="application/x-silverlight-2" 
   width="100%" height="100%">
  <param name="source" value="ClientBin/Silveos.Core.xap" />
  <param name="onError" value="onSilverlightError" />
  <param name="background" value="white" />
  <param name="minRuntimeVersion" value="4.0.50401.0" />
  <param name="uiculture" value="en-US" />
  <param name="initParams" value="IpAddress=" />
  <param name="enablehtmlaccess" value="true" />
  <param name="enableNavigation" value="false" />
  <param name="windowless" value="true" />
  <param name="autoUpgrade" value="true" />
  <param name="splashscreensource" value="ClientBin/Silverlight.Splash.xaml"/>
  <param name="onSourceDownloadProgressChanged" 
         value="onSourceDownloadProgressChanged" />…

Listing 2 was taken from the SilveOS website shown in Figure 4. The multimedia content occupies the whole browser window so, as expected, there is no visual difference when switching from one mode to another.

Figure 4. SilveOS - operating system written entirely in Silverlight technology
‘SilveOS' - operating system written entirely in Silverlight technology

Now it's time to create a simple VBScript (embedded in your ISAM ESSO profile) to switch from one window mode to another. Figure 5 outlines the concept; the relevant part designed for the example script execution is emphasized. The only profile trigger required for this task is to get the VBScript invoked just after the web page completes its loading phase. Sometimes, due to particular website performance problems, it may be necessary to apply a short delay before script invocation. Obviously, such a delay needs to be synchronized with each corresponding website.

Figure 5. Complete ISAM ESSO profile based on UIAutomation plug-in used for SilveOS website
Complete ISAM ESSO profile based on UIAutomation plug-in used for ‘SilveOS' website

Listing 3 shows the entire VBScript designed to parse website structure and disable windowless mode.

Listing 3. VBScript
1 On Error Resume Next
 2 Set pc          = runtime.GetPropertiesContainer()
 3 Set doc         = runtime.GetHTMLDocument()
 4 Set objectElem  = doc.getElementsByTagName("object")
 5 Set objectparam = doc.getElementsByTagName("param")
 7 pc.SetPropValue "objectElem Length", objectElem.length
 8 pc.SetPropValue "param Length", objectparam.length
10 i=0
11 For Each ChildElem In objectElem
12 Found = false
13  For i=0 to objectparam.length-1
14     If (LCase(objectparam(i).name) = "windowless" And LCase(objectparam(i).value) = 
          "true") Then                 'Silverlight
15        objectparam(i).value = false
16        outerHTML = ChildElem.outerHTML
17        If (InStr(LCase(outerHTML), "<param name=""windowless""")=0) Then   
             'If IE < 10...'param name' is not included in the website
18            objectDescEnd = InStr(LCase(outerHTML), "</object>")
19            outerHTML = Left (ChildElem.outerHTML, objectDescEnd-1)
20            For j=0 to objectparam.length-1
21               outerHTML=outerHTML& " <param name="""& objectparam(j).name &""" 
                   value="""& objectparam(j).value &"""> "
22            Next
23            outerHTML=outerHTML& " </object>"
24        End If
25        ChildElem.outerHTML = outerHTML
26        pc.SetPropValue "Info:", "windowless"" attribute has been found and removed"
27        Found = true
28        Exit For
29     ElseIf (LCase(objectparam(i).name) = "wmode" And LCase(objectparam(i).value) <> 
          "window") Then                'Flash/Flex
30        pc.SetPropValue "Info:", "wmode"" attribute has been changed from """& 
             objectparam(i).value &""" to ""window"""
31        objectparam(i).value = "window"
32        childElem.setAttribute "wmode", "window"
33        ChildElem.outerHTML = ChildElem.outerHTML
34        Found = true
35        Exit For
36     End If
37  Next
38 Next

The script in Listing 3 was tested with Internet Explorer versions 7, 8, and 10. At first glance, the script might seem to do more things than are really required; the reason for this is inconsistent behavior between different browser versions. Listing 4 and Listing 5 show this divergence. Both outputs were taken using the outerHTML object property (outerHTML seems to be supported by all Internet Explorer versions). Typically, such content should be given in clear HTML, but with some older browsers parameters, describing a Silverlight object have been removed. To be technically correct, the parameters are not really lost—they're Base64 encoded in the data element of the Silverlight object itself. To make things even more confusing, this conundrum only holds for Silverlight; no such problem exists for Flash or Flex multimedia content.

Let's try to decipher, step by step, what the script is all about. To keep things brief, obvious parts clearly related to the ISAM ESSO profiling API itself are omitted. In Listing 3:

Line 4Retrieves a collection of objects embedded into the website. With simpler Silverlight, Flash, and Flex websites, there should be just one object dedicated to this multimedia content.
Line 5Retrieves a collection of parameters (attributes) describing every excising object.
Line 14 Checks if existing windowless attribute is set to true. Listing 5 has an HTML code sample that shows this case.
Line 15 Disables windowless mode by changing the relevant attribute.
Lines 17 to 24 Construct the readable HTML code from the previously encoded Silverlight object parameters. This part is only executed in the case of some older browsers. (It was done this way because I'm not familiar with any other method to reload the object in the form of Base64 encoded blob. Listing 4 shows the structure.)
Lines 25 and 33Replace the old Silverlight or Flash object with the modified version. Essentially, this reloads the object back into the browser (a kind of swap operation).
Line 29Checks if the existing wmode attribute is set to any value other than window. (Listing 4 shows an HTML code sample where the wmode attribute has been set to opaque.)
Line 32Sets the wmode attribute to window if the above condition has been fulfilled. Basically, this procedure disables windowless mode completely.
Listing 4. Internet Explorer in version 7 and 8 - all parameters describing Silverlight Object are Base64 encoded
<object type="application/x-silverlight-2" width="100%" height="100%" 
Listing 5. Internet Explorer in version 10.0 – everything delivered in clear form
<object width="100%" height="100%" data="data:application/x-oleobject;
   DAAAAGYAYQBsAHMAZQAAAAAAAAAAAAAAAAAAAA==" type="application/x-silverlight-2">
            <param name="source" value="ClientBin/Silveos.Core.xap">
            <param name="onError" value="onSilverlightError">
            <param name="background" value="white">
            <param name="minRuntimeVersion" value="4.0.50401.0">
            <param name="uiculture" value="en-US">
            <param name="initParams" value="IpAddress=">
            <param name="enablehtmlaccess" value="true">
            <param name="enableNavigation" value="false">
            <param name="windowless" value="true">

Now for the results. Figure 6 shows the Silverlight content signature with windowless mode disabled. This particular signature has been taken right after the aforementioned VBScript was executed by the profile. When windowless mode is disabled, AccessStudio is able to construct the proper signature for Silverlight content.

Figure 6. Proper signature for Silverlight content
When windowless mode is disabled, AccessStudio is able to construct the proper signature for Silverlight content

UI Automation overview

Now that you've set up a website to be fully compatible with the UI Automation framework, it's time to explore what this solution really provides in terms of ISAM ESSO functions.

Figure 7 shows a collision of the two existing worlds. On one side is the good old Win32 native world presented by WinSpector; on the other side is UI Automation shown using Inspect. Both applications point out the same Silverlight multimedia-based content embedded into the website. As shown, all the existing UI elements such as buttons, edit fields, and most other controls are invisible to veteran tools such as WinSpector. Inspect, as the UI Automation client, allows you to identify the hierarchy of all the existing window controls compatible with the UI Automation framework. In the right pane, Inspect shows all the property information of the UI element that is currently selected.

Figure 7. WinSpector versus Inspect
‘WinSpector' vs ‘Inspect' – battle of two worlds

There are several available features with the UI Automation framework. The concept is nothing really new and the technology existed as far back as Windows 95 and Windows NT. Previously, the whole thing was called the Microsoft Active Accessibility (MSAA) framework. The initial goal of MSAA was to get information about UI elements and use it to support people with disabilities. For example, when a blind person can't see UI elements they need to use some kind of reader. The reader communicates with the application through the MSAA interface and sends the information back in a more suitable way. This technology has also been used to provide some functions for automated testing scenarios.

Fundamentally, the MSAA framework is divided into two groups: clients and servers. The MSAA client is any program that uses the accessibility interface to access, identify, or even manipulate the UI elements. The MSAA server is any application that exposes UI accessibility information.

The main goal of MSAA (and also of UI Automation) is to expose as much information as possible about every accessible Windows UI control presented at the desktop. Such information includes:

  • Control ID, name, type, and class
  • Location on the screen (coordinates of the rectangle object that encloses an element)
  • Visibility and status, either enabled or disabled
  • Element functionality, such as being clickable or selectable

UI Automation is the next-generation accessibility framework and is supported from Windows XP up to Windows 8. All systems below Windows 7 require a tiny Microsoft update (less than 1MB) to be installed on the machine (see Resources). Compared to MSAA, UI Automation has many new accessibility elements to improve the overall automation support for a wide range of UI interfaces. UI Automation is much faster (offers much better performance results), is more reliable than its predecessor, and is suited for a much wider application domain. Another nice feature is that UI Automation clients can leverage UI Automation services to interact with earlier MSAA implementations, providing full backward compatibility.

Though MSAA does have some support for Flash/Flex and other Adobe products, it cannot work with WPF or Silverlight-based applications. It might seem strange that Microsoft doesn't support itself but helps other vendors. Perhaps that's a contributing factor to why MSAA finally has a successor: the UI Automation framework.

As far as ISAM ESSO is concerned, support for MSAA was added to the profiling API a long time ago. Some profile developers might not know that most of the MSAA functions can be accessed using the IAccessible interface. Listing 6 shows an example where the script captures a host name entry from the window shown in Figure 8. The entry point into the MSAA world is the GetIAccessibleFromWindow function, which initializes the IAccessible COM interface from the given window handle.

Listing 6. VBScript used by ISAM ESSO profile to capture host name from the RDP session
Set wc = runtime.GetWindowController()
hwnd = wc.GetHWNDFromXPath("/child::wnd[@title=""Windows Security"" and 
Set acc = wc.GetIAccessibleFromWindow(hwnd)
Set child = acc.accNavigate(7, Clng(0))
For i = 1 to acc.accChildCount
   title  = child.accName(Clng(0))
   Set child = child.accNavigate(5, Clng(0))
   If InStr(title, "These credentials will be used to connect to ")<>0 Then
     HostName = Right (title, Len(title) - Len("These credentials will be used to 
        connect to "))
     Exit For
   End If
Figure 8. Windows Security window shown during RDP session
‘Windows Security dialog shown during RDP session

There are widely known deficiencies related to MSAA. It:

  • Doesn't seem to be very robust, mainly because of the old design
  • Doesn't support WPF and Silverlight-based applications

Thus, I decided to design a completely new ISAM ESSO plug-in based on the new UI Automation interface.

ISAM ESSO UIAutomation helper plug-in

The new ISAM ESSO plug-in based on the new UI Automation interface is in the form of a Windows dll library that's accessible from the ISAM ESSO profile VBScript/JScript code. There is a separate library dedicated for 32-bit and for 64-bit applications (obviously, in the case of a 64-bit operating system 32/64-bit libraries are not interchangeable). Figure 9 shows the schema of the UIAutomation helper plug-in and ISAM ESSO integration.

Figure 9. UIAutomation and ISAM ESSO integration
UIAutomation and ISAM ESSO integration

The UIAutomation helper plug-in was tested and is fully compatible with Windows XP (SP2) and with Windows 7 (32- and 64-bit). As mentioned, the operating systems before Windows 7 require installation of a Microsoft update before UI Automation can be fully exploited (see Resources). Though my test profiles were designed using ISAM ESSO version 8.2, I'm not familiar with any impediments that would prohibit the use of older products such as TAM ESSO 8.1.

The UIAutomation helper plug-in is designed to be used by scripting languages so, accordingly, it must be registered as a COM component. For that requirement, I prepared a simple registry file to set up proper registry entries into Windows Registry. Alternatively, there is also a way to do it with a profile itself. Read "Extend IBM Security Access Manager for ESSO AccessProfile with Windows native libraries" (Resources) for an example.


This article covered some of the SSO architecture concepts and discussed a few of the more relevant operating system components. You learned about embedding multimedia-based content using example websites. The article also explored MSAA and UI Automation concepts.

Stay tuned for Part 2 of this series, which will provide interesting examples mainly based on Flash and Silverlight multimedia-based content. The examples will walk you through making a few useful ISAM ESSO profiles that take advantage of the latest UIAutomation helper plug-in.



Get products and technologies

  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, or use a product in a cloud environment.


  • Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Security on developerWorks

ArticleTitle=UI automation in the single sign-on world, Part 1: Bridging the ISAM ESSO integration gap