ISAM ESSO under the hood
Troubleshooting IBM Security Access Manager for Enterprise Single Sign-On on 32-bit Windows systems
IBM Security Access Manager for Enterprise Single Sign-On (ISAM ESSO, formerly Tivoli Access Manager for Enterprise Single Sign-On) provides mature single sign-on (SSO) solutions for a wide range of Windows GUI, web, console, and Java-based applications. Standard configuration of the product is sufficient for most of those applications in most requirements scenarios. However, if you need to troubleshoot an ISAM ESSO deployment, it's helpful to understand how ISAM ESSO components running on client machines interact with the operating system and perform the SSO process. From time to time, in certain atypical environments, manual steps are required for implementing SSO support. This article, intended for IBM Business Partners and consultants who develop and deploy ISMA ESSO solutions, provides an overview of ISAM ESSO components and their interaction with the OS. Along the way, it explains the steps for ensuring ISAM ESSO client support for multidesktop systems and Windows command-line applications—introducing OS concepts such as the Windows messaging architecture, hooking, and the delay-load mechanism.
Introduction to ISAM ESSO components
ISAM ESSO is composed of server- and client-side elements. This article focuses on the client side, which is responsible for interaction with the user operating system and its applications. The key element of the ISAM ESSO client side is AccessAgent.
AccessAgent is installed on all Windows workstations and terminal servers that are designated to provide SSO. It automates the application sign-on process by monitoring and interacting with Windows GUI, web, console, and Java-based applications in accordance with a set of behavioral principles that are defined in another ISAM ESSO component called AccessProfile. The sign-on process uses the user logon credentials that are dedicated for each application, which are maintained in a Credential Wallet.
An important ISAM ESSO property is that AccessAgent emulates user login actions by using each application's native logon interface. As such, it works without any supplemental changes to the application's program structure. Figure 1 shows the ISAM ESSO schema.
Figure 1. IBM Security Access Manager for Enterprise Single Sign-On schema
AccessAgent components vary slightly depending on which OS the ISAM ESSO product is installed on. This article focuses on 32-bit versions of the Windows OS, such as Windows XP, Windows 2003, and Windows 2008.
After the user logs in to the OS, the following AccessAgent components are started:
DataProvider— Manages the AccessProfiles container and synchronizes with the Observer (Window, web, or Java) SSO module. AccessAgent uses the Observer module to perform the necessary logon, logoff, and other automated actions.
SOCIAccess— Manages the data on disk, such as the wallet, cryptographic secrets, and second-factor devices.
ObsService— Installs a hook (monitoring procedure) into the OS's message-handling mechanism.
AATray— Is the main process responsible for user login and registration.
Sync— Handles synchronization between AccessAgent and the IMS server.
LogonManager— Is responsible for user and desktop switching.
WalletMgr— Manages application credentials within the user wallet. (This process runs separately, on user request.)
Figure 2 shows AccessAgent components as viewed with the Process Explorer utility (see Related topics).
Figure 2. AccessAgent components
The Observer module's SSO techniques vary depending on the application type. For typical Windows desktop applications, Observer uses the Windows hooking technique. For web applications accessed through a browser, it uses a Browser Helper Object to inspect and modify HTTP traffic. For Java applications, Observer calls the Java Accessibility APIs.
Next, you'll see how AccessAgent monitors and controls the startup sequence of a new application.
ObsService: Gateway to the OS
Interaction between AccessAgent and the OS all starts with
ObsService, whose sole purpose is to install the hook procedure into an OS hook chain. A hook is a pointer in the system message-handling mechanism that enables an application to install a function that monitors and evaluates all the internal messages that are passed between that process and the OS.
To monitor an application's messages, Windows requires that hook procedures be placed in a dynamic-link library (DLL). For this reason, upon startup
ObsService calls the hook-installation procedure, passing one of the ISAM ESSO libraries —
ObsBaseAgent.dll — as an argument value.
In Figure 3, you can see the
ObsBaseAgent library being loaded into the Windows calculator application during the window-initialization procedure.
ObsBaseAgent.dll being loaded during the window-initialization procedure
ISAM ESSO in multidesktop environments
Hooking into the Windows message-handling mechanism lets a process monitor all GUI applications that run in the same desktop as the hook-procedure installer (
ObsService in the case of ISAM ESSO). But what exactly does in the same desktop mean?
When a user logs in to the Windows OS, by default the UI is assigned to the currently visible desktop, also known as the input desktop. In most cases, to be more user-friendly, the input desktop encapsulates the
Explorer process to provide a taskbar, start menu, icons, and the other UI elements Windows users are familiar with. The OS also creates two more desktops:
ScreenSaver (used whenever a screen saver activates) and
Winlogon (used when the user presses the CTRL+ALT+DEL key sequence). Moreover, some users use third-party software that leverages existing desktop boundaries to create a desktop that's similar to a Linux or OS X desktop.
The condition that Windows imposes is that window messages can be sent only between processes that have been created on the same desktop. Similarly, the hook procedure of a process running on a particular desktop can only receive messages that are intended for windows that are created in the same desktop.
As a result, in a multidesktop environment,
ObsService cannot install the hook procedure into any desktops in which it's not running.
For ISAM ESSO to be integrated with the multidesktop environment, the user must create a separate, stand-alone instance of the
ObsService process on every additional desktop — because the single
ObsService instance has no way to do any cross-desktop calls. Because the default
ObsService runs as a Windows service, the only way to start the new instance is through the command line. The statement that needs to be executed on every new desktop is:
Figure 4 shows the Windows calculator running on two separate desktops, each with separate instances of the
Figure 4. Calculator application running on two separate desktops
The example in Figure 4 takes advantage of the Desktops application from Microsoft. The first of the two separate user desktops is the default that is created when the user logged in to the system. The second one was created by the Desktops application. On the second desktop, I started another instance of the
ObsService process manually through the command line, which enabled injection of the
ObsBaseAgent library into the calculator process object space.
Next, I'll discuss ISAM ESSO support for Windows command-line applications — the case in which no actual window with an associated window procedure is created.
Windows command line: The crack in the Windows message-driven architecture
As you know from the preceding section, every window that a GUI program creates has an associated window procedure that binds the program with the Windows message-handling mechanism. This applies to the majority of Windows programs and facilitates interception through Windows hooks. But what if an application is built without an explicit window and message-handling procedure that is attached to it? Although console-based applications are contained within a window on the screen, that window contains only text characters; it's not a real GUI window. (It's possible to create console-based applications that display window pop-ups. Similarly, a GUI-based application can output character text strings to a console window — for debugging purposes, for example.)
As you can see in Figure 3, injection of the
ObsBaseAgent library into the calculator process happens during window creation. But because command-line applications usually don't create a GUI window, ISAM ESSO needs a separate OS mechanism for library injection. Such a mechanism is based on the
AppInit_DLLs registry entry and is wired into the core of the operating system.
AppInit_DLLs is the way to load an external library into every process address space upon application startup. The
AppInit_DLLs registry entry is placed under the
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows registry key. ISAM ESSO takes advantage of this registry key to inject one of its libraries,
ConsoleHookLoader.dll, as shown in Figure 5.
Figure 5. The
AppInit_DLLs registry entry
A common misconception is that this registry change will not take effect until the next system restart, This is not true, at least not for Windows XP and above. Also, given that all of the DLLs listed under this registry key are loaded into every newly created process, it's easy to imagine that such a mechanism is a dream come true for all kinds of malicious software activities. In fact, even Microsoft recommends that no applications use or rely on this feature, which might be not available in future Windows versions.
The OS reads the
AppInit_DLLs registry key when a process start-up routine loads the
User32.dll Windows library into its memory, as shown in Figure 6. Although
User32.dll contains many common system routines, not every application takes immediate advantage of this library. That's also the case with ISAM ESSO running on Windows 2008, which requires some additional tricks to make things run more smoothly.
Figure 6. Process Explorer showing
ConsoleHookLoader.dll being loaded into the
cmd.exe process via
ConsoleHookLoader.dll operation involves an undocumented registry entry:
ConsoleHookExclusionList. This entry includes the list of all console-based applications that the ISAM ESSO Observer module should ignore, as shown in Figure 7.
Figure 7. List of all console-based applications that are excluded from ISAM ESSO activities
During the initialization routine,
ConsoleHookLoader.dll checks to see if the hosting process is included in that list; if it is, no other activities occur during start-up. This registry entry is placed under the
The sole purpose of the
ConsoleHookLoader DLL module is to create an extra empty window with the corresponding window procedure. That window is hidden, so the user cannot interact with it by using the mouse or keyboard. You might wonder how this empty window can imitate the real message-handling routine of the hosting application. In point of fact, it can't. This tiny window only serves as a kind of initiator for the
ObsService process. All other ISAM ESSO procedures, such as console window monitoring and interception of user activities, are done directly using a dedicated console-programming interface.
Delay load, ISAM ESSO, and Windows 2008
This article's last topic is an issue that I recently encountered when working with ISAM ESSO. The issue involves the Windows delay-load mechanism. Typically, when an application starts, all the external DLLs that the application is linked to are loaded automatically. In contrast, delay-loaded DLLs are loaded only when and if they are first used. (The obvious benefits of this feature are that both the initialization time of the application start-up process and the application's memory footprint are reduced.) So, a delay-loaded library isn't used until the first call to one of its exported functions.
There's one exception to this rule. If a DLL has other DLLs implicitly linked to it, the application start-up process loads all those libraries immediately. This is exactly what happens in the case of the Windows command line in Windows operating systems prior to Windows 2008: The delay-load mechanism is ignored because the linking sequence for that process is different in those older versions. (Similar behavior might occur in future releases of the Windows OS, depending on how
cmd will be constructed.)
Delay load is definitely a useful mechanism, but it can sometimes cause disruption in applications that rely on some basic tenets of software behavior. As you know from the preceding section, ISAM ESSO support for console applications is based on DLL injection through the
AppInit_DLLs registry entry, which is interpreted during the
User32.dll start-up routine. I've tested the Windows command-line profiles on various Windows versions and haven't noticed any problems with delay load on versions prior to Windows 2008. On Windows 2008, however, the Windows command line does take meaningful advantage of the delay-load feature by excluding
User32.dll from the start-up routine, as you can see by its absence from Figure 8.
Figure 8. Complete set of the DLLs loaded by the command line in Windows 2008
User32.dll isn't loaded, ISAM ESSO on Windows 2008 is unable to support any profiles that are written directly for the
cmd command prompt.
Solution (trick): Windows Program Compatibility mode
To address this issue, the initial goal is to override the delay-load mechanism.
By design, Windows Program Compatibility mode provides an environment for running programs that were built for earlier versions of Windows operating systems. The core functionality is based on an additional OS-shim helper layer that is implemented in two system DLLs:
AcLayers.dll. Upon application bootstrap, the OS loads the shim layer first and puts it under control. The key fact is that Program Compatibility mode requires
User32.dll for some of its internal routines. So, every time this shim helper is put into action, the
AppInit_DLLs registry entry is used accordingly, subsequently injecting
ConsoleHookLoader into the application memory space.
There's one small problem with this solution. By default, Windows won't let you use the Program Compatibility mode wizard for its internal applications. And, of course,
cmd is one such application. You can see in Figure 9 that Windows won't allow you to use Compatibility Mode for
cmd through the wizard:
Figure 9. Program Compatibility mode unavailable for
To overcome the problem that is depicted in Figure 9, you need to modify the Windows registry directly. Program Compatibility mode settings for each application are stored under the following key:
[HKEY_LOCAL_MACHINE | HKEY_CURRENT_USER]\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers
Layers key, the wizard creates a
String value for each executable and then assigns the compatibility properties to the value of that string.
For example, to set the
cmd executable to run in Windows XP SP2 compatibility mode, you would use the following values:
Figure 10 shows the registry editor reflecting these settings.
Figure 10. Program Compatibility mode in action working as the result of a registry change
This article introduced you the client-side components of ISAM ESSO and explained how they interact with the Windows operating system. It also showed you the manual configuration steps that are required for setting up ISAM ESSO client support in certain atypical environments. With the deeper understanding, you've gained of ISAM ESSO client-support issues and solutions—including the role of some undocumented aspects of the product—you are better-equipped to tackle troubleshooting challenges. The solutions in this article are being used by some internal IBM projects.
- IBM Security Access Manager for Enterprise Single Sign-On: Learn more about ISAM ESSO.
- "Three Ways to Inject Your Code into Another Process": Learn how to inject code into another Windows process's address space, and then execute it in the context of that process.
- About Window Procedures: This Microsoft Developer Network page discusses the Windows message-handling procedure.
- IBM Security Access Manager for Enterprise Single Sign-On: Download a free ISAM ESSO trial version.
- Process Explorer: Process Explorer is a system monitoring and examination utility for Windows.
- Desktops: Learn more about the Microsoft Desktops application.