ISAM ESSO under the hood

Troubleshooting IBM Security Access Manager for Enterprise Single Sign-On on 32-bit Windows systems

This article takes a deep look into IBM Security Access Manager for Enterprise Single Sign-On (ISAM ESSO) configuration troubleshooting. Explore how ISAM ESSO components interact with the client operating system, and learn how to configure single sign-on support for multidesktop client environments and for console applications running on Windows™ 2008.

Oktawian Powązka (Oktawian.Powazka@pl.ibm.com), Software Services Consultant, IBM

Photo of Oktawian PowązkaOktawian Powązka has worked as a services consultant and software engineer for a couple of internal TAM E-SSO IBM projects, mainly in Europe. He is currently working on the TAM E-SSO GIAMA project. Oktawian has four years of experience handling complex solutions in the TAM E-SSO arena.



26 February 2013

IBM Security Access Manager for Enterprise Single Sign-On (ISAM ESSO, formerly Tivoli Access Manager for Enterprise Single Sign-On) provides mature single sign-on (SSO) solutions for a wide range of Windows GUI, web, console, and Java-based applications. Standard configuration of the product is sufficient for most of those applications in most requirements scenarios. However, if you need to troubleshoot an ISAM ESSO deployment, it's helpful to understand how ISAM ESSO components running on client machines interact with the operating system and perform the SSO process. From time to time, in certain atypical environments, manual steps are required for implementing SSO support. This article, intended for IBM Business Partners and consultants who develop and deploy ISMA ESSO solutions, provides an overview of ISAM ESSO components and their interaction with the OS. Along the way, it explains the steps for ensuring ISAM ESSO client support for multidesktop systems and Windows command-line applications—introducing OS concepts such as the Windows messaging architecture, hooking, and the delay-load mechanism.

Introduction to ISAM ESSO components

ISAM ESSO is composed of server- and client-side elements. This article focuses on the client side, which is responsible for interaction with the user operating system and its applications. The key element of the ISAM ESSO client side is AccessAgent.

AccessAgent is installed on all Windows workstations and terminal servers that are designated to provide SSO. It automates the application sign-on process by monitoring and interacting with Windows GUI, web, console, and Java-based applications in accordance with a set of behavioral principles that are defined in another ISAM ESSO component called AccessProfile. The sign-on process uses the user logon credentials that are dedicated for each application, which are maintained in a Credential Wallet.

An important ISAM ESSO property is that AccessAgent emulates user login actions by using each application's native logon interface. As such, it works without any supplemental changes to the application's program structure. Figure 1 shows the ISAM ESSO schema.

Figure 1. IBM Security Access Manager for Enterprise Single Sign-On schema
Diagram of the ISAM ESSO schema. The Integrated Management System (IMS) Server exchanges credentials (wallet), policies, audit logs, and backups with AccessAgents. Through AccessAgents, users sign on automatically from enterprise applications (distributed, mainframe) with various authenticators (Windows, web, Citrix, terminal server).

AccessAgent components vary slightly depending on which OS the ISAM ESSO product is installed on. This article focuses on 32-bit versions of the Windows OS, such as Windows XP, Windows 2003, and Windows 2008.

After the user logs in to the OS, the following AccessAgent components are started:

  • DataProvider — Manages the AccessProfiles container and synchronizes with the Observer (Window, web, or Java) SSO module. AccessAgent uses the Observer module to perform the necessary logon, logoff, and other automated actions.
  • SOCIAccess — Manages the data on disk, such as the wallet, cryptographic secrets, and second-factor devices.
  • ObsService — Installs a hook (monitoring procedure) into the OS's message-handling mechanism.
  • AATray — Is the main process responsible for user login and registration.
    • Sync — Handles synchronization between AccessAgent and the IMS server.
    • LogonManager — Is responsible for user and desktop switching.
    • WalletMgr — Manages application credentials within the user wallet. (This process runs separately, on user request.)

Figure 2 shows AccessAgent components as viewed with the Process Explorer utility (see Resources).

Figure 2. AccessAgent components
Screen shot of Process Explorer list of running processes, including the AccessAgent components

The Observer module's SSO techniques vary depending on the application type. For typical Windows desktop applications, Observer uses the Windows hooking technique. For web applications accessed through a browser, it uses a Browser Helper Object to inspect and modify HTTP traffic. For Java applications, Observer calls the Java Accessibility APIs.

Next, you'll see how AccessAgent monitors and controls the startup sequence of a new application.


ObsService: Gateway to the OS

Window procedures

For GUI applications, the main point of contact between a process and the OS's message-handling mechanism is a window procedure (see Resources). But window procedures aren't a part of applications that are built solely as console-based applications. For this reason, ISAM ESSO has a separate mechanism for deliberately injecting a window control with the corresponding window procedure into every console-based application that the user launches. This mechanism works but has some issues. The Windows command line: The crack in the Windows message-driven architecture section looks closely into ISAM ESSO support for command-line applications.

Interaction between AccessAgent and the OS all starts with ObsService, whose sole purpose is to install the hook procedure into an OS hook chain. A hook is a pointer in the system message-handling mechanism that enables an application to install a function that monitors and evaluates all the internal messages that are passed between that process and the OS.

To monitor an application's messages, Windows requires that hook procedures be placed in a dynamic-link library (DLL). For this reason, upon startup ObsService calls the hook-installation procedure, passing one of the ISAM ESSO libraries — ObsBaseAgent.dll — as an argument value. In Figure 3, you can see the ObsBaseAgent library being loaded into the Windows calculator application during the window-initialization procedure.

Figure 3. ObsBaseAgent.dll being loaded during the window-initialization procedure
Screen shot from Process Explorer, showing the ObsBaseAgent library loading when the calculator application window initiates

ISAM ESSO in multidesktop environments

Hooking into the Windows message-handling mechanism lets a process monitor all GUI applications that run in the same desktop as the hook-procedure installer (ObsService in the case of ISAM ESSO). But what exactly does in the same desktop mean?

When a user logs in to the Windows OS, by default the UI is assigned to the currently visible desktop, also known as the input desktop. In most cases, to be more user-friendly, the input desktop encapsulates the Explorer process to provide a taskbar, start menu, icons, and the other UI elements Windows users are familiar with. The OS also creates two more desktops: ScreenSaver (used whenever a screen saver activates) and Winlogon (used when the user presses the CTRL+ALT+DEL key sequence). Moreover, some users use third-party software that leverages existing desktop boundaries to create a desktop that's similar to a Linux or OS X desktop.

The condition that Windows imposes is that window messages can be sent only between processes that have been created on the same desktop. Similarly, the hook procedure of a process running on a particular desktop can only receive messages that are intended for windows that are created in the same desktop. As a result, in a multidesktop environment, ObsService cannot install the hook procedure into any desktops in which it's not running.

For ISAM ESSO to be integrated with the multidesktop environment, the user must create a separate, stand-alone instance of the ObsService process on every additional desktop — because the single ObsService instance has no way to do any cross-desktop calls. Because the default ObsService runs as a Windows service, the only way to start the new instance is through the command line. The statement that needs to be executed on every new desktop is:

ObsService -standalone

Figure 4 shows the Windows calculator running on two separate desktops, each with separate instances of the Explorer and ObsService processes.

Figure 4. Calculator application running on two separate desktops
Screen shot from Process Explorer with a Calculator application running on two separate desktops

The example in Figure 4 takes advantage of the Desktops application from Microsoft. The first of the two separate user desktops is the default that is created when the user logged in to the system. The second one was created by the Desktops application. On the second desktop, I started another instance of the ObsService process manually through the command line, which enabled injection of the ObsBaseAgent library into the calculator process object space.

Next, I'll discuss ISAM ESSO support for Windows command-line applications — the case in which no actual window with an associated window procedure is created.


Windows command line: The crack in the Windows message-driven architecture

As you know from the preceding section, every window that a GUI program creates has an associated window procedure that binds the program with the Windows message-handling mechanism. This applies to the majority of Windows programs and facilitates interception through Windows hooks. But what if an application is built without an explicit window and message-handling procedure that is attached to it? Although console-based applications are contained within a window on the screen, that window contains only text characters; it's not a real GUI window. (It's possible to create console-based applications that display window pop-ups. Similarly, a GUI-based application can output character text strings to a console window — for debugging purposes, for example.)

As you can see in Figure 3, injection of the ObsBaseAgent library into the calculator process happens during window creation. But because command-line applications usually don't create a GUI window, ISAM ESSO needs a separate OS mechanism for library injection. Such a mechanism is based on the AppInit_DLLs registry entry and is wired into the core of the operating system.

AppInit_DLLs is the way to load an external library into every process address space upon application startup. The AppInit_DLLs registry entry is placed under the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows registry key. ISAM ESSO takes advantage of this registry key to inject one of its libraries, ConsoleHookLoader.dll, as shown in Figure 5.

Figure 5. The AppInit_DLLs registry entry
Screenshot from the Windows registry editor showing the AppInit_DLLs registry entry modified to inject ConsoleHookLoader.dll

A common misconception is that this registry change will not take effect until the next system restart, This is not true, at least not for Windows XP and above. Also, given that all of the DLLs listed under this registry key are loaded into every newly created process, it's easy to imagine that such a mechanism is a dream come true for all kinds of malicious software activities. In fact, even Microsoft recommends that no applications use or rely on this feature, which might be not available in future Windows versions.

The OS reads the AppInit_DLLs registry key when a process start-up routine loads the User32.dll Windows library into its memory, as shown in Figure 6. Although User32.dll contains many common system routines, not every application takes immediate advantage of this library. That's also the case with ISAM ESSO running on Windows 2008, which requires some additional tricks to make things run more smoothly.

Figure 6. Process Explorer showing ConsoleHookLoader.dll being loaded into the cmd.exe process via User32.dll
Screenshot from Process Explorer showing ConsoleHookLoader.dll being loaded into the cmd.exe process User32.dll

This ConsoleHookLoader.dll operation involves an undocumented registry entry: ConsoleHookExclusionList. This entry includes the list of all console-based applications that the ISAM ESSO Observer module should ignore, as shown in Figure 7.

Figure 7. List of all console-based applications that are excluded from ISAM ESSO activities
Screenshot from Process Explorer showing The list of all console based applications excluded from ISAM ESSO activities

An unused registry entry

The ConsoleHookExclusionList registry entry shouldn't be confused with the dllinjectionExclusionList entry. This entry is supposed to include the list of all applications (including some of the ISAM ESSO components themselves) to be excluded from library-injection mechanism. Unfortunately, it seems that this feature is unused, even though the registry entry does contain some predefined values that are filled in after default product installation. The dllinjectionExclusionList registry entry is placed under the HKEY_LOCAL_MACHINE\SOFTWARE\IBM\ISAM ESSO\ECSS\DeploymentOptions registry key.

During the initialization routine, ConsoleHookLoader.dll checks to see if the hosting process is included in that list; if it is, no other activities occur during start-up. This registry entry is placed under the HKEY_LOCAL_MACHINE\SOFTWARE\IBM\ISAM ESSO\ECSS\Temp.

The sole purpose of the ConsoleHookLoader DLL module is to create an extra empty window with the corresponding window procedure. That window is hidden, so the user cannot interact with it by using the mouse or keyboard. You might wonder how this empty window can imitate the real message-handling routine of the hosting application. In point of fact, it can't. This tiny window only serves as a kind of initiator for the ObsService process. All other ISAM ESSO procedures, such as console window monitoring and interception of user activities, are done directly using a dedicated console-programming interface.


Delay load, ISAM ESSO, and Windows 2008

This article's last topic is an issue that I recently encountered when working with ISAM ESSO. The issue involves the Windows delay-load mechanism. Typically, when an application starts, all the external DLLs that the application is linked to are loaded automatically. In contrast, delay-loaded DLLs are loaded only when and if they are first used. (The obvious benefits of this feature are that both the initialization time of the application start-up process and the application's memory footprint are reduced.) So, a delay-loaded library isn't used until the first call to one of its exported functions.

There's one exception to this rule. If a DLL has other DLLs implicitly linked to it, the application start-up process loads all those libraries immediately. This is exactly what happens in the case of the Windows command line in Windows operating systems prior to Windows 2008: The delay-load mechanism is ignored because the linking sequence for that process is different in those older versions. (Similar behavior might occur in future releases of the Windows OS, depending on how cmd will be constructed.)

Delay load is definitely a useful mechanism, but it can sometimes cause disruption in applications that rely on some basic tenets of software behavior. As you know from the preceding section, ISAM ESSO support for console applications is based on DLL injection through the AppInit_DLLs registry entry, which is interpreted during the User32.dll start-up routine. I've tested the Windows command-line profiles on various Windows versions and haven't noticed any problems with delay load on versions prior to Windows 2008. On Windows 2008, however, the Windows command line does take meaningful advantage of the delay-load feature by excluding User32.dll from the start-up routine, as you can see by its absence from Figure 8.

Figure 8. Complete set of the DLLs loaded by the command line in Windows 2008
Process Explorer screenshot showing all DLLs loaded by the command line in Windows 2008. The list does not include User32.dll.

Because User32.dll isn't loaded, ISAM ESSO on Windows 2008 is unable to support any profiles that are written directly for the cmd command prompt.

Solution (trick): Windows Program Compatibility mode

To address this issue, the initial goal is to override the delay-load mechanism.

By design, Windows Program Compatibility mode provides an environment for running programs that were built for earlier versions of Windows operating systems. The core functionality is based on an additional OS-shim helper layer that is implemented in two system DLLs: AcGenral.dll and AcLayers.dll. Upon application bootstrap, the OS loads the shim layer first and puts it under control. The key fact is that Program Compatibility mode requires User32.dll for some of its internal routines. So, every time this shim helper is put into action, the AppInit_DLLs registry entry is used accordingly, subsequently injecting ConsoleHookLoader into the application memory space.

There's one small problem with this solution. By default, Windows won't let you use the Program Compatibility mode wizard for its internal applications. And, of course, cmd is one such application. You can see in Figure 9 that Windows won't allow you to use Compatibility Mode for cmd through the wizard:

Figure 9. Program Compatibility mode unavailable for cmd
Screenshot of the Compatibility tab in the cmd properties screen. The dialog is grayed out.

To overcome the problem that is depicted in Figure 9, you need to modify the Windows registry directly. Program Compatibility mode settings for each application are stored under the following key:

[HKEY_LOCAL_MACHINE | HKEY_CURRENT_USER]\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers

Under the Layers key, the wizard creates a String value for each executable and then assigns the compatibility properties to the value of that string.

For example, to set the cmd executable to run in Windows XP SP2 compatibility mode, you would use the following values:

  • Name: C:\Windows\System32\cmd.exe
  • Data: WINXPSP2

Figure 10 shows the registry editor reflecting these settings.

Figure 10. Program Compatibility mode in action working as the result of a registry change
Screenshot of the registry editor reflecting that Program Compatibility mode is in effect for cmd.exe.

Conclusion

This article introduced you the client-side components of ISAM ESSO and explained how they interact with the Windows operating system. It also showed you the manual configuration steps that are required for setting up ISAM ESSO client support in certain atypical environments. With the deeper understanding, you've gained of ISAM ESSO client-support issues and solutions—including the role of some undocumented aspects of the product—you are better-equipped to tackle troubleshooting challenges. The solutions in this article are being used by some internal IBM projects.

Resources

Learn

Get products and technologies

Discuss

  • ISAM ESSO forum: This forum helps IBM Business Partners, practitioners, and customers develop AccessProfiles, deploy the ISAM ESSO solution, and learn about its capabilities.
  • Tivoli Access Manager for Enterprise Single Sign-on wiki: Get guidance, suggestions, and recommendations to help ensure a successful ISAM ESSO implementation.
  • Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Security on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • Security

    Pragmatic, intelligent, risk-based IT Security practices.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • IBM evaluation software

    Evaluate IBM software and solutions, and transform challenges into opportunities.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Security
ArticleID=858689
ArticleTitle=ISAM ESSO under the hood
publish-date=02262013