IBM Support

Fronting WebSphere - Hints and Tips for Edge Caching Proxy PAC-LDAP config and replacement of TAM Plugin.

Technical Blog Post


Abstract

Fronting WebSphere - Hints and Tips for Edge Caching Proxy PAC-LDAP config and replacement of TAM Plugin.

Body

 

Hello, and welcome to my first blog entry on the subject of "Fronting WebSphere" with bundled Proxy and Load Balancer products like  IBM HTTP Server (IHS webserver), WebSphere Plug-in proxy load balancer module for webservers, Edge Caching Proxy and Edge Load Balancers. The subject for this first blog entry is not the one I had planned but a recent support engagement resulted in some information I think is important to share about one of the more difficult configurations for Edge Caching Proxy.

 

Edge Caching Proxy supports using an LDAP for authorization in its support for PAC-LDAP where PAC stands for "Policy Authentication Control". The related details can be found in the Caching Proxy Administration Guide under Section: Configuring Caching Proxy security, sub section:  Using the PAC-LDAP authorization module. However, configuration of Edge Caching Proxy for LDAP authorization is very rare. Edge Caching Proxy is an old product that has gone through many development support teams over the years. Caching Proxy is a "stabilized" product and therefore has minimal development support and there are no plans for a 64 bit version. It is a known bottleneck under high loads of HTTPS which is becoming ever more prominent in usage so development assumes customers will be looking for alternate general purpose proxy server products. The point being that support teams for Caching Proxy have little hands on experience in PAC-LDAP configuration for Caching Proxy.

 

In Caching Proxy ibmproxy.conf there are a number of directives under the section for "API Directives" to support a "custom" LDAP module for Caching Proxy. However, Caching Proxy does install a module for PAC_LDAP named "libpacwte" and directives for it are found in ibmproxy.conf under the list of "Plug-ins" for Caching Proxy, for example:

 

# ===== LDAP Plug-in =====
ServerInit /opt/ibm/edge/cp/lib/plugins/pac/libpacwte.a.so:pacwte_auth_init /etc/paccp.conf


Authorization http://* /opt/ibm/edge/cp/lib/plugins/pac/libpacwte.a.so:pacwte_auth_policy


ServerTerm /opt/ibm/edge/cp/lib/plugins/pac/libpacwte.a.so:pacwte_shutdown
# ======= End ===========

 

A common LDAP, like IBM Tivoli Directory Server (ITDS), provides a custom LDAP module referred to as the Tivoli Access Manager plug-in or TAM Plug-in. In ibmproxy.conf the configuration would look something like this:

#### TAM PLUGIN ####
ServerInit /opt/pdweb-lite/lib/wesauth.so:WTESeal_Init /opt/pdweb-lite/etc/ibmwesas.conf

PreExit /opt/pdweb-lite/lib/wesauth.so:WTESeal_PreExit

Authorization * /opt/pdweb-lite/lib/wesauth.so:WTESeal_Authorize

ServerTerm   /opt/pdweb-lite/lib/wesauth.so:WTESeal_Term

Transmogrifier  /opt/pdweb-lite/lib/wesauth.so:WTESeal_TransOpen:WTESeal_TransWrite:WTESeal_TransClose:WTESeal_TransError
####################

 

Usage of TAM plugin in Caching Proxy is equally as rare with minimal development support in Tivoli support. It is no surprise to Caching Proxy support, with customers scrambling to migrate to Caching Proxy V8.x to get support for TLS 1.2, to hear that TAM plugin does not work in V8.x Caching Proxy or that support has been withdrawn for TAM plugin. In 2016 Caching Proxy support helped an IBM team migrate two Edge Caching Proxy 8.5.5.x on AIX from using TAM Plugin to using PAC-LDAP plugin to authorize against an ITDS LDAP server. It still requires the ITDS LDAP client be installed on the Caching Proxy OS instance. For other LDAPs OpenLDAP client may be required.  Both provide an libldap file that the Caching Proxy libpacwte module is dependent on. During the migration we discovered some interesting aspects that impeded a quick migration. This blog entry details what we learned that should help avoid some of the problems we encountered and to supplement what is in Caching Proxy Administration Guide under Configuring Caching Proxy security > Using the PAC-LDAP authorization module.

 

RESOLVING:

The first thing to note is that the default ibmproxy.conf for UNIX platforms had LDAP Plug-in directives (as noted above) that want to load module file libpacwte.a.so . However Caching Proxy installed two files:
libpacwte.a
libpacwte.so
The administration guide article suggests libpacwte.so but on the AIX we went with libpacwte.a. The current Caching Proxy Level 3 support reported these appear to be the same file. Be sure to edit the directives to load either libpacwte.so or libpacwte.a because libpacwte.a.so is a file not found. Why the default in ibmproxy.conf is incorrect remains a mystery. Caching Proxy V9 is already released so we would not be able to change the default until V10. Since it is a stabilized product, it may remain.

 

Once you decide on which module file to load then run an LDD against it to check for dependent modules:

ldd libpacwte.a

In the case that resulted in this blog entry it could not find libldap.a:

  # ldd libpacwte.a
  libpacwte.a needs:
  /usr/lib/libpthread.a(shr_xpg5.o)
 Cannot find libldap.a
  /usr/lib/threads/libc.a(shr.o)
  /usr/lib/httpdapi.o
  /unix
  /usr/lib/libpthreads.a(shr_comm.o)
  /usr/lib/libcrypt.a(shr.o)
  /usr/lib/libicuuc18.a
  /usr/lib/libicui18n.a
  /usr/lib/libodm.a(shr.o)
  /usr/lib/libC.a(shr.o)
  /usr/lib/libpthreads_compat.a(shr.o)
  /usr/lib/libC.a(shr2.o)
  /usr/lib/libC.a(ansi_32.o)
  /usr/lib/libC.a(shrcore.o)
  /usr/lib/libpthreads.a(shr.o)
  /usr/lib/libC.a(ansicore_32.o)

 

It turned out this was present in the ITDS client directories:    /opt/IBM/ldap/V6.1/lib/libldap.a

To resolve we had to add to libpath in AIX /etc/environment file:   LIBPATH=$LIBPATH:/opt/IBM/ldap/V6.1/lib

 

This resulted in a correct output of LDD:

root@proxyservercp11:/opt/ibm/edge/cp/lib/plugins/pac
# ldd libpacwte.a
libpacwte.a needs:
/usr/lib/libpthread.a(shr_xpg5.o)
/opt/IBM/ldap/V6.1/lib/libldap.a
/usr/lib/threads/libc.a(shr.o)
/usr/lib/httpdapi.o
/unix
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/opt/IBM/ldap/V6.1/lib/libibmldapdbg.a
/opt/IBM/ldap/V6.1/lib/libidsldapiconv.a
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libicuuc18.a
/usr/lib/libicui18n.a
/usr/lib/libodm.a(shr.o)
/usr/lib/libC.a(shr.o)
/usr/lib/libpthreads_compat.a(shr.o)
/usr/lib/libc_r.a(shr.o)
/usr/lib/libC.a(shr2.o)
/usr/lib/libC.a(ansi_32.o)
/usr/lib/libC.a(shrcore.o)
/usr/lib/libpthreads.a(shr.o)
/usr/lib/libC.a(ansicore_32.o)

 

Besides libpath problems there could be permission issues or other. The point being you may have to work with your OS admin or OS support to resolve the dependencies. The end goal in configuring the LDAP Plug-in directives for the specific libpacwte file and, resolving dependencies, so that when Caching Proxy starts there should be no message in the Caching Proxy Error log about not being able to load DLL module libpacwte. For example:

Caching Proxy Error log:    Error 2 loading DLL module /opt/ibm/edge/cp/lib/plugins/pac/libpacwte.a.so.     A file or directory in the path name does not exist.

There is no need to proceed with configuration of the related PAC configuration files until the module can be loaded by Caching Proxy. We also re-discovered that for the LDAP Plug-in directive, Authorization http://protectarea/opt/ibm/edge/cp/lib/plugins/pac/libpacwte.a.so:pacwt…, that "http://protectarea" is a matching variable that has to be set by the administrator setting up for PAC-LDAP. The Caching Proxy Administration Guide article:   "Using the PAC-LDAP authorization modules" suggests using: http://*. Caching Proxy Trace shows the match is for http:// even for HTTPS requests so the value covers both. I suspect this could even be more specific if needed.

 

Likewise the "Protection" configuration in ibmproxy.conf:

##########################
Protection PROT-LDAP {
           ServerId     Private_LDAP_Authorization
           AuthType     Basic
           Mask         All@(*)
 }

Protect /HOD/* Prot-LDAP
#############################

 

For the "Protect" directive Caching Proxy Trace shows its trying to do a URI match.

For the example:  Protect /aaa/* Prot-LDAP


I suspect the /aaa/* was a generic but, we did not catch that so Trace showed:

[00000102/1466627593]: Protect..... /aaa/* and did NOT match /HOD/test.html.

 

The directive was adjusted to allow the match:

Protect /HOD/* Prot-LDAP

 

The remaining configuration concerns were with the PAC conf files:

pac.conf
paccp.conf
pacpolicy.conf

 

There was also a password stash file:

pac_ldap.cred

 

In the PAC conf files:

[PAC_MAN_SERVER]
Refers to the Edge Caching Proxy - mainly hostname and a free unused port

[LDAP_SERVER]
Refers to the LDAP server

 

PACCP.conf:

The paccp.conf is the simplest and a good one to start with. Use Caching Proxy Hostname and a free port that Caching Proxy will use to send LDAP auth request to LDAP port defined.

[PAC_MAN_SERVER]
hostname: <CP hostname>
port: <A free port that Caching Proxy will use to send LDAP auth request to LDAP port defined>.

In this case we opted for port 2222.  Netstat can be used to check for ports in use. These same values will be put in pac.conf. The remaining configuration in paccp.conf was left at default:

[PACWTE_PLUGIN]
hostname_check:false

 

PAC.conf:

Pac.conf is where we start to add some information about the LDAP server. If the administrator doing the pac.ldap conf is not sure of the values they should consult their LDAP admininistrator or LDAP support. In this case the LDAP administrator gave us a good set of values to start with. We setup to test with a non-secure LDAP call to LDAP port 389. We brought over the values for paccp conf to define [PAC_MAN_SERVER]

[PAC_MAN_SERVER]
hostname: <CP Hostname>
port:2222
#conn_type:ssl
authentication_sequence:primary
authorization_sequence:secondary

It was for non-SSL/TLS configuration so we left the conn_type:ssl commented out. The authentication_sequence and authorization_sequence were left at default:
authentication_sequence:primary
authorization_sequence:secondary

 

For [LDAP_SERVER], based on details from LDAP administrator we started with:

[LDAP_SERVER]
hostname: <LDAP hostname>
port:389
#ssl_port:636
admin_dn:cn=root
search_base:o=ibm,c=us
search_key:uid

The password for admin_dn:cn=root was added to pac_ldap.cred which becomes encrypted when Caching Proxy gets restarted to test. Based on troubleshooting, a failure with the LDAP auth, we needed to use the LDAPSearch Tool, run from the Caching Proxy, to verify what configuration is needed for the pac.conf. The LDAP admin found the LDAP search tool helped to figure out PAC.conf was incorrect for:
from search_base:o=ibm,c=us
to search_base:ou=iespc,o=ibm,c=us

 

So the final working config in PAC.conf for [LDAP_SERVER]:

[LDAP_SERVER]
hostname: <LDAP hostname>
port:389
#ssl_port:636
admin_dn:cn=root
search_base:ou=iespc,o=ibm,c=us
search_key:uid

 

PACPOLICY.conf:

During a test for PAC-LDAP it was suggested to use a policy of:

[POLICY01]
 type:0
 id:ou=iespc,o=ibm
 grant:true
 propagate:true

We first just added this policy to the default config in pacpolicy.conf (which had a lot of sample policies) but had problems. After further investigation we simply removed all the default policy config and left only the one policy above.

 

PAC_LDAP.cred:

The password for admin_dn:cn=root in PAC.conf was added to pac_ldap.cred which becomes encrypted when Caching Proxy gets restarted to test.

 

TROUBLESHOOTING:

Besides the Caching Proxy Error and Trace logs for errors specific to the PAC-LDAP there are two logs files that can be present:
1.  PacErr_Client.log
2.  PacErr_Server.log
Include these in any Mustgather for PAC-LDAP problems if present.

 

Two environment variables to load noted below before starting Caching Proxy will get more debug detail related to PAC-LDAP:
1. PAC_DEBUG_LEVEL=64
2. LDAP_DEBUG=65535

 

Some Caching Proxy are started with a start script that is already pre-loading some ENV variables before loading ibmproxy. They could also be added on the command line before ibmproxy start.

 

A binary, unlimited length packet trace capture (TCPDUMP) at the Caching Proxy of the port 389/636 traffic leaving the Caching Proxy system (Destination port) is also useful.

------------------

Hopefully these notes will reduce the time to get a config for PAC-LDAP working. The primary hint/tips are:

- Resolve the module loading and dependencies first
- Work with LDAP admin to run the LDAPsearch tool to ensure a correct set of values for the PAC conf files.

 

 

[{"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":""}]

UID

ibm11080357