Troubleshooting the PE to CE Security Configuration

Resolving The Problem

Troubleshooting the PE to CE Security Configuration

One of the major changes in the Process Engine 4.0.x and Process Engine 4.5 releases is that the PE no longer communicates directly with the LDAP server, as the old 3.5 PE did. In PE 4.0 and newer releases, all of the cases where the PE used to call the LDAP APIs directly to get user and group information have been changed so that the PE now uses the CE APIs to get that information.

After a PE 4.x has been installed, you need to configure its security information on the Security tab of the Process Task Manager. There, you’ll enter a CE URI, a service user and password, and the names of the Configuration and Administration groups.

After entering the configuration information, you click the Apply button. At that point, the PE will attempt to communicate to the CE. If everything has been done correctly, the PE will be able to communicate and you’ll be asked to click OK to restart the PE.

But, because the installation and configuration of the CE and PE are complicated task, there are many opportunities to have made a mistake. And many of the configuration mistakes will not show up until you try to define the PE’s security connection to the CE.

What follows is a process for troubleshooting these problems.

Throughout these troubleshooting tips, we reference one of the default ports used by the PE: 32776. If your PE Server is configured to use another port, substitute that port number throughout all these tips.

The part of the PE that communicates with the CE is the PEDirectoryServer java program. That program is automatically initiated by the PE. This program is known informally as the “little pipe.” When the PE needs to talk to the CE, the request is sent through the "little pipe" and this java code calls the CE, gets the required information, and returns the information to the PE server code that needed that information.

Troubleshooting – New for 4.0.3 and 4.5 – Automatic Trace

There is now an automatic "little pipe" trace created in a file called PEDirectoryServerConnectionDebug.txt.
This trace file is created in the \FNSW_LOC\logs\TM_daemon\ folder on the PE Server.

This trace information is created unconditionally so that if an initial setup of PE has a problem connecting to the CE Server, the trace information will be present without having to go back and enable the "little pipe" traces.

This trace is automatically enabled for the first 10 calls on the PEDirectoryServer. Then this automatic trace is automatically disabled.

This trace is completely independent of the normal log4j traces. The log4j tracing can be enabled and disabled just as before.

If your PE has trouble connecting to the CE, this is probably the first place to check. Look at that trace file and that may give you the info you need to diagnose the problem.

Can the PE actually see the CE?

Try pinging the CE box from the PE box using the name of the CE box. This should verify that DNS can translate the CE’s hostname into an IP address.

In a web browser on the PE Server, go to the CE’s app server. In the following examples, you will need to use the hostname of your CE server in place of your-ce-server. And if you’re using a port other than the default, you will need to use that port number.

If your CE is deployed in a WebSphere application server, set the URL to http://your-ce-server:9090/admin

And if your CE is deployed in a WebLogic application server, set the URL to http://your-ce-server:7001/console

And if your CE is deployed in a JBoss application server, set the URL to http://your-ce-server:9080/web-console/

If a web browser running on the PE server can’t connect to the CE host, you’ll need to figure out why. You may have a firewall issue. Or a DNS issue. The PE must be able to talk to the CE’s app server using http.


Is the CE actually alive and accessible from the PE box?

In a web browser on the PE Server enter the URL:

CE in WebSphere: http://your-ce-server:9090/FileNet/Engine
CE in WebLogic: http://your-ce-server:7090/FileNet/Engine
CE in JBoss: http://your-ce-server:9080/FileNet/Engine

You will need to adjust the URL to specifically reference your CE server’s name and port.

You should get a nice response page from CE that includes CE Version information.
If you get any sort of error, you need resolve that issue.

The PE Server must be able to talk HTTP to the CE Server. If you can’t get this page of information, you need to resolve that issue first. You may have the wrong URL. DNS may not know the name of your CE server. You may have the wrong port. Your CE may not be properly functioning.

Whatever the actual problem is, it must be resolved.


Check the PE log files for any messages that explain what has happened

Check the Windows event viewer, and/or look in the \fnsw\ logs (syslog, vwtrace file in fnsw_loc\sd or /fnsw/local/sd). If you’re using PE 4.0.3 or newer, check the \FNSW_LOC\logs\TM_daemon\ PEDirectoryServerConnectionDebug.txt file. Something may have been logged that explains the problem.


Do an ‘initfnsw status’ to verify that the PE is started

This just verifies that the PE (IS) software as been initiated properly. If the response doesn’t include "Software started since…" you need to do an initfnsw –y restart.


Verify that the Process Engine Services Manager is started

On a Windows-based PE Server, use the Services applet to verify that the Process Engine Services Manager is started. If it isn’t started, get it started.


Is the vwior process alive and functioning?

This process is required so that the PEDirectoryServer process can communicate with the CE. To see if the process is running, look at the list of running tasks to see if there is one called vwior.

To verify that the vwior process is up and functioning correctly, bring up a web browser on the PE Server and try this URL (if you’re running on a port other than 32776, adjust the port as necessary):
http://localhost:32776/IOR/ping

A "good" sample response from the vwior looks something like this:

PE Server
HQ-PE400 [Win32 Server Production (MS-SQL Blob 512 MB) -- {-rx-ux-nx} Jan 18 2007 11:03:24 en ]

If the vwior process isn’t running, it may be because some other program is using a socket or port that vwior requires. This information would appear in the logs.

If vwior isn’t running you might try to initiate it "by hand:"
vwior –p 32776 –b 32777

The vwior process needs to have access to the required sockets. If it can’t open the right socket, it will display and log a message stating this problem.

You then need to figure out why the vwior can’t use the required socket. You may need to restart your PE Server to cause other applications to release the required socket.

After you get the vwior process successfully initiated, try to apply the PE security settings again.


Is the PEDirectoryServer running?

The PEDirectoryServer is a java application that provides the connection from the PE to the CE. Because there will likely be several java applications running on the PE Server, you’ll need to look at the running tasks on the server with a tool that shows you the initiation strings. For example, Process Explorer from SysInternals/Microsoft nicely displays the whole java command line that initiated running programs.

You’re looking for an instance of java running that is running the filenet.pe.ceorb.server.PEDirectoryServer class. If the PEDirectoryServer java application isn’t running, you’ll need to figure out why.

To verify that the PEDirectoryServer process is up and functioning correctly, on the PE Server, bring up a web browser and try this URL (if you’re running on a port other than 32776, adjust the port as necessary):
http://localhost:32776/IOR/FileNet.CE.PEDirectoryServer

A "good" sample response from the PEDirectoryServer looks something like this:

IOR:000000000000002049444c3a46696c654e65745f63656d707270632f63656d707270633a31
2e300000000001000000000000007e000102000000000c31302e31352e372e31303800071d000
000000031afabcb000000002084240cba00000001000000000000000100000008526f6f74504f41
0000000008000000010000000014000000000000020000000100000018000000000501000100
00000000010109000000010001010000000026000000020002

You can look in the PE logs. You should see at least one log message that gives the full java command used to start the PE Directory Server program. For example:

VW: PEDirectoryServer start command: C:\fnsw\jre\bin\java –Xmx512M -Xrs -cp C:\fnsw\bin\pe.jar;C:\fnsw\CE_API\lib\Jace.jar;C:\fnsw\CE_API\lib\log4j.jar;C:\fnsw\CE_API\wsi\lib\wasp.jar -Djava.security.auth.login.config=C:\fnsw\CE_API\config\jaas.conf.WSI -Dwasp.location=C:\fnsw\CE_API\wsi filenet.pe.ceorb.server.PEDirectoryServer /port=32776

If you see multiple instances of the above log record, it means that the PE may be having trouble getting the PEDirectoryServer to start properly and that might be a problem.

If the PE Directory Server isn’t running, try running the program “by hand” in a command prompt window (log on as fnsw to do this test). Use the initiation string that was logged on your server. Here you may see an exception from java that might explain why the PEDirectoryServer can’t be initialized. For example, a NoClassDefFoundError exception might imply that a jar file isn't where it should be.

Are all the required jars present in the right location? Is jaas.conf.WSI file present in the \fnsw\bin\ directory? Is the Jace.jar file present in the PE Server’s \fnsw\CE_API\lib\ directory? Remember, as of PE 4.5, you must use the "CE Client Installer" to put the CE client jars on the PE Server. If you don’t do this, the PE will not be able to talk to the CE.

Is the \fnsw\CE_API\lib\Jace.jar the same version as the Jace.jar file that is deployed on the CE Server? If not, use the CE’s "client installer" to update the PE’s "CE Client" software. The PE must have the correct CE Client software to successfully connect to the CE.

After you get the PEDirectoryServer process successfully initiated, try to apply the PE security settings again.


Some other quick things to check

Verify that your basic PE Security configuration information is good. Is the Service Username and Service Password (as specified on the Security tab of the Process Task Manager) correct? Is your specified "Security Username" a member of the group specified as the Administrator Group (on that same Security tab)? You can read through the PEDirectoryServerConnectionDebug.txt file to see if the PE is able to find information about the specified user and group.

In the CE’s application server, make sure that the various authentication providers are configured to be "sufficient" as necessary. Make sure that the application’s "inbuilt" authentication provider isn’t configured as "required." Any authentication provider that is configured as "required" means that all users who will authenticate must be defined in that provider.

If you’re having trouble with the PE to CE communications, there may be something wrong with the java environment. Try the following tests in a command prompt window on the PE Server. You will need to substitute appropriate server names, ports, user names, group names, and passwords as appropriate for your configuration.

On your PE Server:

cd \fnsw\bin

Try a basic "little pipe" ping:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=ping
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=ping

Set the little pipe’s CE connection configuration:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=setConfig /uri=http://hqbpm34:7001/wsi/FNCEWS40MTOM/ /user=PEAdmin /password=secret
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=setConfig /uri=http://hqbpm34:7001/wsi/FNCEWS40MTOM/ /user=PEAdmin /password=secret

Get a list of all realm names:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getRealmNames
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getRealmNames

Get info about a user named PEAdmin:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getUserInfo /user=PEAdmin
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getUserInfo /user=PEAdmin

Get info about a user named PEAdmin in a specific realm:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getUserInfo /user=PEAdmin /realm= dc=epbdc,dc=eng,dc=filenet,dc=com
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getUserInfo /user=PEAdmin /realm= dc=epbdc,dc=eng,dc=filenet,dc=com

Get info about a group named PEAdministrators:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getUserInfo /user=PEAdministrators
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=getUserInfo /user=PEAdministrators

Get “group expansion info” for a group named PEAdministrators:
Windows:
\fnsw\java\jre\bin\java -classpath pe.jar;pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=expandGroup /group=PEAdministrators
Unix:
/fnsw/java/jre/bin/java -classpath pe.jar:pe3pt.jar filenet.pe.ceorb.test.PEDirectoryClientTest /port=32776 /rpc=expandGroup /group=PEAdministrators

The above runs should produce output of some sort that might shed light on a CE configuration or connection problem…


Get some detailed information about the PE to CE communications

If the PE still can not get connected to the CE, we need to enable some of the java tracing to see what’s happening when the PEDirectoryServer communicates with the CE.

For PE 4.0.2 and newer, including PE 4.5

To enable the tracing, we need an fnlog4j.properties file in the \fnsw\java\jre\lib directory on the PE Server. The fnlog4j.properties file needs to include a line something like this:

log4j.logger.filenet.pe.ceorb.server=DEBUG, TXT

The easiest way to do this is to copy the \fnsw_loc\sd\fnlog4j.properties.sample file as \fnsw\jre\lib\fnlog4j.properties (without the “.sample”). Then, using a simple text editor (e.g., notepad), add the “, TXT” to the above line of text in the fnlog4j.properties file.

The PEDirectoryServer will see the fnlog4j.properties file and that will trigger additional tracing and logging to occur. After putting the fnlog4j.properties file in the \fnsw\java\jre\lib directory, click the Apply button in the Process Task Manager’s Security tab again.

After it fails, check the java trace file. The location of the java trace file is defined in the fnlog4j.properties file itself. By default, it's c:\PE.txt.


For PE 4.0 and PE 4.0.1

To enable the tracing, we need an fnlogging.properties file in the \fnsw\jre\lib directory on the PE Server. The fnlogging.properties file needs to include this line:

filenet.pe.ceorb.server.level = FINEST

The easiest way to do this is to copy the \fnsw_loc\sd\fnlogging.properties.sample file as \fnsw\jre\lib\fnlogging.properties (without the ".sample").

The PEDirectoryServer will see the fnlogging.properties file and that will trigger additional tracing and logging to occur.

Assuming you’re having trouble getting the PE’s security settings applied, after putting the fnlogging.properties file in the \fnsw\jre\lib directory, click the Apply button in the Process Task Manager’s Security tab again.

After it fails, check the java trace file. The location of the java trace file is defined in the fnlogging.properties file itself. By default, it’s c:\PE.txt.


The Trace File

The trace file will have information about what the PEDirectoryServer is doing. You’ll see it building JAAS credentials for the service user, and then using those credentials to connect to the CE Server. And, since something’s going wrong, we’d expect some sort of error information to be logged in this trace file. This information might lead you to resolving the problem.

If there’s a java exception being traced, look at the deepest "caused by..." information. That’s the real issue.

Once you’ve got the PE Security configuration working, disable the java tracing by renaming the fnlogging.properties or fnlog4j.properties file to something like fnlogging.properties.hide or fnlog4j.properties.hide.


Appendix – The Error Numbers

These are the error numbers that may show up in the exceptions that occur when you’re trying to configure the PE’s connection to the CE. Generally, the text of the exception itself already has the information that the exception number conveys.

public static final int _ERR_IDL_VERSION_MISMATCH = 0;
public static final int _ERR_PROBLEM_CONFIGURING_CE_CONNECTION = 1;
public static final int _ERR_LITTLE_PIPE_ORB_NOT_CONFIGURED = 2;
public static final int _ERR_CANT_CREATE_JAAS_CONTEXT = 3;
public static final int _ERR_CANT_CREATE_LOGIN_CONTEXT = 4;
public static final int _ERR_EXPIRED_ACCOUNT = 5;
public static final int _ERR_EXPIRED_CREDENTIALS = 6;
public static final int _ERR_AUTHENTICATION_FAILED = 7;
public static final int _ERR_PROBLEM_GETTING_JAAS_SUBJECT = 8;
public static final int _ERR_LOGIN_PROBLEM = 9;
public static final int _ERR_DUPLICATE_NAME = 10;
public static final int _ERR_PROBLEM_GETUSERINFO = 11;
public static final int _ERR_REALM_NOT_FOUND = 12;
public static final int _ERR_PROBLEM_GROUP_EXPANSION = 13;
public static final int _ERR_PROBLEM_GETTING_REALM = 14;
public static final int _ERR_NO_MEMBERS_FOUND = 15;
public static final int _ERR_NAME_NOT_FOUND = 16;
public static final int _ERR_NO_REALMS_FOUND = 17;
public static final int _ERR_NO_DECIMALFORMATTER_FOUND = 18;
public static final int _ERR_NO_DATEFORMATTER_FOUND = 19;