Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
21 replies Latest Post - ‏2012-02-01T02:25:40Z by NPfister
NPfister
NPfister
43 Posts
ACCEPTED ANSWER

Pinned topic SSL0204E

‏2012-01-17T18:54:32Z |
Hello,

I am getting an SSL0204E message, seemingly randomly. It seems to just choose a single user here or there and affect them. When they close their browser and open a new one, it usually clears up.

I found that:
Message: SSL0204E: Handshake Failed, Internal memory allocation failure.
Reason: The server could not allocate memory needed to complete the operation.
Solution: Take action to free up some additional memory. Try reducing the number of threads or processes running, or increasing virtual memory.

The solution is to free up some additional memory...but I don't seem to see that IHS is using a lot of memory. I've set the MEMLIMIT of the task to 512M, which by the calculation provided in one of the manuals, should be more than plenty (the calculation being 6 * (ThreadsPerChild + 3) meg; translating to 6 * (25 + 3) for my instance which equates to 168M, I upped it to 512M to ensure that this wasn't an issue.)

So my question is has anyone else run across this issue? How can I fix this? I want to get off of IHS 5.3, but IHS 7 isn't cooperating as well as I would have hoped it would be.

Thanks.
Updated on 2012-02-01T02:25:40Z at 2012-02-01T02:25:40Z by NPfister
  • Sunit
    Sunit
    194 Posts
    ACCEPTED ANSWER

    Re: SSL0204E

    ‏2012-01-19T14:38:10Z  in response to NPfister
    Is this on z/OS? What happens when you run with MEMLIMIT 0?

    • Sunit
    • NPfister
      NPfister
      43 Posts
      ACCEPTED ANSWER

      Re: SSL0204E

      ‏2012-01-19T14:55:10Z  in response to Sunit
      Yes, this is on z/OS. The book that I was referencing that said to set the MEMLIMIT the way I did, said that it will not work properly with MEMLIMIT 0. I had it running as MEMLIMIT 0 for a while and was getting the same issues, I was hoping that by changing that I would get around this issue...but that wasn't the case.

      This server has been nothing but a headache trying to make it work properly on z/OS
      • Sunit
        Sunit
        194 Posts
        ACCEPTED ANSWER

        Re: SSL0204E

        ‏2012-01-20T15:29:55Z  in response to NPfister
        Interesting. I have configured and used this HTTP server without any issues in the past.

        MEMLIMIT is used to control usage of memory above the bar (2GB). To use anything above the bar the executable must have AMODE 64. As you are using IHS 5.3 you might be running out of memory below the bar (I think it has AMODE of 24 or 31).

        The memory usage will depend on what you are doing within the HTTP Server address space. Are you using caching? How much memory have you allocated to garbage collection? Are you using GWAPI?

        Having said that, do you know if you have IEFUSI exit on your installation and if yes what are the limits specified in that? Because IEFUSI overrides everything.

        • Sunit
        • NPfister
          NPfister
          43 Posts
          ACCEPTED ANSWER

          Re: SSL0204E

          ‏2012-01-20T17:38:26Z  in response to Sunit
          Currently under IHS 5.3 we're not experiencing any issues.

          On IHS 7 is where we get the SSL error.

          I've tried running both 31 and 64 bit; getting the same error with both versions.

          Garbage collection and caching I have just left at default settings...I've been trying to do as little customization as possible to the conf, just to get this working then I was planning on getting everything customized.

          I just looked, and we are indeed using the IEFUSI exit. The MEMLIMIT in the SMFPRM is set to 0M. Does that mean this will override whatever I have the ALTUSER command set it to? I thought this was just a default setting, but then again, I don't know that much about exits.

          Do you have any suggestions to what I should set caching and other things, and with which directives, in the conf file? Or any other system settings that might alleviate this problem?

          Thanks in advance.
  • gregames
    gregames
    4 Posts
    ACCEPTED ANSWER

    Re: SSL0204E

    ‏2012-01-20T19:45:53Z  in response to NPfister
    Do you know how much memory the IHS 7.0 processes are using? Here's how it looks on my test system which is sitting idle:

    $ ps -eo ruser,pid,ppid,vsz,vsz64,args | egrep "PID|httpd" | grep -v grep
    RUSER PID PPID VSZ VSZ64 COMMAND
    USER193 33554513 1 11216 7340032 /u/USER193/svn/2.2.8/built/bin/httpd -d /u/USER193/svn/2.2.8/built -f conf/ssl
    USER193 146 33554513 14680 414187520 /u/USER193/svn/2.2.8/built/bin/httpd -d /u/USER193/svn/2.2.8/built -f conf/ssl

    VSZ and VSZ64 are 31 bit and 64 bit memory. The process with the big VSZ64 is the Apache child process which does most of the work, including calling System SSL for handshakes.

    Greg
    • NPfister
      NPfister
      43 Posts
      ACCEPTED ANSWER

      Re: SSL0204E

      ‏2012-01-20T20:41:18Z  in response to gregames
      Here's what I come back with at idle:
      RUSER PID PPID VSZ VSZ64 COMMAND
      APACHE 33554472 50331693 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
      APACHE 50331693 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN
      APACHE 33554480 33554472 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
      APACHE 83886131 33554480 14200 426770432 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
      APACHE 67108916 33554480 13452 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
      APACHE 50331829 33554480 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
      APACHE 183 33554480 14256 517996544 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
      • Sunit
        Sunit
        194 Posts
        ACCEPTED ANSWER

        Re: SSL0204E

        ‏2012-01-23T14:10:01Z  in response to NPfister
        How have you sized the server? What are your settings for StartServers, maxClients, ThreadsPerChild, etc?

        IEFUSI settings override everything.

        • Sunit
        • NPfister
          NPfister
          43 Posts
          ACCEPTED ANSWER

          Re: SSL0204E

          ‏2012-01-23T17:27:13Z  in response to Sunit
          <IfModule worker.c>
          ThreadLimit 25
          ServerLimit 64
          StartServers 1
          MaxClients 250
          MinSpareThreads 25
          MaxSpareThreads 75
          ThreadsPerChild 25
          MaxRequestsPerChild 0
          </IfModule>
  • gregames
    gregames
    4 Posts
    ACCEPTED ANSWER

    Re: SSL0204E

    ‏2012-01-23T22:48:42Z  in response to NPfister
    You've got a couple of Apache child processes that are using quite a bit of 64 bit memory. One has 426M, another 518M. The second is probably bumping into the MEMLIMIT 512 setting. It could be that the failures happen for HTTP connections which the process with 518M is trying unsuccessfully to handle. Then when the user restarts the browser, the new connection is routed to another process using less memory and it works.

    It's hard to say whether 518M for ThreadsPerChild 25 is a leak. That's roughly 12M per thead. By default, Apache keeps all memory it uses in per-thread pools to save on alloc()/free() system call cycles. The downside of this is that unused memory isn't shared among threads. You can change that behavior by coding "MaxMemFree nnnn" (kbytes) in your httpd.conf file.
    http://publib.boulder.ibm.com/httpserv/manual70/mod/mpm_common.html#maxmemfree This will limit how much memory each thread can hang on to before it is free()d so other threads can use it. I would try coding MaxMemFree 6000 (about half of the current 12M/thread max usage) and monitor the overall usage with ps periodically. "bin/apachectl graceful" will restart the server with your new config without disrupting users and get rid of the processes that are using a lot of memory.
  • gregames
    gregames
    4 Posts
    ACCEPTED ANSWER

    Re: SSL0204E

    ‏2012-01-23T23:04:39Z  in response to NPfister
    I forgot to mention that my test server had ThreadsPerChild 128, so the 414M of memory it used is only about 3M/thread. However it never did any work either so I would expect it to have low memory use.
    • NPfister
      NPfister
      43 Posts
      ACCEPTED ANSWER

      Re: SSL0204E

      ‏2012-01-25T16:06:03Z  in response to gregames
      I changed my setting, but I still get some high VSZ64. It doesn't seem to be freeing it. I also continue to get the SSL0204E errors.
      • NPfister
        NPfister
        43 Posts
        ACCEPTED ANSWER

        Re: SSL0204E

        ‏2012-01-25T17:36:48Z  in response to NPfister
        Also, of note is that the memory being used by the server, even when idle, starts pretty high (this is a snapshot taken right after taking down the server then starting it):

        Wed Jan 25 11:24:28 EST 2012
        RUSER PID PPID VSZ VSZ64 COMMAND
        APACHE 67108904 83886125 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 83886125 67109276 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
        APACHE 83886251 67108904 14152 94371840 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 83886252 67108904 14164 340787200 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109179 67108904 14152 94371840 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109273 67108904 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 50332059 67108904 13364 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109276 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN

        Inevitably, one of the threads starts out at a much higher utilization and eventually reaches the point that it gets above the 512 mark. Even with a MaxMemFree set to 6000, it doesn't seem to free.

        I ran some testing today having about a dozen users actively hitting the site, just to see how it reacted and it doesn't seem to have ever freed anything:

        Here, at idle before I had people log on:
        Wed Jan 25 10:51:14 EST 2012
        RUSER PID PPID VSZ VSZ64 COMMAND
        APACHE 33554472 50331693 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
        APACHE 50331693 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN
        APACHE 33554480 33554472 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554749 33554480 13452 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 318 33554480 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 16777535 33554480 14196 432013312 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109196 33554480 14244 480247808 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c

        Here, about 2 minutes into it:
        Wed Jan 25 11:01:34 EST 2012
        RUSER PID PPID VSZ VSZ64 COMMAND
        APACHE 33554472 50331693 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
        APACHE 50331693 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN
        APACHE 33554480 33554472 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554749 33554480 13452 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 318 33554480 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 16777535 33554480 14196 432013312 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109196 33554480 14248 480247808 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c

        Here, about 5 minutes into it:
        Wed Jan 25 11:05:25 EST 2012
        RUSER PID PPID VSZ VSZ64 COMMAND
        APACHE 33554472 50331693 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
        APACHE 50331693 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN
        APACHE 33554480 33554472 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554749 33554480 13452 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 318 33554480 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 16777535 33554480 14196 432013312 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109196 33554480 14256 530579456 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c

        Here, at 10 minutes into it we saw significant degredation of the server. It was utilizing approx 40% CPU according to the SDSF:
        Wed Jan 25 11:10:15 EST 2012
        RUSER PID PPID VSZ VSZ64 COMMAND
        APACHE 33554472 50331693 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
        APACHE 50331693 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN
        APACHE 33554480 33554472 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554749 33554480 13452 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 318 33554480 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 16777535 33554480 14200 432013312 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109196 33554480 14256 530579456 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554828 33554480 14160 398458880 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c

        Here, at 12 minutes in everyone was just hanging, it stopped serving up pages:
        Wed Jan 25 11:12:05 EST 2012
        RUSER PID PPID VSZ VSZ64 COMMAND
        APACHE 33554472 50331693 8008 4194304 /bin/sh /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -D
        APACHE 50331693 1 488 0 -sh -c /usr/lpp/IHSA/V7R0/server/bin/apachectl -k start -f conf/httpd.conf -DN
        APACHE 33554480 33554472 13460 14680064 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554749 33554480 13452 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 318 33554480 0 12582912 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 16777535 33554480 14208 480247808 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 67109196 33554480 14256 530579456 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c
        APACHE 33554828 33554480 14204 524288000 /usr/lpp/IHSA/V7R0/server/bin/httpd -d /usr/lpp/IHSA/V7R0/server -k start -f c

        Are there any other memory freeing actions that I can take so that this doesn't happen? I started seeing more of the SSL0204 errors at about 5 minutes into my testing, which I would expect seeing the memory usage. Should this be using this much memory from the start, and shouldn't it be freeing it from time to time? I thought I had an idea of what I was doing with Apache directives, but apparently I am unclear about something. Any suggestions would be greatly appreciated, and thank you for all your help thus far.
        • Sunit
          Sunit
          194 Posts
          ACCEPTED ANSWER

          Re: SSL0204E

          ‏2012-01-26T15:44:25Z  in response to NPfister
          configure mod_status to check what the threads are doing.

          • Sunit
          • NPfister
            NPfister
            43 Posts
            ACCEPTED ANSWER

            Re: SSL0204E

            ‏2012-01-30T15:11:20Z  in response to Sunit
            Ok, so looking at this....

            Mon Jan 30 10:01:24 EST 2012
            RUSER PID PPID VSZ VSZ64 COMMAND
            APACHE 33555149 33555152 8008 4194304 /bin/sh /etc/devserver/bin/apachectl -k start -f conf/httpd.conf -DNO_DETACH
            APACHE 50332366 33555149 13464 14680064 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
            APACHE 33555152 1 488 0 -sh -c /etc/devserver/bin/apachectl -k start -f conf/httpd.conf -DNO_DETACH
            APACHE 735 50332366 13368 12582912 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
            APACHE 736 50332366 0 12582912 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
            APACHE 738 50332366 14172 94371840 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
            APACHE 739 50332366 14172 341835776 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
            APACHE 740 50332366 14172 94371840 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
            Here's what server-status tells me:

            Current Time: Monday, 30-Jan-2012 10:04:47 EST
            Restart Time: Monday, 30-Jan-2012 09:05:22 EST
            Parent Server Generation: 0
            Server uptime: 59 minutes 24 seconds
            Total accesses: 17 - Total Traffic: 61 kB
            CPU Usage: u.04 s.01 cu0 cs0 - .0014% CPU load
            .00477 requests/sec - 17 B/second - 3674 B/request
            1 requests currently being processed, 74 idle workers
            Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
            2-0 739 0/5/5 _ 0.05 2300 11 0.0 0.04 0.04 IP HOST GET /favicon.ico HTTP/1.1
            2-0 739 0/3/3 _ 0.03 2300 61 0.0 0.01 0.01 IP HOST GET /pub/img/disa/DISA_sidebar.jpg HTTP/1.1
            2-0 739 0/1/1 _ 0.03 2300 44 0.0 0.01 0.01 IP HOST GET /pub/img/ie_icon.jpg HTTP/1.1
            2-0 739 0/1/1 _ 0.03 2300 33 0.0 0.01 0.01 IP HOST GET /pub/img/disa/DISA_bottom_seal.gif HTTP/1.1
            2-0 739 0/5/5 _ 0.05 1291 4 0.0 0.00 0.00 IP HOST GET /pub/img/disa/DISA_bottom_seal.gif HTTP/1.1
            2-0 739 0/2/2 _ 0.04 1289 5 0.0 0.00 0.00 IP HOST GET /pub/img/ie_icon.jpg HTTP/1.1
            2-0 739 0/0/0 W 0.00 0 0 0.0 0.00 0.00 IP HOST GET /server-status HTTP/1.1

            It doesn't look to me like it's doing anything while idle? Am I missing something?
  • gregames
    gregames
    4 Posts
    ACCEPTED ANSWER

    Re: SSL0204E

    ‏2012-01-30T21:53:38Z  in response to NPfister
    The server-status output is normal for an idle server.

    Since MaxMemFree has no effect, the problem isn't in the Apache memory pools used by most of the core server. Maybe it's caused by one of the modules. Can you copy the config file and disable mod_ibm_ssl.so and mod_was_ap22_http.so and any others that didn't come with IHS (comment out the LoadModules plus any module specific config lines), start it, and do the ps display?

    It's odd that the child processes have such different memory use when you first start it...
    • NPfister
      NPfister
      43 Posts
      ACCEPTED ANSWER

      Re: SSL0204E

      ‏2012-01-31T12:45:45Z  in response to gregames
      I will comment out the modules and such that didn't come with the server, but I thought of something else that may be what's causing my issue.

      Does IHS cache revocation lists? Since we're a government entity we are require to use CAC login to the system, and we have ridiculously large revocation lists. Would this show up in the memory utilization, if IHS caches them?
      • SystemAdmin
        SystemAdmin
        3908 Posts
        ACCEPTED ANSWER

        Re: SSL0204E

        ‏2012-01-31T15:37:31Z  in response to NPfister
        It seems like System SSL caches this by default, you can try running with GSK_CRL_CACHE_TIMEOUT=0 in your environment.
        • NPfister
          NPfister
          43 Posts
          ACCEPTED ANSWER

          Re: SSL0204E

          ‏2012-01-31T15:57:34Z  in response to SystemAdmin
          Where would I set this?
          • NPfister
            NPfister
            43 Posts
            ACCEPTED ANSWER

            Re: SSL0204E

            ‏2012-01-31T19:02:52Z  in response to NPfister
            Nevermind, I figured out that I should set it in the envvars file. This didn't seem to help at all. Still did the same thing with the memory.
            • NPfister
              NPfister
              43 Posts
              ACCEPTED ANSWER

              Re: SSL0204E

              ‏2012-02-01T02:25:40Z  in response to NPfister
              My management has requested that I open an ETR for this issue due to the need to get the server up and running. Thank you everyone for your help.

              In case you want to know, the ETR is 83705,180,000
    • NPfister
      NPfister
      43 Posts
      ACCEPTED ANSWER

      Re: SSL0204E

      ‏2012-01-31T13:44:35Z  in response to gregames
      Having removed JUST mod_ibm_ssl, here are my results:

      Tue Jan 31 08:46:06 EST 2012
      RUSER PID PPID VSZ VSZ64 COMMAND
      APACHE 50332375 33555166 12608 14680064 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
      APACHE 33555160 50332375 0 12582912 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
      APACHE 33555161 1 488 0 -sh -c /etc/devserver/bin/apachectl -k start -f conf/httpd.conf -DNO_DETACH
      APACHE 33555166 33555161 8008 4194304 /bin/sh /etc/devserver/bin/apachectl -k start -f conf/httpd.conf -DNO_DETACH
      APACHE 16777955 50332375 13300 94371840 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
      APACHE 16777956 50332375 13340 94371840 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
      APACHE 50332412 50332375 13300 94371840 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
      APACHE 50332419 50332375 0 12582912 /etc/devserver/bin/httpd -d /etc/devserver -k start -f conf/httpd.conf -DNO_DE
      BPXROOT 775 16777983 9788 4194304 vi httpd.conf
      This seems to me to confirm my suspicions that the revocation lists have a big thing to do with the memory utilization.

      I will remove other modules and see if I get other results, but this seems to me to point to ssl.