Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
18 replies Latest Post - ‏2010-06-01T03:07:04Z by mowusua
portalPC
portalPC
4 Posts
ACCEPTED ANSWER

Pinned topic Portal Search on System I - Very very slow

‏2009-11-26T17:36:49Z |
Hi all.

I thought I'd try here first in the hope that I don't have to raise a PMR.

I've set up search across the WCM content on our Portal 6.1.0.2 System I environment and when I kick it off it seem to take forever to gather any documents and the processor load and paging is practically zero.
For example, the run I just finished I had set for a maximum of 4 hours and in that time it gathered 110 documents (out of about 10,000).

We do have errors in the logs as shown below:

11/27/09 3:21:06:478 CST 0000002f OutsideInConv E com.ibm.hrl.ws.util.OutsideInConverter convertInputStreamToByteWithTimeLimit: thread didn't fnish normaly...
11/27/09 3:21:06:481 CST 00000032 SystemOut O Stellent Conversion Error code:
11/27/09 3:21:06:481 CST 00000032 SystemOut O The process was interrupted from the Java side
11/27/09 3:26:06:517 CST 0000002f OutsideInConv E com.ibm.hrl.ws.util.OutsideInConverter convertInputStreamToByteWithTimeLimit: thread didn't fnish normaly...
11/27/09 3:26:06:519 CST 00000032 SystemOut O Stellent Conversion Error code:
11/27/09 3:26:06:519 CST 00000032 SystemOut O The process was interrupted from the Java side

I cannot see any other errors. I will probably try again shortly with tracing set to a higher level.

Has anyone seen this before? If so, what was the solution?

Thanks in advance for any help anyone can offer.

All the best,
Tony
Updated on 2010-06-01T03:07:04Z at 2010-06-01T03:07:04Z by mowusua
  • mowusua
    mowusua
    41 Posts
    ACCEPTED ANSWER

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T22:49:15Z  in response to portalPC
    PortalPC,
    If you are not running into any memory issues then it does appears there may be some documents conversion issues .
    My first guess would be that you do a thread dump (dmpjvm) of the portal server job. Ideally a couple of them a few mins apart during the crawling process.
    Ideally you should see some "stuck" threads that may lead us to the problem
    • mowusua
      mowusua
      41 Posts
      ACCEPTED ANSWER

      Re: Portal Search on System I - Very very slow

      ‏2009-11-26T22:51:43Z  in response to mowusua
      Here is the search mustgather that can help gather more data

      Link: http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21319722
      Here is the essential debug traces
      com.ibm.hrl.*=all:com.ibm.crawler.*=all:com.ibm.portal.search.*=all
      • mowusua
        mowusua
        41 Posts
        ACCEPTED ANSWER

        Re: Portal Search on System I - Very very slow

        ‏2009-11-26T23:21:55Z  in response to mowusua
        If you need direction on using the dmpjvm command
        MustGather: WebSphere Portal Performance issues on i5/OS
        • mowusua
          mowusua
          41 Posts
          ACCEPTED ANSWER

          Re: Portal Search on System I - Very very slow

          ‏2009-11-26T23:38:27Z  in response to mowusua
          You may increase the DCS debugging level as follows

          On the Portal server machine, access the filesystem and cd to wps_root/shared/app and edit the file convertors.xml. You will make the following change:

          <global>
          <property name="ODC_LOG" value="ODC_ERROR"/>
          </global>

          Should be changed to:

          <global>
          <property name="ODC_LOG" value="ODC_DEBUG"/>
          </global>
          Restart server after that.
          It should generate more DCS logging which may be helpful ther is an issue with a particular document being converted
  • mowusua
    mowusua
    41 Posts
    ACCEPTED ANSWER

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T23:39:01Z  in response to portalPC
    Having said all of this I say it probably best to open a pmr
    • portalPC
      portalPC
      4 Posts
      ACCEPTED ANSWER

      Re: Portal Search on System I - Very very slow

      ‏2009-11-28T05:07:11Z  in response to mowusua
      Hi mowusua.

      Thanks for all the info. I thought the PMR route would be the one suggested.

      I also know as part of the PMR process that one of the first things they will suggest is to upgrade to 6.1.0.3 as that has some search related fixes.

      I have a few things I need to check first but I think I'll upgrade to 6.1.0.3 first before raising the PMR.

      I'm not a huge fan of the PMR process as it usually takes at least 2 weeks to get to a solution.

      Thanks for your help though.

      All the best,
      Tony
      • mowusua
        mowusua
        41 Posts
        ACCEPTED ANSWER

        Re: Portal Search on System I - Very very slow

        ‏2009-11-29T18:40:38Z  in response to portalPC
        Yeah a pmr is the best route.. Certainly all the information I asked about will be the same for a pmr. It is clearly obvious that some aspect of document conversion is failing . Where exactly si difficult to ascertain. I would think if you supply all the information requested to the pmr it should take no more than a few days to determine where exactly the problem lies and a possible fix
        Often the delay is with getting the necessary data
        • mowusua
          mowusua
          41 Posts
          ACCEPTED ANSWER

          Re: Portal Search on System I - Very very slow

          ‏2009-11-30T02:42:45Z  in response to mowusua
          if you open a pmr i
        • mowusua
          mowusua
          41 Posts
          ACCEPTED ANSWER

          Re: Portal Search on System I - Very very slow

          ‏2009-11-30T02:42:49Z  in response to mowusua
          if you open a pmr i
        • mowusua
          mowusua
          41 Posts
          ACCEPTED ANSWER

          Re: Portal Search on System I - Very very slow

          ‏2009-11-30T02:45:35Z  in response to mowusua
          if you open a pmr i`d be interested in what the number is.
  • mowusua
    mowusua
    41 Posts
    ACCEPTED ANSWER

    Re: Portal Search on System I - Very very slow

    ‏2010-02-01T01:51:26Z  in response to portalPC
    PortalPC,
    Wwas this ever resolved ? I assume it was and if so whatwas the issue.. just curious
    • portalPC
      portalPC
      4 Posts
      ACCEPTED ANSWER

      Re: Portal Search on System I - Very very slow

      ‏2010-02-01T03:12:01Z  in response to mowusua
      Hi mowusua.

      Actually the PMR is still ongoing. (if your from IBm then see here -> PMR 87628,999,616)
      We had the Christmas break which didn't help.

      We're going through the process now and it appears that crawling individual files is OK and after the first couple of hundred it appears to go faster too.
      We have another issue in that there is 14,000+ documents (WCM HTML + file resources) but it only seems to crawl about 1400 or so.

      I'll keep you posted on this once we have solved it.

      All the best,
      Tony
      • mowusua
        mowusua
        41 Posts
        ACCEPTED ANSWER

        Re: Portal Search on System I - Very very slow

        ‏2010-03-01T02:03:03Z  in response to portalPC
        PortalPC,
        So I just checked that pmr and looks closed . It appears the convrsion problem is now resolved. Is that how you see it from your end as well ?
      • mowusua
        mowusua
        41 Posts
        ACCEPTED ANSWER

        Re: Portal Search on System I - Very very slow

        ‏2010-03-01T02:03:14Z  in response to portalPC
        PortalPC,
        So I just checked that pmr and looks closed . It appears the convrsion problem is now resolved. Is that how you see it from your end as well ?
        • mowusua
          mowusua
          41 Posts
          ACCEPTED ANSWER

          Re: Portal Search on System I - Very very slow

          ‏2010-03-01T02:09:28Z  in response to mowusua
          A follow on pmr 89106,999,616 was opened but it is not clear on the current status
        • mowusua
          mowusua
          41 Posts
          ACCEPTED ANSWER

          Re: Portal Search on System I - Very very slow

          ‏2010-03-01T02:09:53Z  in response to mowusua
          A follow on pmr 89106,999,616 was opened but it is not clear on the current status
  • portalPC
    portalPC
    4 Posts
    ACCEPTED ANSWER

    Re: Portal Search on System I - Very very slow

    ‏2010-03-01T06:11:29Z  in response to portalPC
    The crawling part is OK now. I believe the search colection I initially created was somehow faulty and I've noticed that crawling is very sensitive to anything else going on at the same time.

    Anyway, another PMR has been opened as the crawl is now not an issue (15,000 content items in just over 2 hours) but the index takes forever. Just under 9 hours to crawl and index 15,000 content items.

    Currently I'm trying to work out what the average file size is as well as the maximum file size as the bigger the file, the longer it takes to index.

    So, still going.

    All the best,
    Tony
    • mowusua
      mowusua
      41 Posts
      ACCEPTED ANSWER

      Re: Portal Search on System I - Very very slow

      ‏2010-06-01T03:07:04Z  in response to portalPC
      portalPC
      Has this been adequately handled ? What did you eventually settle on ?