Topic
  • 18 replies
  • Latest Post - ‏2010-06-01T03:07:04Z by mowusua
portalPC
portalPC
4 Posts

Pinned topic Portal Search on System I - Very very slow

‏2009-11-26T17:36:49Z |
Hi all.

I thought I'd try here first in the hope that I don't have to raise a PMR.

I've set up search across the WCM content on our Portal 6.1.0.2 System I environment and when I kick it off it seem to take forever to gather any documents and the processor load and paging is practically zero.
For example, the run I just finished I had set for a maximum of 4 hours and in that time it gathered 110 documents (out of about 10,000).

We do have errors in the logs as shown below:

11/27/09 3:21:06:478 CST 0000002f OutsideInConv E com.ibm.hrl.ws.util.OutsideInConverter convertInputStreamToByteWithTimeLimit: thread didn't fnish normaly...
11/27/09 3:21:06:481 CST 00000032 SystemOut O Stellent Conversion Error code:
11/27/09 3:21:06:481 CST 00000032 SystemOut O The process was interrupted from the Java side
11/27/09 3:26:06:517 CST 0000002f OutsideInConv E com.ibm.hrl.ws.util.OutsideInConverter convertInputStreamToByteWithTimeLimit: thread didn't fnish normaly...
11/27/09 3:26:06:519 CST 00000032 SystemOut O Stellent Conversion Error code:
11/27/09 3:26:06:519 CST 00000032 SystemOut O The process was interrupted from the Java side

I cannot see any other errors. I will probably try again shortly with tracing set to a higher level.

Has anyone seen this before? If so, what was the solution?

Thanks in advance for any help anyone can offer.

All the best,
Tony
Updated on 2010-06-01T03:07:04Z at 2010-06-01T03:07:04Z by mowusua
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T22:49:15Z  
    PortalPC,
    If you are not running into any memory issues then it does appears there may be some documents conversion issues .
    My first guess would be that you do a thread dump (dmpjvm) of the portal server job. Ideally a couple of them a few mins apart during the crawling process.
    Ideally you should see some "stuck" threads that may lead us to the problem
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T22:51:43Z  
    • mowusua
    • ‏2009-11-26T22:49:15Z
    PortalPC,
    If you are not running into any memory issues then it does appears there may be some documents conversion issues .
    My first guess would be that you do a thread dump (dmpjvm) of the portal server job. Ideally a couple of them a few mins apart during the crawling process.
    Ideally you should see some "stuck" threads that may lead us to the problem
    Here is the search mustgather that can help gather more data

    Link: http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21319722
    Here is the essential debug traces
    com.ibm.hrl.*=all:com.ibm.crawler.*=all:com.ibm.portal.search.*=all
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T23:21:55Z  
    • mowusua
    • ‏2009-11-26T22:51:43Z
    Here is the search mustgather that can help gather more data

    Link: http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21319722
    Here is the essential debug traces
    com.ibm.hrl.*=all:com.ibm.crawler.*=all:com.ibm.portal.search.*=all
    If you need direction on using the dmpjvm command
    MustGather: WebSphere Portal Performance issues on i5/OS
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T23:38:27Z  
    • mowusua
    • ‏2009-11-26T23:21:55Z
    If you need direction on using the dmpjvm command
    MustGather: WebSphere Portal Performance issues on i5/OS
    You may increase the DCS debugging level as follows

    On the Portal server machine, access the filesystem and cd to wps_root/shared/app and edit the file convertors.xml. You will make the following change:

    <global>
    <property name="ODC_LOG" value="ODC_ERROR"/>
    </global>

    Should be changed to:

    <global>
    <property name="ODC_LOG" value="ODC_DEBUG"/>
    </global>
    Restart server after that.
    It should generate more DCS logging which may be helpful ther is an issue with a particular document being converted
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-26T23:39:01Z  
    Having said all of this I say it probably best to open a pmr
  • portalPC
    portalPC
    4 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-28T05:07:11Z  
    • mowusua
    • ‏2009-11-26T23:39:01Z
    Having said all of this I say it probably best to open a pmr
    Hi mowusua.

    Thanks for all the info. I thought the PMR route would be the one suggested.

    I also know as part of the PMR process that one of the first things they will suggest is to upgrade to 6.1.0.3 as that has some search related fixes.

    I have a few things I need to check first but I think I'll upgrade to 6.1.0.3 first before raising the PMR.

    I'm not a huge fan of the PMR process as it usually takes at least 2 weeks to get to a solution.

    Thanks for your help though.

    All the best,
    Tony
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-29T18:40:38Z  
    • portalPC
    • ‏2009-11-28T05:07:11Z
    Hi mowusua.

    Thanks for all the info. I thought the PMR route would be the one suggested.

    I also know as part of the PMR process that one of the first things they will suggest is to upgrade to 6.1.0.3 as that has some search related fixes.

    I have a few things I need to check first but I think I'll upgrade to 6.1.0.3 first before raising the PMR.

    I'm not a huge fan of the PMR process as it usually takes at least 2 weeks to get to a solution.

    Thanks for your help though.

    All the best,
    Tony
    Yeah a pmr is the best route.. Certainly all the information I asked about will be the same for a pmr. It is clearly obvious that some aspect of document conversion is failing . Where exactly si difficult to ascertain. I would think if you supply all the information requested to the pmr it should take no more than a few days to determine where exactly the problem lies and a possible fix
    Often the delay is with getting the necessary data
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-30T02:42:45Z  
    • mowusua
    • ‏2009-11-29T18:40:38Z
    Yeah a pmr is the best route.. Certainly all the information I asked about will be the same for a pmr. It is clearly obvious that some aspect of document conversion is failing . Where exactly si difficult to ascertain. I would think if you supply all the information requested to the pmr it should take no more than a few days to determine where exactly the problem lies and a possible fix
    Often the delay is with getting the necessary data
    if you open a pmr i
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-30T02:42:49Z  
    • mowusua
    • ‏2009-11-29T18:40:38Z
    Yeah a pmr is the best route.. Certainly all the information I asked about will be the same for a pmr. It is clearly obvious that some aspect of document conversion is failing . Where exactly si difficult to ascertain. I would think if you supply all the information requested to the pmr it should take no more than a few days to determine where exactly the problem lies and a possible fix
    Often the delay is with getting the necessary data
    if you open a pmr i
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2009-11-30T02:45:35Z  
    • mowusua
    • ‏2009-11-29T18:40:38Z
    Yeah a pmr is the best route.. Certainly all the information I asked about will be the same for a pmr. It is clearly obvious that some aspect of document conversion is failing . Where exactly si difficult to ascertain. I would think if you supply all the information requested to the pmr it should take no more than a few days to determine where exactly the problem lies and a possible fix
    Often the delay is with getting the necessary data
    if you open a pmr i`d be interested in what the number is.
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-02-01T01:51:26Z  
    PortalPC,
    Wwas this ever resolved ? I assume it was and if so whatwas the issue.. just curious
  • portalPC
    portalPC
    4 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-02-01T03:12:01Z  
    • mowusua
    • ‏2010-02-01T01:51:26Z
    PortalPC,
    Wwas this ever resolved ? I assume it was and if so whatwas the issue.. just curious
    Hi mowusua.

    Actually the PMR is still ongoing. (if your from IBm then see here -> PMR 87628,999,616)
    We had the Christmas break which didn't help.

    We're going through the process now and it appears that crawling individual files is OK and after the first couple of hundred it appears to go faster too.
    We have another issue in that there is 14,000+ documents (WCM HTML + file resources) but it only seems to crawl about 1400 or so.

    I'll keep you posted on this once we have solved it.

    All the best,
    Tony
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-03-01T02:03:03Z  
    • portalPC
    • ‏2010-02-01T03:12:01Z
    Hi mowusua.

    Actually the PMR is still ongoing. (if your from IBm then see here -> PMR 87628,999,616)
    We had the Christmas break which didn't help.

    We're going through the process now and it appears that crawling individual files is OK and after the first couple of hundred it appears to go faster too.
    We have another issue in that there is 14,000+ documents (WCM HTML + file resources) but it only seems to crawl about 1400 or so.

    I'll keep you posted on this once we have solved it.

    All the best,
    Tony
    PortalPC,
    So I just checked that pmr and looks closed . It appears the convrsion problem is now resolved. Is that how you see it from your end as well ?
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-03-01T02:03:14Z  
    • portalPC
    • ‏2010-02-01T03:12:01Z
    Hi mowusua.

    Actually the PMR is still ongoing. (if your from IBm then see here -> PMR 87628,999,616)
    We had the Christmas break which didn't help.

    We're going through the process now and it appears that crawling individual files is OK and after the first couple of hundred it appears to go faster too.
    We have another issue in that there is 14,000+ documents (WCM HTML + file resources) but it only seems to crawl about 1400 or so.

    I'll keep you posted on this once we have solved it.

    All the best,
    Tony
    PortalPC,
    So I just checked that pmr and looks closed . It appears the convrsion problem is now resolved. Is that how you see it from your end as well ?
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-03-01T02:09:28Z  
    • mowusua
    • ‏2010-03-01T02:03:14Z
    PortalPC,
    So I just checked that pmr and looks closed . It appears the convrsion problem is now resolved. Is that how you see it from your end as well ?
    A follow on pmr 89106,999,616 was opened but it is not clear on the current status
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-03-01T02:09:53Z  
    • mowusua
    • ‏2010-03-01T02:03:14Z
    PortalPC,
    So I just checked that pmr and looks closed . It appears the convrsion problem is now resolved. Is that how you see it from your end as well ?
    A follow on pmr 89106,999,616 was opened but it is not clear on the current status
  • portalPC
    portalPC
    4 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-03-01T06:11:29Z  
    The crawling part is OK now. I believe the search colection I initially created was somehow faulty and I've noticed that crawling is very sensitive to anything else going on at the same time.

    Anyway, another PMR has been opened as the crawl is now not an issue (15,000 content items in just over 2 hours) but the index takes forever. Just under 9 hours to crawl and index 15,000 content items.

    Currently I'm trying to work out what the average file size is as well as the maximum file size as the bigger the file, the longer it takes to index.

    So, still going.

    All the best,
    Tony
  • mowusua
    mowusua
    41 Posts

    Re: Portal Search on System I - Very very slow

    ‏2010-06-01T03:07:04Z  
    • portalPC
    • ‏2010-03-01T06:11:29Z
    The crawling part is OK now. I believe the search colection I initially created was somehow faulty and I've noticed that crawling is very sensitive to anything else going on at the same time.

    Anyway, another PMR has been opened as the crawl is now not an issue (15,000 content items in just over 2 hours) but the index takes forever. Just under 9 hours to crawl and index 15,000 content items.

    Currently I'm trying to work out what the average file size is as well as the maximum file size as the bigger the file, the longer it takes to index.

    So, still going.

    All the best,
    Tony
    portalPC
    Has this been adequately handled ? What did you eventually settle on ?