Topic
6 replies Latest Post - ‏2014-01-16T14:37:02Z by Alessandro.Damiani
Alessandro.Damiani
Alessandro.Damiani
51 Posts
ACCEPTED ANSWER

Pinned topic [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

‏2014-01-07T12:19:54Z |

Hello,

I am experiencing a problem in ICN 2.0.1 working with a CM 8.4.3.2 repository.

I'd also like to say that I've tried doing this same thing on ICN 2.0.2, but I get the same behavior there as well.

Please forgive the verbosity of this post, but the problem is quite intricate to explain...

I have developed a response filter to limit the number of returned results based on customer criteria that cannot be implemented simply with an appropriate ACL.

Thus, what happens is:

1 - the user performs a search

2 - results are returned by ICN and processed by my response filter

3 - the response filter examines the results and removes the ones that do not match the customer's external criteria

4 - the filtered results are passed to the ICN interface and shown to the end user

 

If the overall query results, BEFORE the filter is applied, are less than 200, everything works fine.

If, however, the results are more than 200, the query allows for "continuation", and the troubles begin.

If I just filter results both for the cm/search and cm/continueQuery actions, I get a loop in the continuation, ICN will show the same (filtered) results over and over again.

I figured out that I also need to implement a REQUEST filter, changing the value of the "skip" and "itemsNeeded" parameters, because ICN will automatically populate those respectively with the number of shown (filtered) results, and the number of shown (filtered) results plus 20.

Thus, if the query returns 200 results, but my filter only shows 2, skip will automatically be set to 2, and itemsNeeded to 22. This makes the continuation work badly, and causes weird result loops to be shown.

What I did was then to keep track of the continuation "iterations", and set the values of "skip" and "itemsNeeded" appropriately (for example, for the first iterations, I set them respectively to 200 - 220, 400 - 420, 600 - 620, etc...).

This was a step forward, results now don't loop. However, it still doesn't work 100% well. What happens now is that the filtered results are shown, but a number of "sliding rectangle" wait animations are shown below the filtered results. Despite the animations, nothing is being done by ICN, except if I slide down to the last row of animations, in which case the next "continuation" of the query is triggered (producing more sliding rectangles animations).

It seems that, since the animations fill the results list in the interface, the continuation is suspended.

What I was expecting, instead, was that the queries would continue to the end of the overall result set.

Am I making a mistake in the interpretation and setting of the "skip" and "itemsNeeded" parameters? How can I avoid ICN from filling up the list with "wait" animations, and instead continue querying to the end of the results?

I would really appreciate any help on this, as it is starting to become a critical issue...

Thanks in advance,

 

Alessandro

  • Alessandro.Damiani
    Alessandro.Damiani
    51 Posts
    ACCEPTED ANSWER

    Re: [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

    ‏2014-01-09T09:27:41Z  in response to Alessandro.Damiani

    I have been experimenting a bit more, and perhaps now I am able to better describe the problem.

    It seems that, whenever a ResponseFilter applied to /cm/search or /cm/continueQuery filters out ALL results of the current results "page", ICN will not remove the scrolling rectangle wait animation, and won't move to the next search results page (in other words, it won't call "/cm/continueQuery").

    If the ResponseFilter returns at least one result, however, ICN gets rid of the wait animation, and correctly calls /cm/continueQuery to fetch the next page of results.

    I believe the behavior should be the same both if the filter returns no results, or some results. In both cases, ICN should advance to the next page of results until it fills the ContentList widget with "real" results.

    Is this a dojo/js issue upon receiving an empty search results object, even if it has continuationData?

    Is there a way to solve it?

    • damorris
      damorris
      653 Posts
      ACCEPTED ANSWER

      Re: [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

      ‏2014-01-09T13:28:58Z  in response to Alessandro.Damiani

      Alessandro,

      I've asked my team to look at this to see if there is a way to make this work simply using a response filter.  However, the initial reaction is that this isn't going to work with a response filter.  Generally, the continuation logic for pages of results is built in to the existing implementation.  The OOTB ICN continuation logic expects that any results filtering has already occurred, meaning that P8, CM8, etc already return result sets that have had security filtering applied.  The UI is expecting to have a complete page of results or that the query is actually complete - your code is removing results from that which the UI won't handle properly.

      Generally the best way to implement this would be a CM8 Library Server exit, then it applies to any client application that calls in to CM8 and not just ICN.  The IBM ECM development team strongly discourages customers from trying to implement application level security in 90% of cases because it's not really a secure solution since any other application would still be able to access the data in the repository.

      If I get any update from my team that you can make this work with just a response filter then I'll ask them to post details here.

    • arronla
      arronla
      65 Posts
      ACCEPTED ANSWER

      Re: [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

      ‏2014-01-10T06:30:12Z  in response to Alessandro.Damiani

      Technically what you are doing should work (ignoring that filtering results is a good or bad thing).  If the only thing that is blocking you is when you return no results after filtering and there a bunch of animation blocks left in the content list, then it sounds like you hit a javascript error and the content list is failing to "finish".  Can you check to make sure you aren't?

      cm/continueQuery should be called if the following conditions are met:

      if (itemsNeeded && handler) {
                              // no continuation or we already have all the needed items
                                      if (!this.continuationData || this.items.length >= itemsNeeded) {
                                              handler(this.items);
                                      } else {
                                              var self = this;
                                              var retrievedCallback = function() {
                                                      handler(self.items);
                                              };
                                              this.retrieveNextPage(retrievedCallback, itemsNeeded);
                                      }
                              } else {
                                      return this.items; // just returns the cached items
                              }
      

      Can you trace that in ResultSet.js and make sure that retrieveNextPage is called?  If retrieveNextPage is called, are you getting stuck when it calls buildItems in there?  It's hard to envision what is different due to the filtered results but I would debug those areas and see if we can figure out where it is getting stuck and then we can figure out a solution.

      • Alessandro.Damiani
        Alessandro.Damiani
        51 Posts
        ACCEPTED ANSWER

        Re: [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

        ‏2014-01-10T11:38:05Z  in response to arronla

        Hi Arron, thanks a lot for your reply!

        I will perform the checks on the javascript functions of ResultSet.js as you described as soon as I can get back to the customer's site.

        It is that by the Js code you posted, it looks as if the zero-length of the items array should not be a problem, provided continuationData is present and set (which it is in my case)...

        In the meantime, can I ask you to please explain the usage of the itemsNeeded parameter? I've observed how it is used by ICN during normal unfiltered querying, and noticed that it is always set to 20 + (200 * n), where n is the number of pages already fetched and shown in the ContentList.

        I replicated this same behaviour also for filtered queries, and it seems to work (besides the empty results problem I posted for).

        For example, this means that for fetching the second page of results, I set parameters as follows in a request filter:

        - skip: 200

        - itemsNeeded: 220

        Even if my filter for page 1 returned less than 200 results (let's say 100), and thus ICN would set them this other way:

        - skip: 100

        - itemsNeeded: 120

        By forcing them to 200 and 220, I get the expected behavior, provided my response filter returns SOME results.

        Does it make sense, done this way? Or is there some specific logic in the setting of that "20" increment, that will cause other problems when used with a response filter that reduces results?

        Thanks again, I'll get back to you as soon as I perform those checks.

        • arronla
          arronla
          65 Posts
          ACCEPTED ANSWER

          Re: [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

          ‏2014-01-14T18:54:40Z  in response to Alessandro.Damiani

          The +20 is added to stimulate additional rows so it will put those empty blocks animations.  It is usually automatically set for you inside _ModelStore _fetchHandler so you shouldn't have to set it manually.

          Either way, the way the paging works from my understanding is after a page has gone by, the skip to set to end of those items so it will grab the items between itemsNeeded - skip.  The problem is that after filtering items, the paging is completely different.  I am not sure how cm8 is handling the paging aspect but it will definitely be a problem if it was handled by the p8 pageIterator which wouldn't be able to resume correctly and fit into the correct paging.

          From the top of my head,I would probably set skip to after each pageSize - the items you have filtered out.  Set itemsNeeded to the original itemsNeeded - the items you have filtered out as well.  Do that after each filtering.  See if that works.  You shouldn't have to set the +20 thing yourself.

          • Alessandro.Damiani
            Alessandro.Damiani
            51 Posts
            ACCEPTED ANSWER

            Re: [ICN 2.0.1 - CM 8.4.3.2] - Continued queries / filter problem

            ‏2014-01-16T14:37:02Z  in response to arronla

            Thanks Arron, I'll try experimenting with the itemsNeeded setting once I get onsite at the customer's.

            In order to arrive there well prepared, could you please give me any suggestions you might have on how to collect the info you need to troubleshoot the situation I described in my opening post?

            You told me to trace ResultSet.js and make sure that retrieveNextPage is called, and if retrieveNextPage is called, to see if I am getting stuck when it calls buildItems.

            What is the best practical way to trace this behaviour?

            I will have just one day to collect all the info to post to you, so I would like to be as effective as possible in doing that.

             

            Thanks,

            Alessandro