Topic
  • 17 replies
  • Latest Post - ‏2014-03-27T15:13:13Z by Andy Choi
mdreza1472
mdreza1472
50 Posts

Pinned topic Extending search results to filter some doc classes

‏2014-01-09T13:47:22Z |

Hello,

 

I have one business requirement where we need to filter out 2 doc classes from the search if some documents of that doc class exists in the result set of search.

 

I have seen one sample plugin where there is a feature with custom search; I believe i can integrate the logic of filtering out 2 doc classes using this custom search.

Please update if i am wrong on above perception.

 

However is there any other method with which i can extend the current Search Feature(ICN default feature) to filter out the 2 doc classes.

 

Any help is greatly appreciated.

 

Kind Regards,

Md Reza.

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-13T06:10:45Z  

    Hello,

    I really did not got any workaround for this.

    Let me just once again summarize my requirement; i need to create one custom search where i need to filter some of the doc classes however, i need to add that custom search in Saved Search Template & not as a Feature.

     

    There is one example to create a Feature for this however can some one give a head start where i can add that custom search to the saved searches.

     

    Kind Regards,

    Md Reza.

  • Gabriel Garcia
    Gabriel Garcia
    112 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-13T20:50:24Z  

    Md,

    If you are not interested in documents of a given class, then you can select to search in all classes except the ones you want to exclude (i.e., multi-select each individual class). Now if you don't want to select each class and instead search for Document and all its subclasses, you could write a response filter for the search.do service that removes the documents with a given class from the results JSON payload.

    Please let me know if this addresses your question.

    Gabriel Garcia

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-14T06:25:53Z  

    Md,

    If you are not interested in documents of a given class, then you can select to search in all classes except the ones you want to exclude (i.e., multi-select each individual class). Now if you don't want to select each class and instead search for Document and all its subclasses, you could write a response filter for the search.do service that removes the documents with a given class from the results JSON payload.

    Please let me know if this addresses your question.

    Gabriel Garcia

    Thanks Gabriel for the Update.

     

    Well i will implement this; however i want my response filter to work only for a specific Search Template & not for other Templates.

     

    That means all other search should work in a normal way except for one.

     

    Can you add any suggestions on this?

     

    Thanks

     

    Regards,

    Md Reza.

  • Gabriel Garcia
    Gabriel Garcia
    112 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-14T23:27:19Z  

    Thanks Gabriel for the Update.

     

    Well i will implement this; however i want my response filter to work only for a specific Search Template & not for other Templates.

     

    That means all other search should work in a normal way except for one.

     

    Can you add any suggestions on this?

     

    Thanks

     

    Regards,

    Md Reza.

    There may be limitations using the response filter approach to remove items from the search results. For example, If the search yields more than one page (200) of items and you only return 150 items for the first page then paging may not work properly on the client list.

    I recommend that you create a search and select only the classes you are interested in, instead of trying to exclude two classes. This is the best way to ensure proper functionality from the OOTB search feature.

    If you still want to try the response filter, you can get the ID of the saved search from the request object passed into the filter method of the response filter and compare it with the ID of your search. The request parameter containing the ID in template_name.

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-16T18:13:59Z  

    There may be limitations using the response filter approach to remove items from the search results. For example, If the search yields more than one page (200) of items and you only return 150 items for the first page then paging may not work properly on the client list.

    I recommend that you create a search and select only the classes you are interested in, instead of trying to exclude two classes. This is the best way to ensure proper functionality from the OOTB search feature.

    If you still want to try the response filter, you can get the ID of the saved search from the request object passed into the filter method of the response filter and compare it with the ID of your search. The request parameter containing the ID in template_name.

    Hello Gabriel,

     

    I have created a filter as per the sample code & also as per you're suggestions; however when i test it ... the code shows "Failure to load the data".

     

    Below code checks for specific document title & removes the first occurence of that row where Doc2 title appears.

     

    package com.ibm.ecm.extension.customsearch;

    import javax.servlet.http.HttpServletRequest;
    import java.util.*;

    import com.ibm.ecm.extension.PluginResponseFilter;
    import com.ibm.ecm.extension.PluginServiceCallbacks;
    import com.ibm.ecm.json.JSONResultSetColumn;
    import com.ibm.ecm.json.JSONResultSetResponse;
    import com.ibm.ecm.json.JSONResultSetRow;
    import com.ibm.json.java.JSONObject;

    import java.io.BufferedWriter;
    import java.io.File;
    import java.io.FileReader;
    import java.io.BufferedReader;
    import java.io.FileWriter;
    import java.util.Enumeration;

    /**
     * This sample filter modifies the search results JSON to demonstrate two
     * capabilities:
     * <ol>
     * <li>To add a custom property formatter. Custom property formatters can be
     * used to provide an alternate HTML for the property values in the content
     * list. The custom property formatter for this sample is in
     * com.ibm.ecm.extension
     * .sample.WebContent.samplePluginDojo.SamplePropertyFormatter.js
     * <li>To modify the results to add a custom column. The column added is titled
     * "Sample" and is filled with sequentially increasing integer values.
     * </ol>
     * To prevent the results changes from always happening, the logic will only
     * take effect if the desktop's id is "sample".
     */
    public class SamplePluginResponseFilter extends PluginResponseFilter {

        @Override
        public String[] getFilteredServices() {
            return new String[] { "/p8/search" };
        }

        @Override
        public void filter(String serverType, PluginServiceCallbacks callbacks,
                HttpServletRequest request, JSONObject jsonResponse)
                throws Exception {        
            String desktopId = request.getParameter("desktop");
            if (desktopId != null && desktopId.equals("sample")) {            
                
                File file = new File("D:\\testing.txt");
                file.setWritable(true);
                                    
                // if file doesnt exists, then create it
                if (!file.exists()) {
                    file.createNewFile();
                }
                
                 FileWriter fstream = new FileWriter(file);
                 BufferedWriter out = new BufferedWriter(fstream);

                 out.write("this is a test");
                 
                 Enumeration<String> parameterNames = request.getParameterNames();
                 out.write("template id:"+ request.getParameter("template_name"));
                 while(parameterNames.hasMoreElements())
                 {
                     String paramName = parameterNames.nextElement();
                     out.write(paramName);
                     out.write("\n");
                 }        
                 JSONResultSetResponse jsonResultSetResponse = (JSONResultSetResponse) jsonResponse;  
                     
                 for (int i = 0; i < jsonResultSetResponse.getRowCount(); i++) {
                        JSONResultSetRow row = jsonResultSetResponse.getRow(i);
                        
                        out.write("doc title:"+ row.getAttributeValue("DocumentTitle"));
                        
                        out.write("row id:"+ row.getId());
                        if(row.getAttributeValue("DocumentTitle").toString().equalsIgnoreCase("Doc2"))
                        {    
                            row.clear();
                            out.write("Doc2 row added");                       
                            break;
                        }                    
                    }
                 
                 out.close();
                 fstream.close();
                 
                 
            }
        }

    }

     

    Am i removing the row properly from the JSONResponse using this code----->> row.clear();

     

    Thanks.

     

    Md Reza.
     

     

     

  • Gabriel Garcia
    Gabriel Garcia
    112 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-16T21:56:22Z  

    Hello Gabriel,

     

    I have created a filter as per the sample code & also as per you're suggestions; however when i test it ... the code shows "Failure to load the data".

     

    Below code checks for specific document title & removes the first occurence of that row where Doc2 title appears.

     

    package com.ibm.ecm.extension.customsearch;

    import javax.servlet.http.HttpServletRequest;
    import java.util.*;

    import com.ibm.ecm.extension.PluginResponseFilter;
    import com.ibm.ecm.extension.PluginServiceCallbacks;
    import com.ibm.ecm.json.JSONResultSetColumn;
    import com.ibm.ecm.json.JSONResultSetResponse;
    import com.ibm.ecm.json.JSONResultSetRow;
    import com.ibm.json.java.JSONObject;

    import java.io.BufferedWriter;
    import java.io.File;
    import java.io.FileReader;
    import java.io.BufferedReader;
    import java.io.FileWriter;
    import java.util.Enumeration;

    /**
     * This sample filter modifies the search results JSON to demonstrate two
     * capabilities:
     * <ol>
     * <li>To add a custom property formatter. Custom property formatters can be
     * used to provide an alternate HTML for the property values in the content
     * list. The custom property formatter for this sample is in
     * com.ibm.ecm.extension
     * .sample.WebContent.samplePluginDojo.SamplePropertyFormatter.js
     * <li>To modify the results to add a custom column. The column added is titled
     * "Sample" and is filled with sequentially increasing integer values.
     * </ol>
     * To prevent the results changes from always happening, the logic will only
     * take effect if the desktop's id is "sample".
     */
    public class SamplePluginResponseFilter extends PluginResponseFilter {

        @Override
        public String[] getFilteredServices() {
            return new String[] { "/p8/search" };
        }

        @Override
        public void filter(String serverType, PluginServiceCallbacks callbacks,
                HttpServletRequest request, JSONObject jsonResponse)
                throws Exception {        
            String desktopId = request.getParameter("desktop");
            if (desktopId != null && desktopId.equals("sample")) {            
                
                File file = new File("D:\\testing.txt");
                file.setWritable(true);
                                    
                // if file doesnt exists, then create it
                if (!file.exists()) {
                    file.createNewFile();
                }
                
                 FileWriter fstream = new FileWriter(file);
                 BufferedWriter out = new BufferedWriter(fstream);

                 out.write("this is a test");
                 
                 Enumeration<String> parameterNames = request.getParameterNames();
                 out.write("template id:"+ request.getParameter("template_name"));
                 while(parameterNames.hasMoreElements())
                 {
                     String paramName = parameterNames.nextElement();
                     out.write(paramName);
                     out.write("\n");
                 }        
                 JSONResultSetResponse jsonResultSetResponse = (JSONResultSetResponse) jsonResponse;  
                     
                 for (int i = 0; i < jsonResultSetResponse.getRowCount(); i++) {
                        JSONResultSetRow row = jsonResultSetResponse.getRow(i);
                        
                        out.write("doc title:"+ row.getAttributeValue("DocumentTitle"));
                        
                        out.write("row id:"+ row.getId());
                        if(row.getAttributeValue("DocumentTitle").toString().equalsIgnoreCase("Doc2"))
                        {    
                            row.clear();
                            out.write("Doc2 row added");                       
                            break;
                        }                    
                    }
                 
                 out.close();
                 fstream.close();
                 
                 
            }
        }

    }

     

    Am i removing the row properly from the JSONResponse using this code----->> row.clear();

     

    Thanks.

     

    Md Reza.
     

     

     

    Row.clear() just empties the row's data. To remove the row, get the list of rows (jsonResultSetResponse.get("rows")), iterate through them, and remove the one you want from the iterator (iterator.remove()).

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-17T05:21:18Z  

    Row.clear() just empties the row's data. To remove the row, get the list of rows (jsonResultSetResponse.get("rows")), iterate through them, and remove the one you want from the iterator (iterator.remove()).

    Hello Gabriel,

     

    Thanks a million ... it is now working but a firm testing of my solution is still remaining.

     

    I will update this post soon for paging bugs in Testing phase.

     

    Kind Regards,

    Md Reza.

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-28T11:59:11Z  

    Hello Gabriel,

     

    Thanks a million ... it is now working but a firm testing of my solution is still remaining.

     

    I will update this post soon for paging bugs in Testing phase.

     

    Kind Regards,

    Md Reza.

    Hello Gabriel,

     

    Can you help me to get the currently logged on user id & user group name in the response filter.

     

    Since for some users i will filter out specific doc classes.

     

    Kind Regards,

    Md Reza.

  • damorris
    damorris
    825 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-28T19:14:03Z  

    Hello Gabriel,

     

    Can you help me to get the currently logged on user id & user group name in the response filter.

     

    Since for some users i will filter out specific doc classes.

     

    Kind Regards,

    Md Reza.

    Group isn't available as far as I know.  The logged on username should be available from the http request.

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-01-29T11:23:15Z  
    • damorris
    • ‏2014-01-28T19:14:03Z

    Group isn't available as far as I know.  The logged on username should be available from the http request.

    Thanks da.

    I have found a workaround solution for this by below code:

     

    UserImpl user =(UserImpl)Factory.User.fetchCurrent(callbacks.getP8Connection(request.getParameter("repositoryId")), null);
             
            GroupSet set= user.get_MemberOfGroups();       

            Iterator itr = set.iterator();

     

    However, after removing some rows from the response i see that the Total value set is 4

     

    However there are only 3 rows visible on the screen since i filtered one row based on some business requirement.

     

    Please suggest where i am going wrong?

     

    Kind Regards,

    Md Reza.

     

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-02-04T10:51:22Z  

    Thanks da.

    I have found a workaround solution for this by below code:

     

    UserImpl user =(UserImpl)Factory.User.fetchCurrent(callbacks.getP8Connection(request.getParameter("repositoryId")), null);
             
            GroupSet set= user.get_MemberOfGroups();       

            Iterator itr = set.iterator();

     

    However, after removing some rows from the response i see that the Total value set is 4

     

    However there are only 3 rows visible on the screen since i filtered one row based on some business requirement.

     

    Please suggest where i am going wrong?

     

    Kind Regards,

    Md Reza.

     

    Hello Dana,

     

    I had filtered now some of the doc classes based on my config file which i specifically created for this plugin.

     

    So now the scenario is if the search is getting 4 records in-total; after my filteration it brings only 3 records.

     

    However, down below for the total count for the content-list it still shows 4 records whereas it should show 3 records since that is what visible to our end users.

     

    Please provide you're valuable inputs on this. How should i be able to now show a count of 3 instead of 4.

     

    Thanks.

     

    Kind Regards,

    Md Reza.

  • Andy Choi
    Andy Choi
    127 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-02-04T16:10:44Z  

    Hello Dana,

     

    I had filtered now some of the doc classes based on my config file which i specifically created for this plugin.

     

    So now the scenario is if the search is getting 4 records in-total; after my filteration it brings only 3 records.

     

    However, down below for the total count for the content-list it still shows 4 records whereas it should show 3 records since that is what visible to our end users.

     

    Please provide you're valuable inputs on this. How should i be able to now show a count of 3 instead of 4.

     

    Thanks.

     

    Kind Regards,

    Md Reza.

    Hello Md,

    There should be "totalCount" property set to 4.
    Can you try updating the value accordingly and see if that works for you?

    Andy

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-02-05T10:27:33Z  
    • Andy Choi
    • ‏2014-02-04T16:10:44Z

    Hello Md,

    There should be "totalCount" property set to 4.
    Can you try updating the value accordingly and see if that works for you?

    Andy

    Many thanks Andy, this is now working perfectly.

     

    Regards,

    Reza.

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-03-26T13:22:37Z  

    Many thanks Andy, this is now working perfectly.

     

    Regards,

    Reza.

    Hello Andy,Dana

     

    Referring to the above discussion the custom response filter plugin is filtering some values & is working fine on the initial load for say suppose 50 items. (I created a custom response filter which removes some documents based on a particular property value).

     

    But I have observed that when there are more than say 200 items upcoming from the search then the filter does not work on these records.

    Please provide a quick suggestion since this is coming in production environment.

     

    Kind Regards,

    Md Reza.

  • damorris
    damorris
    825 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-03-26T13:24:41Z  

    Hello Andy,Dana

     

    Referring to the above discussion the custom response filter plugin is filtering some values & is working fine on the initial load for say suppose 50 items. (I created a custom response filter which removes some documents based on a particular property value).

     

    But I have observed that when there are more than say 200 items upcoming from the search then the filter does not work on these records.

    Please provide a quick suggestion since this is coming in production environment.

     

    Kind Regards,

    Md Reza.

    Have you read the redbook?  Chapters 5 and 6 cover how to create a custom search service and how you also have to handle the 'p8/continueQuery' service if you want to support paging.

  • mdreza1472
    mdreza1472
    50 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-03-26T17:34:09Z  
    • damorris
    • ‏2014-03-26T13:24:41Z

    Have you read the redbook?  Chapters 5 and 6 cover how to create a custom search service and how you also have to handle the 'p8/continueQuery' service if you want to support paging.

    Many Thanks Dana for the response.

     

    Yaah i have seen that now & i found the loophole. however can you help me with the scenario now.

    I have a search template from where the search will be triggered can i some how get the search parameters which will then be used in the request filter to build the SQL query.

    Well, currently i am just filtering the response but now i believe i have to make the request filter after you're inputs.

    However since this is the PROD issue can you help me a fast workaround which can be integrated in the response filter; since i set the totalCount after deleting the rows in the response filter

    Thanks once again.

     

    Kind Regards,

    Md Reza.

     

     

  • Andy Choi
    Andy Choi
    127 Posts

    Re: Extending search results to filter some doc classes

    ‏2014-03-27T15:13:13Z  

    Many Thanks Dana for the response.

     

    Yaah i have seen that now & i found the loophole. however can you help me with the scenario now.

    I have a search template from where the search will be triggered can i some how get the search parameters which will then be used in the request filter to build the SQL query.

    Well, currently i am just filtering the response but now i believe i have to make the request filter after you're inputs.

    However since this is the PROD issue can you help me a fast workaround which can be integrated in the response filter; since i set the totalCount after deleting the rows in the response filter

    Thanks once again.

     

    Kind Regards,

    Md Reza.

     

     

    Md,

    Following forum topic may help addressing your needs.

    https://www.ibm.com/developerworks/community/forums/html/topic?id=1cf2ed81-76ce-4127-afcc-0f9bd6dafade&ps=100

    It looks like Alexandro did following to resolve the paging issue.

    1) Add a response filter to both search and continueQuery actions.
    2) Add a request filter to override "skip" and "itemsNeeded" parameters.

    "What I did was then to keep track of the continuation "iterations", and set the values of "skip" and "itemsNeeded" appropriately (for example, for the first iterations, I set them respectively to 200 - 220, 400 - 420, 600 - 620, etc...)."

    It seems the remaining issue is that it doesn't work well when all the results were filtered out but I think you can try above first and see how it works out before we try addressing the remaining issue.

    Thanks,

    Andy