Pinned topic Serializable Issue ?

• joshuad 0600022106
• ‏2013-04-02T15:00:03Z

For which applications have you enabled WXS session replication? Portal has several applications that do not support session replication (due to serialization issues), so you should only enable specifically the wps application as well as any custom portlets you have created that store serializable attributes. Can you verify WXS session replication is only set at the application level, and only for wps and your custom apps?

More...

• SystemAdmin 110000D4XK
• ‏2013-04-03T08:06:51Z

Thanks for your reply.

I have enabled the the Session Replication at the server level (i.e. in the WebSphere application servers level).

I would like to know how does enabling session replication at server level different than enabling from the wps/custom application level?

Can you provide some pointers for the same and also any other parameters as well

Thanks...

More...

• joshuad 0600022106
• ‏2013-04-03T13:28:12Z

When you enable WXS session replication at the server level, all applications installed on that server will be configured to replicate session data to the remote grid. When using Portal, due to serialization issues with their applications, you must enable WXS at the application level. In the WAS admin console, Enterprise Applications -> <app name> -> Session management -> eXtreme Scale session management settings. Enable session management and input the proper catalog service domain and session grid. Do this for the wps.ear as well as your custom applications. Then disable the server level session replication. Restart the portal server. That should take care of the serialization issues.

More...

• joshuad 0600022106
• ‏2013-04-03T13:28:12Z

When you enable WXS session replication at the server level, all applications installed on that server will be configured to replicate session data to the remote grid. When using Portal, due to serialization issues with their applications, you must enable WXS at the application level. In the WAS admin console, Enterprise Applications -> <app name> -> Session management -> eXtreme Scale session management settings. Enable session management and input the proper catalog service domain and session grid. Do this for the wps.ear as well as your custom applications. Then disable the server level session replication. Restart the portal server. That should take care of the serialization issues.

More...

Hi Joshuad,

I am facing the similar issue. I read this post and have an question here.

Can you let me know how this solution would fit in case of multiple Virtual Portal existing in a Portal environment. I have couple of Virtual Portals in my Portal architecture and would like to implement WXS caching on a specific virtual portal only.

These all Virtual Portals are sharing some common Base Portal resources.

Does enabling the session on wps.ear would invoke it on all the virtual portals as well.

Can you let me know the best practice for the same and also highlight the correct mechanism for the architecture as specified above.

Thanks.

--Devendra

• joshuad 0600022106
• ‏2013-04-03T13:28:12Z

When you enable WXS session replication at the server level, all applications installed on that server will be configured to replicate session data to the remote grid. When using Portal, due to serialization issues with their applications, you must enable WXS at the application level. In the WAS admin console, Enterprise Applications -> <app name> -> Session management -> eXtreme Scale session management settings. Enable session management and input the proper catalog service domain and session grid. Do this for the wps.ear as well as your custom applications. Then disable the server level session replication. Restart the portal server. That should take care of the serialization issues.

More...

Hi,

I have gone through your post and enabled the WXS caching at the portlet level. This means I have enabled caching at my Custom application along with wps.ear and then  restarted the Portal server.

The issue I am facing now is hung threads.

Below is the snippet of the SystemOut.log.

==

==

Environment WebSphere Extreme Scale 8.6.0.2 + Interim Fix

                                WebSphere Portal Server 8.0.0.1 + CF 5

Can anyone let me know what could be done to solve the issue?

• Dev_Dhoot 2700026NV5
• ‏2013-07-12T09:26:40Z

Hi,

 

I have gone through your post and enabled the WXS caching at the portlet level. This means I have enabled caching at my Custom application along with wps.ear and then  restarted the Portal server.

The issue I am facing now is hung threads.

 

Below is the snippet of the SystemOut.log.

==

Caused by: com.ibm.ws.xsspi.xio.exception.MessageTimeOutException [originating=x.x.x.x:0;causedby=*.*.*.*:4009;exid=235]: com.ibm.ws.xs.xio.protobuf.ContainerMessages$GetRequestMessage/reqID=24879 await timeout after 30000 ms contacting *.*.*.*:4009 queue size on insert=138, waiter {id: 8312, index: 14}

 

 

[7/11/13 8:12:52:292 CDT] 00000023 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 11" (0000005e) has been active for 610161 milliseconds and may be hung.  There is/are 7 thread(s) in total in the server that may be hung.

        at java.lang.Object.wait(Native Method)

        at java.lang.Object.wait(Object.java:167)

        at com.ibm.ws.xs.sessionmanager.SessionsLRUOffLoader.checkAndWaitLockKey(SessionsLRUOffLoader.java:284)

        at com.ibm.ws.xs.sessionmanager.SessionsLRUOffLoader.accessObject(SessionsLRUOffLoader.java:557)

        at com.ibm.ws.xs.sessionmanager.HttpSessionRequestWrapper.getSession(HttpSessionRequestWrapper.java:690)

        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)

        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)

        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)

        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)

        at com.ibm.wps.engine.PortalRequestWrapper.getSession(PortalRequestWrapper.java:805)

        at com.ibm.wps.services.authentication.impl.AuthenticationServiceImpl.handleAppServerSessionSecurity(AuthenticationServiceImpl.java:1194)

        at com.ibm.wps.engine.Servlet.doGet(Servlet.java:500)

        at com.ibm.wps.engine.Servlet.doPost(Servlet.java:852)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)

        at com.ibm.wps.engine.Servlet.doFilter(Servlet.java:411)

        at com.ibm.wps.resolver.servlet.ContentHandlerCleanup.doFilter(ContentHandlerCleanup.java:780)

        at com.ibm.wps.resolver.servlet.AbstractFilter.doFilter(AbstractFilter.java:96)

        at com.ibm.wps.engine.Servlet.service(Servlet.java:1374)

==

Environment WebSphere Extreme Scale 8.6.0.2 + Interim Fix

                                WebSphere Portal Server 8.0.0.1 + CF 5

 

Can anyone let me know what could be done to solve the issue?

More...

Can you reproduce the issue with "session=all" trace enabled on the portal server?  If so, please attach the logs directory of the Portal server zipped up and I will take a look:

Stop the portal server

enable "*=info: session=all" trace on the portal server and restart

run as few requests as possible to reproduce the failure

Attach the resulting logs directory of the portal server zipped up

Thanks.

• joshuad 0600022106
• ‏2013-07-12T12:28:53Z

Can you reproduce the issue with "session=all" trace enabled on the portal server?  If so, please attach the logs directory of the Portal server zipped up and I will take a look:

Stop the portal server

enable "*=info: session=all" trace on the portal server and restart

run as few requests as possible to reproduce the failure

Attach the resulting logs directory of the portal server zipped up

 

Thanks.

More...

Joshuad,

Thanks for the reply.

I have followed the above steps and collected the logs for the same timeframe.

Please find attached logs in teh zip format.

Please let me know if you need anything else from my side.

--Devendra

• Dev_Dhoot 2700026NV5
• ‏2013-07-15T12:06:16Z

Joshuad,

 

Thanks for the reply.

I have followed the above steps and collected the logs for the same timeframe.

Please find attached logs in teh zip format.

Please let me know if you need anything else from my side.

 

--Devendra

 

More...

Please try this fix for your problem.  You'll need to stop your portal servers, put this jar in your WAS_HOME/lib directory (backup the jar that's already there), restart the portal servers and retry your scenario.  Please let me know if this works for you.

• joshuad 0600022106
• ‏2013-07-15T16:27:02Z

Please try this fix for your problem.  You'll need to stop your portal servers, put this jar in your WAS_HOME/lib directory (backup the jar that's already there), restart the portal servers and retry your scenario.  Please let me know if this works for you.

More...

Hi Joshuad,

Appreciate your quick response on this!!

I have replaced the "sessionobjectgrid.jar" and reproduced the scenario. The Hung thread exceptions are  now gone!!

The primary exception is the below one:

==

I have attached relevant logs generated for the particular time frame. 

The dojo exceptions are frequently seen in the logs and are causing some application functionality to break.

Also can you let me know the root cause behind the hung thread exception, so that it can be avoided in the future.

Thanks!

--Devendra

• Dev_Dhoot 2700026NV5
• ‏2013-07-16T14:28:06Z

Hi Joshuad,

Appreciate your quick response on this!!

 

I have replaced the "sessionobjectgrid.jar" and reproduced the scenario. The Hung thread exceptions are  now gone!!

The primary exception is the below one:

==

com.ibm.ws.webcontainer.filter.FilterInstanceWrapper doFilter SRVE8109W: Uncaught exception thrown by filter ZipHandler: java.io.FileNotFoundException: SRVE0190E: File not found: /v1.7/dojo/nls/dojo_en.js

at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:893)

at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:874)

at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:435)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)

at com.ibm.wps.resolver.servlet.ContentHandlerGzip.internalDoFilter(ContentHandlerGzip.java:704)

at com.ibm.wps.resolver.servlet.ContentHandlerGzip.doFilter(ContentHandlerGzip.java:451)

at com.ibm.wps.resolver.servlet.AbstractFilter.doFilter(AbstractFilter.java:96)

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)

==

I have attached relevant logs generated for the particular time frame. 

The dojo exceptions are frequently seen in the logs and are causing some application functionality to break.

Also can you let me know the root cause behind the hung thread exception, so that it can be avoided in the future.

Thanks!

--Devendra

 

More...

The dojo issue looks to be a known problem unrelated to eXtreme Scale.  Please take a look at the following link and hopefully it will help:  https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014880367

The hung thread issue was simply a bug in the WXS client code which has been fixed with the jar. There was a certain path that Portal is more likely to go down with how it uses sessions, and that path was not properly releasing certain locks, which caused the hung threads.  The official fix will be available in the near future in the next fixpack for WXS 8.6

• joshuad 0600022106
• ‏2013-07-16T16:53:01Z

The dojo issue looks to be a known problem unrelated to eXtreme Scale.  Please take a look at the following link and hopefully it will help:  https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014880367

 

The hung thread issue was simply a bug in the WXS client code which has been fixed with the jar. There was a certain path that Portal is more likely to go down with how it uses sessions, and that path was not properly releasing certain locks, which caused the hung threads.  The official fix will be available in the near future in the next fixpack for WXS 8.6

More...

I applied the changes as mentioned in your earlier  post and tested the functionality, but it didn't work.

I was not seeing the dojo exception before the WXS was enabled in the Portal server.  I guess, this might be the root cause behind  the functionality to break. If I remove the WXS session configuration, the functionality works. 

Ex: When I traverse on a IBM -default page (Portal 8 theme) and select the Edit Page option, it spins forever and doesn't open the fly-out, the logs shows dojo exception.

Can you guide me what went wrong when WXS session configuration is enabled?

--Devendra

• Dev_Dhoot 2700026NV5
• ‏2013-07-17T12:07:41Z

I applied the changes as mentioned in your earlier  post and tested the functionality, but it didn't work.

I was not seeing the dojo exception before the WXS was enabled in the Portal server.  I guess, this might be the root cause behind  the functionality to break. If I remove the WXS session configuration, the functionality works. 

Ex: When I traverse on a IBM -default page (Portal 8 theme) and select the Edit Page option, it spins forever and doesn't open the fly-out, the logs shows dojo exception.

Can you guide me what went wrong when WXS session configuration is enabled?

--Devendra

More...

I don't see any problems from the provided trace. I can take a look at a WebSphere trace as well if you'd like to rerun:

com.ibm.ws.session.*=all:com.ibm.ws.webcontainer.srt.*=all:session=all

Set that trace on your portal servers, and attach the traces.

• joshuad 0600022106
• ‏2013-07-18T14:59:05Z

I don't see any problems from the provided trace. I can take a look at a WebSphere trace as well if you'd like to rerun:

com.ibm.ws.session.*=all:com.ibm.ws.webcontainer.srt.*=all:session=all

Set that trace on your portal servers, and attach the traces.

More...

I have enabled the traces as mentioned in your previous post and then reproduced  the scenario.

Here is the attached logs folder.  I hope this would be useful in identifying the root cause.

--Devendra

• Dev_Dhoot 2700026NV5
• ‏2013-07-22T12:42:50Z

I have enabled the traces as mentioned in your previous post and then reproduced  the scenario.

Here is the attached logs folder.  I hope this would be useful in identifying the root cause.

 

--Devendra

More...

The last attached logs still show the following trace spec:  [7/22/13 5:45:51:253 CDT] 00000000 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info:session=all.

Please retry with the previously mentioned trace spec:  com.ibm.ws.session.*=all:com.ibm.ws.webcontainer.srt.*=all:session=all

• joshuad 0600022106
• ‏2013-07-22T16:45:26Z

The last attached logs still show the following trace spec:  [7/22/13 5:45:51:253 CDT] 00000000 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info:session=all.

 

Please retry with the previously mentioned trace spec:  com.ibm.ws.session.*=all:com.ibm.ws.webcontainer.srt.*=all:session=all

More...

These are the updated logs with the trace settings enabled. I hope these would be helpful in identifying the root cause.

--Devendra

• Dev_Dhoot 2700026NV5
• ‏2013-07-23T13:07:17Z

These are the updated logs with the trace settings enabled. I hope these would be helpful in identifying the root cause.

 

--Devendra

 

More...

The trace isn't showing any problems with WXS HttpSession.  You could try to enable the Dojo_Resources application to use the same grid that you're using with your other portlets.  Otherwise would have to ask the Portal experts what could be causing the FileNotFoundExceptions.

Have you also followed this step in your configuration?

1. Some versions of WebSphere Portal server can have runtime errors when cookies are added to an HTTP response. Since adds cookies for failover and other purposes, these cookies need to be added to WebSphere Portal server cookie ignore list. For more information, see the cookie.ignore.regex parameter section of Caching pages shared by multiple users on the IBM WebSphere Portal wiki. The two cookies that need to be added to the list are IBMID.* and IBMSessionHandle.*. The updated list may look like this for example"digest\\.ignore.*|LtpaToken|LtpaToken2|JSESSIONID|IBMID.*|IBMSessionHandle.*". For more information, see Caching pages shared by multiple users on the IBM WebSphere Portal wiki.

• joshuad 0600022106
• ‏2013-07-15T16:27:02Z

Please try this fix for your problem.  You'll need to stop your portal servers, put this jar in your WAS_HOME/lib directory (backup the jar that's already there), restart the portal servers and retry your scenario.  Please let me know if this works for you.

More...

We may be having the same issue, where can we see an article about this fix and download a patch?

Running 8.5.0.3 with error messages like this: