Topic
34 replies Latest Post - ‏2012-12-17T14:25:14Z by SystemAdmin
FrankoiTlse
FrankoiTlse
3 Posts
ACCEPTED ANSWER

Pinned topic "Stream ended unexpectedly" exception when processing huge file uploads

‏2012-07-21T14:29:34Z |
Hello everyone,

Our customer recently decided to migrate my application to WebSphere AS 7.0. It is presently deployed on JBoss 4.2 servers. One of the features of this application is to enable large (let's say huge) file uploads using HTTP multipart POST requests. The limit size of such requests is 4 GB.

The multipart POST request processing is achieved using Apache's commons-fileupload library, version 1.2.2. We moved to version 1.2.2 because version 1.2 had a 2 GB limit for HTTP requests (due to the usage of an int instead of a long to store the content length).

The application works perfectly on any Tomcat-based application server. The only condition is to configure Tomcat by setting a proper POST request size limit (parameter *maxPostSize*).

My problem is that when I deploy the application to WebSphere AS 7.0 and I try to send a file upload request greater than 2 GB, I have the same exception as when I was using commons-fileupload 1.2:

org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly
+ at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:359)+
+ at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)+

I do not face this problem when my file upload request has a size smaller than 2 GB. Plus, I am not using any Apache/IHS web server and I perform my tests only by sending my requests directly to the WebSphere AS.

I tried to modify the Web container's custom property disableRequestMessageChunking but it seems to have no effect.

Did I miss other server properties to modify ? Or is there a 2 GB built-in limit in WAS ?

Thanks in advance,

FrankoiTlse
Updated on 2012-12-17T14:25:14Z at 2012-12-17T14:25:14Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    37422 Posts
    ACCEPTED ANSWER

    Re: "Stream ended unexpectedly" exception when processing huge file uploads

    ‏2012-07-21T14:53:02Z  in response to FrankoiTlse
    > Did I miss other server properties to modify ? Or is there a 2 GB built-in limit in WAS ?
    >

    Definitely in the WAS Plugin for request bodies sent with content-length.

    Can you force the request to use Transfer-Encoding: chunked and see if it affects the WAS behavior?
    • FrankoiTlse
      FrankoiTlse
      3 Posts
      ACCEPTED ANSWER

      Re: "Stream ended unexpectedly" exception when processing huge file uploads

      ‏2012-07-21T18:46:47Z  in response to SystemAdmin
      Hello Eric, and thank you.

      No, I do not have a way to force the clients to send the requests in chunked mode.

      Isn't there a way to get rid of this 2 GB limit ? Is it hard-coded in WebSphere AS ?

      FrankoiTlse
      • SystemAdmin
        SystemAdmin
        37422 Posts
        ACCEPTED ANSWER

        Re: "Stream ended unexpectedly" exception when processing huge file uploads

        ‏2012-07-22T01:21:44Z  in response to FrankoiTlse
        > FrankoiTlse wrote:
        > Hello Eric, and thank you.
        >
        > No, I do not have a way to force the clients to send the requests in chunked mode.
        >
        > Isn't there a way to get rid of this 2 GB limit ? Is it hard-coded in WebSphere AS ?
        >

        Without experimentation or a confirmation from IBM in a PMR/technote, it's hard to tell what's actually going on.
        • FrankoiTlse
          FrankoiTlse
          3 Posts
          ACCEPTED ANSWER

          Re: "Stream ended unexpectedly" exception when processing huge file uploads

          ‏2012-07-22T12:47:24Z  in response to SystemAdmin
          I do not think that the problem comes from the WAS plugin because for my tests I did not use any web server, only a standalone WebSphere AS 7.0.0.19 installation.

          It seems that IHS enables to modify the size limit of a POST request, or to disable this limit using property PostSizeLimit in the plugin configuration. But the application server itself does not enable to modify this limit, and the WAS's limit is neither modifiable nor documented.

          I cannot believe that I am the first user having this problem. Do you know where I can look for a technical note regarding this issue ?

          Thank you,

          FrankoiTlse
  • SystemAdmin
    SystemAdmin
    37422 Posts
    ACCEPTED ANSWER

    Re: "Stream ended unexpectedly" exception when processing huge file uploads

    ‏2012-12-11T14:27:46Z  in response to FrankoiTlse
    Hi FrankoiTlse,

    Finally did you get the solution to your problem of upload with WAS ?
    I've got exactly the same exception and I haven't found any solution on the web.
    Thanks in advance.

    Best regards.
    • bpaskin
      bpaskin
      3837 Posts
      ACCEPTED ANSWER

      Re: "Stream ended unexpectedly" exception when processing huge file uploads

      ‏2012-12-11T14:42:56Z  in response to SystemAdmin
      Hi, which operating system are you using, including the version, and which version of WAS are you using?

      Regards,
      Brian
      • SystemAdmin
        SystemAdmin
        37422 Posts
        ACCEPTED ANSWER

        Re: "Stream ended unexpectedly" exception when processing huge file uploads

        ‏2012-12-11T15:02:27Z  in response to bpaskin
        Hi,

        OS : SunOS 5.10 (Solaris 10 x86)
        WAS : 6.1.0.29

        In addition, uploading a huge file (gt 4GB) works fine on JBoss Server on my desktop environment with Firefox 17 and IE9.
        On the deployment environment, the JEE AS is WAS and it's KO. A simple upload CGI which only uses the Web server (IHS) is OK, that's why I guess the WAS config must be modified.

        Thanks.
        Sam
        • bpaskin
          bpaskin
          3837 Posts
          ACCEPTED ANSWER

          Re: "Stream ended unexpectedly" exception when processing huge file uploads

          ‏2012-12-11T15:13:32Z  in response to SystemAdmin
          Hi, Windows desktop and Solaris are completely different in how they handle files. As far as I know WAS does not set a limit to file sizes. Check the maximum size of a file that can be written by using "ulimit -a" and see what the maximum blocks are allowed. Also, where are you seeing this error? In which log?

          Regards,
          Brian
          • bpaskin
            bpaskin
            3837 Posts
            ACCEPTED ANSWER

            Re: "Stream ended unexpectedly" exception when processing huge file uploads

            ‏2012-12-11T15:17:45Z  in response to bpaskin
            Also, if you are using the plugin, make sure you have changed the POST size limit.

            Regards,
            Brian
            • SystemAdmin
              SystemAdmin
              37422 Posts
              ACCEPTED ANSWER

              Re: "Stream ended unexpectedly" exception when processing huge file uploads

              ‏2012-12-11T15:24:03Z  in response to bpaskin
              I don't know how to check that.
              I can tell you that by default the config of the web plugin of my application server is PostSizeLimit="-1" in the plugin-cfg.xml file.
              In the WAS administration interface I don't know how to change this parameter.
          • SystemAdmin
            SystemAdmin
            37422 Posts
            ACCEPTED ANSWER

            Re: "Stream ended unexpectedly" exception when processing huge file uploads

            ‏2012-12-11T15:36:21Z  in response to bpaskin
            > ulimit -a
            time(seconds) unlimited
            file(blocks) unlimited
            data(kbytes) unlimited
            stack(kbytes) 10240
            coredump(blocks) unlimited
            nofiles(descriptors) 8192
            vmemory(kbytes) unlimited

            This error appears in the WAS error log file :
            "org.apache.struts.upload.CommonsMultipartRequestHandler - Failed to parse multipart request
            org.apache.commons.fileupload.FileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly
            at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:518)
            at org.apache.struts.upload.CommonsMultipartRequestHandler.handleRequest(CommonsMultipartRequestHandler.java:193)
            at org.apache.struts.util.RequestUtils.populate(RequestUtils.java:442)
            at org.apache.struts.action.RequestProcessor.processPopulate(RequestProcessor.java:816)
            at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:203)
            at com.inetpsa.fwk.struts.action.FWKTilesRequestProcessor.parentProcess(FWKTilesRequestProcessor.java:202)
            at com.inetpsa.fwk.struts.action.FWKRequestProcessorWrapper.process(FWKRequestProcessorWrapper.java:592)
            at com.inetpsa.fwk.struts.action.FWKTilesRequestProcessor.process(FWKTilesRequestProcessor.java:75)
            at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
            at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
            at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1146)
            at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1087)
            at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:145)
            at com.inetpsa.bqs.utils.Encoding.doFilter(Encoding.java:36)
            at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
            at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:130)
            at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
            at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:837)
            at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:680)
            at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:588)
            at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:525)
            at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
            at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:751)
            at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1478)
            at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:126)
            at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
            at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
            at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:102)
            at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:556)
            at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:606)
            at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:979)
            at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1064)
            at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)"
            • bpaskin
              bpaskin
              3837 Posts
              ACCEPTED ANSWER

              Re: "Stream ended unexpectedly" exception when processing huge file uploads

              ‏2012-12-11T15:41:54Z  in response to SystemAdmin
              Hi, can you check the ConnectionIOTimeout in the plugin file?

              Regards,
              Brian
              • bpaskin
                bpaskin
                3837 Posts
                ACCEPTED ANSWER

                Re: "Stream ended unexpectedly" exception when processing huge file uploads

                ‏2012-12-11T15:43:48Z  in response to bpaskin
                Also, in the httpd.conf file for the web server, make sure it is setup so it can stay open long enough for the file to be transferred.

                WAS is only responding to the events happening in front of it.

                Regards,
                Brian
              • SystemAdmin
                SystemAdmin
                37422 Posts
                ACCEPTED ANSWER

                Re: "Stream ended unexpectedly" exception when processing huge file uploads

                ‏2012-12-11T15:50:11Z  in response to bpaskin
                Hi,

                I've found the following line in the plugin-cfg.xml file for my server :
                <Server ConnectTimeout="0" ExtendedHandshake="false" MaxConnections="-1" Name="node01_bqs00Dev_as" ServerIOTimeout="0" WaitForContinue="false">

                Is it the answer to your quetion ?
                • bpaskin
                  bpaskin
                  3837 Posts
                  ACCEPTED ANSWER

                  Re: "Stream ended unexpectedly" exception when processing huge file uploads

                  ‏2012-12-11T15:58:10Z  in response to SystemAdmin
                  Hi,

                  Check the following:
                  Servers > Server Types > WebSphere application servers > server_name > Web container settings > Web container > HTTP transport. What is the ConnectionIOTimeout value?

                  In the httpd.conf what is the value for Timeout ?

                  Regards,
                  Brian
                  • SystemAdmin
                    SystemAdmin
                    37422 Posts
                    ACCEPTED ANSWER

                    Re: "Stream ended unexpectedly" exception when processing huge file uploads

                    ‏2012-12-11T16:11:57Z  in response to bpaskin
                    In WAS 6.1.0.29 I didn't fine the same menu tree as you.
                    I've seen something approaching in "Serveurs d'applications > bqs00Dev_as > Chaînes de transport du conteneur Web > WCInboundDefaultSecure" (sorry it's in French). :
                    "Canal des communications entrantes HTTP (HTTP_4)

                    Utiliser des connexions persistantes (conserver actives) Activé
                    Nombre maximal de demandes persistantes par connexion 100
                    Délai de lecture 60 secondes
                    Délai d'écriture 60 secondes
                    Délai persistant 30 secondes

                    Canal des communications entrantes du conteneur Web (WCC_4) "

                    httpd.conf :
                    ...
                    #
                    1. Timeout: The number of seconds before receives and sends time out.
                    #
                    Timeout 300
                    ...
                    • bpaskin
                      bpaskin
                      3837 Posts
                      ACCEPTED ANSWER

                      Re: "Stream ended unexpectedly" exception when processing huge file uploads

                      ‏2012-12-11T16:25:05Z  in response to SystemAdmin
                      Hi,

                      No problem, I can read French. Ok, so the question is how long does it take you to upload your 4GB file?

                      Brian
                      • SystemAdmin
                        SystemAdmin
                        37422 Posts
                        ACCEPTED ANSWER

                        Re: "Stream ended unexpectedly" exception when processing huge file uploads

                        ‏2012-12-11T16:58:28Z  in response to bpaskin
                        Considering my connection speed, it should take at least 10 minutes.
                        But I've got the exception immediatly after pressing the submit button on the form page.
                        It's not the case with a 500 MB file which takes several minutes to upload correctly.
                        How to explain this behavior ?

                        Thanks.
                        • bpaskin
                          bpaskin
                          3837 Posts
                          ACCEPTED ANSWER

                          Re: "Stream ended unexpectedly" exception when processing huge file uploads

                          ‏2012-12-11T17:09:33Z  in response to SystemAdmin
                          Hi, you have to change a few things. I would change the web server timeout from 300 seconds (5 mins) to at least 600 seconds (10 mins). You also need to change the connection timeout in the transport for HTTP or HTTPS, whichever you are using to 10 mins, as well. We would need to do tracing on what is causing the issue. I will try this evening / tomorrow morning with a simple test.

                          Regards,
                          Brian
                          • SystemAdmin
                            SystemAdmin
                            37422 Posts
                            ACCEPTED ANSWER

                            Re: "Stream ended unexpectedly" exception when processing huge file uploads

                            ‏2012-12-11T17:14:53Z  in response to bpaskin
                            In addition, I've just successfully uploaded a 1.66GB file in about 5min50s.

                            Otherwise I will apply your advices.

                            Thanks.
                          • SystemAdmin
                            SystemAdmin
                            37422 Posts
                            ACCEPTED ANSWER

                            Re: "Stream ended unexpectedly" exception when processing huge file uploads

                            ‏2012-12-11T17:25:07Z  in response to bpaskin
                            I set Timeout value at 600 in the httpd.conf of the web server.
                            For the WAS config I set this :

                            Canal des communications entrantes HTTP (HTTP_4)

                            Utiliser des connexions persistantes (conserver actives) Activé
                            Nombre maximal de demandes persistantes par connexion 100
                            Délai de lecture 60 secondes
                            Délai d'écriture 60 secondes
                            Délai persistant 600 secondes

                            Is it what you expected ?
                            FYI I've got the same result ...

                            Thanks.
                            • bpaskin
                              bpaskin
                              3837 Posts
                              ACCEPTED ANSWER

                              Re: "Stream ended unexpectedly" exception when processing huge file uploads

                              ‏2012-12-11T20:59:14Z  in response to SystemAdmin
                              Hi,

                              I was able to recreate the problem under versions 6.1, 7, 8 and 8.5. It appears that the header is not being passed correctly. The content-disposition is null. Now I need to investigate why this is happening.

                              Regards,
                              Brian
                              • bpaskin
                                bpaskin
                                3837 Posts
                                ACCEPTED ANSWER

                                Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                ‏2012-12-11T21:26:43Z  in response to bpaskin
                                Hi, I believe I found the issue. Are you using a 32 or 64 bit version of WAS?

                                Brian
                                • SystemAdmin
                                  SystemAdmin
                                  37422 Posts
                                  ACCEPTED ANSWER

                                  Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                  ‏2012-12-12T07:41:40Z  in response to bpaskin
                                  Hi Brian,

                                  How can I check the 32 or 64 bit version of my WAS ?
                                  Thanks.
                                  Sam
                                • SystemAdmin
                                  SystemAdmin
                                  37422 Posts
                                  ACCEPTED ANSWER

                                  Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                  ‏2012-12-12T08:13:09Z  in response to bpaskin
                                  It's a 64 bits version.
                                  Regards.
                                  • bpaskin
                                    bpaskin
                                    3837 Posts
                                    ACCEPTED ANSWER

                                    Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                    ‏2012-12-12T11:47:06Z  in response to SystemAdmin
                                    Hi, if you chunk the request/reply it should work. I need to understand why it is not working with a straight POST. I have an email out to the developer of the module.

                                    Regards,
                                    Brian
                                    • SystemAdmin
                                      SystemAdmin
                                      37422 Posts
                                      ACCEPTED ANSWER

                                      Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                      ‏2012-12-12T12:07:23Z  in response to bpaskin
                                      Hi,

                                      How do you chunk the request/reply ?
                                      Is it a configuration option in WAS ?
                                      Please explain me how to do that, I'm very instered in testing it.
                                      Thanks.

                                      Sam
                                      • bpaskin
                                        bpaskin
                                        3837 Posts
                                        ACCEPTED ANSWER

                                        Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                        ‏2012-12-12T13:42:22Z  in response to SystemAdmin
                                        Hi, This looks like an OS issue. Which operating system are you using to send the file? What I am seeing is that on older OSes, like XP the content-length field is set incorrectly for HTTP/1.1 . From my test I can see the following: Content-Length: -191341355 . If you can capture your headers then you will be able to see if you are having the same issue.

                                        Regards,
                                        Brian
                                        • SystemAdmin
                                          SystemAdmin
                                          37422 Posts
                                          ACCEPTED ANSWER

                                          Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                          ‏2012-12-12T13:58:30Z  in response to bpaskin
                                          Hi,
                                          I'm using Seven64.
                                          With Firefox 17, when I post the form with a 2 GB file, the header infomations are :
                                          (Request-Line)      POST /BDSTYLE_Web/ajoutDocument.do HTTP/1.1
                                          Host    app.inetpsa.com
                                          User-Agent      Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0
                                          Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 Accept-Encoding gzip, deflate Connection keep-alive Referer https://app.inetpsa.com/BDSTYLE_Web/ajoutDocument.do Cookie application=001; style=1; largeur_layergauche=295; AFFICHE_W=TyoEnITJtgTp37; WEBO_PSA=Dir-_Pays-_Etab-_Date-2012%3A12%3A14; JSESSIONID=0000jy_bNUVpbYO8N7QKq1Jldhn:-1 Content-Type multipart/form-data; boundary=---------------------------265001916915724 Content-Length 2383931200
                                          

                                          So Content-length is correctly valued.

                                          Regards.
                                          Updated on 2014-03-25T23:51:17Z at 2014-03-25T23:51:17Z by iron-man
                                          • bpaskin
                                            bpaskin
                                            3837 Posts
                                            ACCEPTED ANSWER

                                            Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                            ‏2012-12-17T11:48:35Z  in response to SystemAdmin
                                            Hi, Can you see if this addresses your issue? http://www-01.ibm.com/support/docview.wss?uid=swg1PM03146

                                            Brian
                                            • SystemAdmin
                                              SystemAdmin
                                              37422 Posts
                                              ACCEPTED ANSWER

                                              Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                              ‏2012-12-17T13:11:42Z  in response to bpaskin
                                              Hi,

                                              So I seems to be the same problem but it doesn't match with my WAS version.
                                              And, if I understand well, the problem is solved with WAS 8 and greater, and you said you reproduced it with this version, so what can you say about that ?
                                              Furthermore, the description of the problem tells that the return code should be "-1", what is the return code ? If it is the content-length value, it is not my case, so I'm not sure it is exactly the same issue.

                                              Regards.
                                              • bpaskin
                                                bpaskin
                                                3837 Posts
                                                ACCEPTED ANSWER

                                                Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                                ‏2012-12-17T13:37:55Z  in response to SystemAdmin
                                                Hi,

                                                You mentioned in your first post that you were using WASv7, is this correct? I can cause the same issue by using an older OS, like WinXP, however, it works fine for me accessing the App Server using Win7, Win8, Ubuntu 12.04 and OSX 10.8. If you the correct number of bytes are being passed in the header, then it would be more likely a timeout issue getting the data to WAS than an actual issue with WAS. As I mentioned, I would upgrade the timeout values and see if that helps. You can open a PMR with IBM Support to help you if you feel there is more help needed.

                                                Regards,
                                                Brian
                                                • SystemAdmin
                                                  SystemAdmin
                                                  37422 Posts
                                                  ACCEPTED ANSWER

                                                  Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                                  ‏2012-12-17T14:25:14Z  in response to bpaskin
                                                  Hi,
                                                  No, the first post was not from me, but as the description of the issue was exactly the same than me, I asked for news about this one.
                                                  As you asked me before (cf history) , I said my WAS version was 6.1.0.29.
                                                  So, I think that a PMR should be opened with IBM Support, because we don't plan to upgrade the WAS version for the moment and I use Win7.
                                                  Thank you very much for your help and the time you spent on this issue.

                                                  Regard.
                                                  Sam
                                        • SystemAdmin
                                          SystemAdmin
                                          37422 Posts
                                          ACCEPTED ANSWER

                                          Re: "Stream ended unexpectedly" exception when processing huge file uploads

                                          ‏2012-12-12T14:00:13Z  in response to bpaskin
                                          In addition, the content result is :
                                          Error loading content (NS_ERROR_DOCUMENT_NOT_CACHED)