Topic
  • 8 replies
  • Latest Post - ‏2017-03-16T15:44:29Z by Daviiid
Daviiid
Daviiid
486 Posts

Pinned topic gatewayScript : buffer.toString('hex')

‏2017-03-09T15:31:29Z |

Hello

i try to encode input message in hex string through a gatewayScript (because dp:radix-convert is limited by 68Mo input, if i remember what Hermann already said Grin)

So I use this script

session.input.readAsBuffers(function (error, buffer) {
  if (error)
  {
    // handle error
    session.output.write (error.errorMessage);
  } else {
    /* write the default output buffer */
	session.output.write(buffer.toString('hex'));
  }
});

It works for little file. But as I play a big file, i have this error below. I guess my file is too big but i want to be sure with Hermann Smile
 

 

16:25:36 gatewayscript error 626641 request 10.75.97.77 0x85800007 mpgw (CHARGEMENT_EXPORT_STATWATCH)GatewayScript processing Error 'Error: toString failed In file 'gatewayscript:///modules/buffer.js' line:377, stack:Error: toString failed at Buffer.toString() (gatewayscript:///modules/buffer.js:377:11) at Buffers.toString (gatewayscript:///modules/buffers.js:303:35) at Object.<anonymous> (local:///dt01/fourTout/myTest.js:10:30)'
  • HermannSW
    HermannSW
    8544 Posts
    ACCEPTED ANSWER

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-15T18:53:18Z  

    Reading through, my bad I thought same issue "Converting hex"

     

    Regards,

    Salla

    OK, Daviid's task was a nice GatewayScript challenge.

    Until now I did most of my scripts with "buffer" datatype and "session.input.readAsBuffer()".

    Since more than 1GB of data should be processed, Buffers object has to be used.
    Since string length is restricted to 256MB, I used 120MB slices, allowing to double without issues.

    I started with inspecting the Buffers API in knowledge center:
    https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.5.0/com.ibm.dp.doc/buffers_js.html

    The programming task and API directed me to this GatewayScript:

    var sz = 120000000;
    
    session.input.readAsBuffers (function (error, bufs) {
      if (error) { session.reject(error); return; }
    
      var i, ret = new Buffers();
    
      for(i=0; i<bufs.length; i+=sz) {
        ret.push(new Buffer(bufs.slice(i, i+sz).toString('hex')));
      }
    
      session.output.write(ret);
    });
    


    Small sample demonstrates that it works:

    $ coproc2 bufs.js <(echo 'foo123äöüBAR') http://dp3-l3:2227 ; echo
    666f6f313233c3a4c3b6c3bc4241520a
    $ echo 'foo123äöüBAR' | od -tx1
    0000000 66 6f 6f 31 32 33 c3 a4 c3 b6 c3 bc 42 41 52 0a
    0000020
    $
    


    Here you see that processing of big files work (output size is twice the input size):

    $ ls -l 40M pi-billion.txt 2G
    -rw-rw-r--. 1 stammw stammw 2000000004 Mar 15 19:17 2G
    -rw-rw-r--. 1 stammw stammw   40000000 Nov 12  2015 40M
    -rw-rw-r--. 1 stammw stammw 1000000002 Mar 13  2009 pi-billion.txt
    $ 
    $ coproc2 bufs.js 2G http://dp3-l3:2227 -s | wc --bytes
    4000000008
    $ coproc2 bufs.js pi-billion.txt http://dp3-l3:2227 -s | wc --bytes
    2000000004
    $ coproc2 bufs.js 40M http://dp3-l3:2227 -s | wc --bytes
    80000000
    $
    


    And her you can see the GatewayScript processing times, determined from extended latency log:

    2G   36.863s ...,AXF=16996,AGS=53859,...
    pi   15.916s ...,AXF=8499,AGS=24415,...
    40M   0.604s ...,AXF=341,AGS=918,...
    


    Just for comparison, since AGS-AXF contains reading from context, processing adn output to context, the processing of a simple "wc --bytes" (word count) on a 3GHz Linux server (not completely comparable since read is from harddisk, and there is no output besides the small statistics):

    $ time wc 2G
             0          1 2000000004 2G
    
    real    0m29.726s
    user    0m29.436s
    sys     0m0.284s
    $
    


    Hermann.

     

    P.S:
    "ret.length" is bytelength of Buffers object "ret".

    "ret.buffers.length" is number of "buffer" objects used to build "Buffers" object "ret".

    You can address ith buffer of "ret" via "ret.buffers[i]", but as you can see in above script that is not needed.

    P.P.S:
    Just checked the Buffer sizes of Buffers object bufs returned from readAsBuffers() on input pi-billion.txt:
    the first 976562 are of size 1024, and the last Buffer is of size 514

    Attachments

    Updated on 2017-03-15T22:12:23Z at 2017-03-15T22:12:23Z by HermannSW
  • HermannSW
    HermannSW
    8544 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-09T21:40:00Z  

    Hi David,

    a big file
    >
    GatewayScript "string" has 256MB maximal size, "Buffer" has 1GB maximal size.

    If you need to process bigger input, you need to use "Buffers" object.

    Hermann.

  • Daviiid
    Daviiid
    486 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-10T12:43:42Z  
    • HermannSW
    • ‏2017-03-09T21:40:00Z

    Hi David,

    a big file
    >
    GatewayScript "string" has 256MB maximal size, "Buffer" has 1GB maximal size.

    If you need to process bigger input, you need to use "Buffers" object.

    Hermann.

    Thanks Hermann

    Can you show a script to convert to hex string a file bigger than  1Go ?

  • Daviiid
    Daviiid
    486 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-13T09:13:50Z  
    • Daviiid
    • ‏2017-03-10T12:43:42Z

    Thanks Hermann

    Can you show a script to convert to hex string a file bigger than  1Go ?

    Sorry

    Anyone know how to handle a file more 1 Go ?

  • soaApiDevArch
    soaApiDevArch
    156 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-15T13:35:36Z  
    • Daviiid
    • ‏2017-03-13T09:13:50Z

    Sorry

    Anyone know how to handle a file more 1 Go ?

    David,

     

    Were you able to resolve?

     

    Can you reply to the post in IIB dev forum ? 

     

    Regards,

    Salla

  • Daviiid
    Daviiid
    486 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-15T13:57:31Z  

    David,

     

    Were you able to resolve?

     

    Can you reply to the post in IIB dev forum ? 

     

    Regards,

    Salla

    Hello Salla

    I don't understant why you want i answer on iib forum while it's a Datapower problem

  • soaApiDevArch
    soaApiDevArch
    156 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-15T14:11:20Z  
    • Daviiid
    • ‏2017-03-15T13:57:31Z

    Hello Salla

    I don't understant why you want i answer on iib forum while it's a Datapower problem

    Reading through, my bad I thought same issue "Converting hex"

     

    Regards,

    Salla

  • HermannSW
    HermannSW
    8544 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-15T18:53:18Z  

    Reading through, my bad I thought same issue "Converting hex"

     

    Regards,

    Salla

    OK, Daviid's task was a nice GatewayScript challenge.

    Until now I did most of my scripts with "buffer" datatype and "session.input.readAsBuffer()".

    Since more than 1GB of data should be processed, Buffers object has to be used.
    Since string length is restricted to 256MB, I used 120MB slices, allowing to double without issues.

    I started with inspecting the Buffers API in knowledge center:
    https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.5.0/com.ibm.dp.doc/buffers_js.html

    The programming task and API directed me to this GatewayScript:

    var sz = 120000000;
    
    session.input.readAsBuffers (function (error, bufs) {
      if (error) { session.reject(error); return; }
    
      var i, ret = new Buffers();
    
      for(i=0; i<bufs.length; i+=sz) {
        ret.push(new Buffer(bufs.slice(i, i+sz).toString('hex')));
      }
    
      session.output.write(ret);
    });
    


    Small sample demonstrates that it works:

    $ coproc2 bufs.js <(echo 'foo123äöüBAR') http://dp3-l3:2227 ; echo
    666f6f313233c3a4c3b6c3bc4241520a
    $ echo 'foo123äöüBAR' | od -tx1
    0000000 66 6f 6f 31 32 33 c3 a4 c3 b6 c3 bc 42 41 52 0a
    0000020
    $
    


    Here you see that processing of big files work (output size is twice the input size):

    $ ls -l 40M pi-billion.txt 2G
    -rw-rw-r--. 1 stammw stammw 2000000004 Mar 15 19:17 2G
    -rw-rw-r--. 1 stammw stammw   40000000 Nov 12  2015 40M
    -rw-rw-r--. 1 stammw stammw 1000000002 Mar 13  2009 pi-billion.txt
    $ 
    $ coproc2 bufs.js 2G http://dp3-l3:2227 -s | wc --bytes
    4000000008
    $ coproc2 bufs.js pi-billion.txt http://dp3-l3:2227 -s | wc --bytes
    2000000004
    $ coproc2 bufs.js 40M http://dp3-l3:2227 -s | wc --bytes
    80000000
    $
    


    And her you can see the GatewayScript processing times, determined from extended latency log:

    2G   36.863s ...,AXF=16996,AGS=53859,...
    pi   15.916s ...,AXF=8499,AGS=24415,...
    40M   0.604s ...,AXF=341,AGS=918,...
    


    Just for comparison, since AGS-AXF contains reading from context, processing adn output to context, the processing of a simple "wc --bytes" (word count) on a 3GHz Linux server (not completely comparable since read is from harddisk, and there is no output besides the small statistics):

    $ time wc 2G
             0          1 2000000004 2G
    
    real    0m29.726s
    user    0m29.436s
    sys     0m0.284s
    $
    


    Hermann.

     

    P.S:
    "ret.length" is bytelength of Buffers object "ret".

    "ret.buffers.length" is number of "buffer" objects used to build "Buffers" object "ret".

    You can address ith buffer of "ret" via "ret.buffers[i]", but as you can see in above script that is not needed.

    P.P.S:
    Just checked the Buffer sizes of Buffers object bufs returned from readAsBuffers() on input pi-billion.txt:
    the first 976562 are of size 1024, and the last Buffer is of size 514

    Attachments

    Updated on 2017-03-15T22:12:23Z at 2017-03-15T22:12:23Z by HermannSW
  • Daviiid
    Daviiid
    486 Posts

    Re: gatewayScript : buffer.toString('hex')

    ‏2017-03-16T15:44:29Z  
    • HermannSW
    • ‏2017-03-15T18:53:18Z

    OK, Daviid's task was a nice GatewayScript challenge.

    Until now I did most of my scripts with "buffer" datatype and "session.input.readAsBuffer()".

    Since more than 1GB of data should be processed, Buffers object has to be used.
    Since string length is restricted to 256MB, I used 120MB slices, allowing to double without issues.

    I started with inspecting the Buffers API in knowledge center:
    https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.5.0/com.ibm.dp.doc/buffers_js.html

    The programming task and API directed me to this GatewayScript:

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">var sz = 120000000; session.input.readAsBuffers (function (error, bufs) { if (error) { session.reject(error); return; } var i, ret = new Buffers(); for(i=0; i<bufs.length; i+=sz) { ret.push(new Buffer(bufs.slice(i, i+sz).toString('hex'))); } session.output.write(ret); }); </pre>


    Small sample demonstrates that it works:

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">$ coproc2 bufs.js <(echo 'foo123äöüBAR') http://dp3-l3:2227 ; echo 666f6f313233c3a4c3b6c3bc4241520a $ echo 'foo123äöüBAR' | od -tx1 0000000 66 6f 6f 31 32 33 c3 a4 c3 b6 c3 bc 42 41 52 0a 0000020 $ </pre>


    Here you see that processing of big files work (output size is twice the input size):

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">$ ls -l 40M pi-billion.txt 2G -rw-rw-r--. 1 stammw stammw 2000000004 Mar 15 19:17 2G -rw-rw-r--. 1 stammw stammw 40000000 Nov 12 2015 40M -rw-rw-r--. 1 stammw stammw 1000000002 Mar 13 2009 pi-billion.txt $ $ coproc2 bufs.js 2G http://dp3-l3:2227 -s | wc --bytes 4000000008 $ coproc2 bufs.js pi-billion.txt http://dp3-l3:2227 -s | wc --bytes 2000000004 $ coproc2 bufs.js 40M http://dp3-l3:2227 -s | wc --bytes 80000000 $ </pre>


    And her you can see the GatewayScript processing times, determined from extended latency log:

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">2G 36.863s ...,AXF=16996,AGS=53859,... pi 15.916s ...,AXF=8499,AGS=24415,... 40M 0.604s ...,AXF=341,AGS=918,... </pre>


    Just for comparison, since AGS-AXF contains reading from context, processing adn output to context, the processing of a simple "wc --bytes" (word count) on a 3GHz Linux server (not completely comparable since read is from harddisk, and there is no output besides the small statistics):

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">$ time wc 2G 0 1 2000000004 2G real 0m29.726s user 0m29.436s sys 0m0.284s $ </pre>


    Hermann.

     

    P.S:
    "ret.length" is bytelength of Buffers object "ret".

    "ret.buffers.length" is number of "buffer" objects used to build "Buffers" object "ret".

    You can address ith buffer of "ret" via "ret.buffers[i]", but as you can see in above script that is not needed.

    P.P.S:
    Just checked the Buffer sizes of Buffers object bufs returned from readAsBuffers() on input pi-billion.txt:
    the first 976562 are of size 1024, and the last Buffer is of size 514

    This a really nice explication and i want to thank you :-)