Topic
  • 4 replies
  • Latest Post - ‏2012-04-24T10:16:37Z by HermannSW
SystemAdmin
SystemAdmin
6772 Posts

Pinned topic Processing both XML and non-xml MQ payloads in a single MPGW

‏2012-04-23T00:17:28Z |
I have a scenario where over MQ the MPGW receives XMLs from some services and non-xmls from some other services. If XML I have to use xpath to get the meta data and for non-xml meta data is sent in MQRFH2. I couldn't see any obvious match attributes that I could use in the match action, to use different processing rules. I am first looking for MQRFH2 when it's there I am using it otherwise trying to use XPATH but since my request type as non-xml I am not able to use the xpath. Any suggestions to for a solution involving only one service?
Thanks in Advance

Cheers,
Updated on 2012-04-24T10:16:37Z at 2012-04-24T10:16:37Z by HermannSW
  • HermannSW
    HermannSW
    4657 Posts

    Re: Processing both XML and non-xml MQ payloads in a single MPGW

    ‏2012-04-23T12:57:22Z  
    Hi,

    please see isXML.xsl attached to this posting, the discussion on non-maskable log errors, ...:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14768477#14768477

     
    Hermann<myXsltBlog/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Processing both XML and non-xml MQ payloads in a single MPGW

    ‏2012-04-23T21:02:24Z  
    Thanks Hermann, I am now ablw to determine whether it was XML or not. However is there way to surpress the error being logged when I call dp:parse on a non xml?
  • HermannSW
    HermannSW
    4657 Posts

    Re: Processing both XML and non-xml MQ payloads in a single MPGW

    ‏2012-04-24T10:14:05Z  
    Thanks Hermann, I am now ablw to determine whether it was XML or not. However is there way to surpress the error being logged when I call dp:parse on a non xml?
    Hi,

    as stated in the link I provided the dp:parse error is not maskable (and there is an ER open on making this maskable).

    You have two options if you do not want to see that dp:parse error in the logs:
    • Add suppression for code 0x80a002b2 to default log (but then all dp:parse errors do not show up)
    • use alternative detection mechanism described below.

    The first alternative isXML detection mechanism is not using dp:parse to avoid the log error.
    Instead it just looks whether an xml declaration (<?xml ... ?>) is present in input (it has to start at the very 1st byte by the spec).

    Another alternative is to use firts alternative isXML detection
    mechanism first, and only, if that returns true, do the dp:parse
    detection mechanism -- a parse error here is most likely to be correct
    because not so much "Non-XML" files do start with a valid xml declaration.
    The first alternative is not perfect because
    • XML documents are not required to contain a xml declaration
    • you need to be able to handle all needed input encodings

    If you can live with that, and the encodings you need to support are
    all ASCII based (like UTF-8, UTF-16..., ...) then you just need to
    check for "starts-with($input, '<?xml ')" (below stylesheet checks for '<?xml ...?>').
    But even that is not trivial for the different UTF-16 encodings, with and without BOM.

    If you have to support EBCDIC in addition it becomes more difficult.
    I verified that 'var://context/txn/isXML' was set correctly for the
    testfiles (will be attached in next posting) by stylesheet check.xsl
    (attached to this posting:
    
    $ 
    
    for f in testfiles
    /* > do > echo $f > coproc2 check.xsl $f http://nightcrawler:2224 > echo > done testfiles/bom.utf16be.xml ��<?xml version="1.0" encoding="UTF-16"?> <data>123</data> testfiles/bom.utf16le.xml ��<?xml version="1.0" encoding="UTF-16LE"?> <data>123</data> testfiles/bom.utf8.xml <?xml version="1.0" encoding="utf-8"?> <data>123</data> testfiles/nonXML tet testfiles/utf16be.xml <?xml version="1.0" encoding="UTF-16"?> <data>123</data> testfiles/utf16le.xml <?xml version="1.0" encoding="UTF-16LE"?> <data>123</data> testfiles/utf8.xml <?xml version="1.0" encoding="utf-8"?> <data>123</data> $
    


    And this is stylesheet check.xsl, fine of ASCII, UTF-8, UTF-16 and ISO-8859-* encodings:
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" xmlns:regexp=
    "http://exslt.org/regular-expressions" extension-element-prefixes=
    "dp regexp" exclude-result-prefixes=
    "regexp" > <xsl:output method=
    "xml" /> <dp:input-mapping  href=
    "store:///pkcs7-convert-input.ffd" type=
    "ffd"/> <dp:output-mapping href=
    "store:///pkcs7-convert-input.ffd" type=
    "ffd"/>     <!-- <? --> <xsl:variable name=
    "ltqm" select=
    "'3C3F'"/>   <!-- <? (le) --> <xsl:variable name=
    "ltqmle" select=
    "'3C003F00'"/>   <!-- <? (be) --> <xsl:variable name=
    "ltqmbe" select=
    "'003C003F'"/>   <!-- ?> --> <xsl:variable name=
    "qmgt" select=
    "'3F3E'"/>   <!-- ?> (le) --> <xsl:variable name=
    "qmgtle" select=
    "'3F003E00'"/>   <!-- ?> (be) --> <xsl:variable name=
    "qmgtbe" select=
    "'003F003E'"/>   <!-- BOM (utf-8) --> <xsl:variable name=
    "bom" select=
    "'EFBBBF'"/>   <!-- BOM (le) --> <xsl:variable name=
    "bomle" select=
    "'FFFE'"/>   <!-- BOM (be) --> <xsl:variable name=
    "bombe" select=
    "'FEFF'"/>     <!-- any sequence of hexadecimal bytes until $qmgt --> <xsl:variable name=
    "noqmgt" select=
    "'([0-24-9A-F][0-9A-F]|3[0-9A-E]|3F[0-24-9A-F][0-9A-F]|3F3[0-9A-DF])*'"/>   <!-- any sequence of hexadecimal+00 bytes until $qmgtle --> <xsl:variable name=
    "noqmgtle" select=
    "'([0-24-9A-F][0-9A-F]00|3[0-9A-E]00|3F00[0-24-9A-F][0-9A-F]00|3F003[0-9A-DF]00)*'"/>   <!-- any sequence of 00+hexadecimal bytes until $qmgtbe --> <xsl:variable name=
    "noqmgtbe" select=
    "'(00[0-24-9A-F][0-9A-F]|003[0-9A-E]|003F00[0-24-9A-F][0-9A-F]|003f003[0-9A-DF])*'"/>     <!-- match xml declaration --> <xsl:variable name=
    "xmldecl" select=
    "concat('^',$ltqm,$noqmgt,$qmgt)" />   <!-- match bom xml declaration --> <xsl:variable name=
    "bomxmldecl" select=
    "concat('^',$bom,$ltqm,$noqmgt,$qmgt)" />   <!-- match xml declaration (le) --> <xsl:variable name=
    "xmldeclle" select=
    "concat('^',$ltqmle,$noqmgtle,$qmgtle)" />   <!-- match bom xml declaration (le) --> <xsl:variable name=
    "bomxmldeclle" select=
    "concat('^',$bomle,$ltqmle,$noqmgtle,$qmgtle)" />   <!-- match xml declaration (be) --> <xsl:variable name=
    "xmldeclbe" select=
    "concat('^',$ltqmbe,$noqmgtbe,$qmgtbe)" />   <!-- match bom xml declaration (be) --> <xsl:variable name=
    "bomxmldeclbe" select=
    "concat('^',$bombe,$ltqmbe,$noqmgtbe,$qmgtbe)" />     <!-- base64 encoded FFFFFF prefix --> <xsl:variable name=
    "FFFFFF" select=
    "dp:radix-convert('FFFFFF',16,64)" />     <xsl:template match=
    "/"> <xsl:variable name=
    "input64" select=
    "dp:binary-encode(/object/message/node())"/>   <xsl:variable name=
    "input16" select=
    "substring( dp:radix-convert(concat($FFFFFF,$input64),64,16), 7 )
    " />   <object> <message> <!-- look 
    
    for xml declaration in the input msg to see 
    
    if the payload is xml vs bin, so we know what rule to call 
    
    for processing the input. --> <xsl:choose> <xsl:when test=
    "regexp:test($input16,$xmldecl,'')"> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "true()"/> </xsl:when> <xsl:when test=
    "regexp:test($input16,$bomxmldecl,'')"> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "true()"/> </xsl:when> <xsl:when test=
    "regexp:test($input16,$xmldeclle,'')"> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "true()"/> </xsl:when> <xsl:when test=
    "regexp:test($input16,$bomxmldeclle,'')"> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "true()"/> </xsl:when> <xsl:when test=
    "regexp:test($input16,$xmldeclbe,'')"> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "true()"/> </xsl:when> <xsl:when test=
    "regexp:test($input16,$bomxmldeclbe,'')"> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "true()"/> </xsl:when> <xsl:otherwise> <dp:set-variable name=
    "'var://context/txn/isXML'" value=
    "false()"/> </xsl:otherwise> </xsl:choose>   <xsl:copy-of select=
    "dp:binary-decode($input64)"/> </message> </object> </xsl:template> </xsl:stylesheet>
    


     
    Hermann<myXsltBlog/>
  • HermannSW
    HermannSW
    4657 Posts

    Re: Processing both XML and non-xml MQ payloads in a single MPGW

    ‏2012-04-24T10:16:37Z  
    • HermannSW
    • ‏2012-04-24T10:14:05Z
    Hi,

    as stated in the link I provided the dp:parse error is not maskable (and there is an ER open on making this maskable).

    You have two options if you do not want to see that dp:parse error in the logs:
    • Add suppression for code 0x80a002b2 to default log (but then all dp:parse errors do not show up)
    • use alternative detection mechanism described below.

    The first alternative isXML detection mechanism is not using dp:parse to avoid the log error.
    Instead it just looks whether an xml declaration (<?xml ... ?>) is present in input (it has to start at the very 1st byte by the spec).

    Another alternative is to use firts alternative isXML detection
    mechanism first, and only, if that returns true, do the dp:parse
    detection mechanism -- a parse error here is most likely to be correct
    because not so much "Non-XML" files do start with a valid xml declaration.
    The first alternative is not perfect because
    • XML documents are not required to contain a xml declaration
    • you need to be able to handle all needed input encodings

    If you can live with that, and the encodings you need to support are
    all ASCII based (like UTF-8, UTF-16..., ...) then you just need to
    check for "starts-with($input, '<?xml ')" (below stylesheet checks for '<?xml ...?>').
    But even that is not trivial for the different UTF-16 encodings, with and without BOM.

    If you have to support EBCDIC in addition it becomes more difficult.
    I verified that 'var://context/txn/isXML' was set correctly for the
    testfiles (will be attached in next posting) by stylesheet check.xsl
    (attached to this posting:
    <pre class="jive-pre"> $ for f in testfiles /* > do > echo $f > coproc2 check.xsl $f http://nightcrawler:2224 > echo > done testfiles/bom.utf16be.xml ��<?xml version="1.0" encoding="UTF-16"?> <data>123</data> testfiles/bom.utf16le.xml ��<?xml version="1.0" encoding="UTF-16LE"?> <data>123</data> testfiles/bom.utf8.xml <?xml version="1.0" encoding="utf-8"?> <data>123</data> testfiles/nonXML tet testfiles/utf16be.xml <?xml version="1.0" encoding="UTF-16"?> <data>123</data> testfiles/utf16le.xml <?xml version="1.0" encoding="UTF-16LE"?> <data>123</data> testfiles/utf8.xml <?xml version="1.0" encoding="utf-8"?> <data>123</data> $ </pre>

    And this is stylesheet check.xsl, fine of ASCII, UTF-8, UTF-16 and ISO-8859-* encodings:
    <pre class="jive-pre"> <xsl:stylesheet version= "1.0" xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" xmlns:regexp= "http://exslt.org/regular-expressions" extension-element-prefixes= "dp regexp" exclude-result-prefixes= "regexp" > <xsl:output method= "xml" /> <dp:input-mapping href= "store:///pkcs7-convert-input.ffd" type= "ffd"/> <dp:output-mapping href= "store:///pkcs7-convert-input.ffd" type= "ffd"/> <!-- <? --> <xsl:variable name= "ltqm" select= "'3C3F'"/> <!-- <? (le) --> <xsl:variable name= "ltqmle" select= "'3C003F00'"/> <!-- <? (be) --> <xsl:variable name= "ltqmbe" select= "'003C003F'"/> <!-- ?> --> <xsl:variable name= "qmgt" select= "'3F3E'"/> <!-- ?> (le) --> <xsl:variable name= "qmgtle" select= "'3F003E00'"/> <!-- ?> (be) --> <xsl:variable name= "qmgtbe" select= "'003F003E'"/> <!-- BOM (utf-8) --> <xsl:variable name= "bom" select= "'EFBBBF'"/> <!-- BOM (le) --> <xsl:variable name= "bomle" select= "'FFFE'"/> <!-- BOM (be) --> <xsl:variable name= "bombe" select= "'FEFF'"/> <!-- any sequence of hexadecimal bytes until $qmgt --> <xsl:variable name= "noqmgt" select= "'([0-24-9A-F][0-9A-F]|3[0-9A-E]|3F[0-24-9A-F][0-9A-F]|3F3[0-9A-DF])*'"/> <!-- any sequence of hexadecimal+00 bytes until $qmgtle --> <xsl:variable name= "noqmgtle" select= "'([0-24-9A-F][0-9A-F]00|3[0-9A-E]00|3F00[0-24-9A-F][0-9A-F]00|3F003[0-9A-DF]00)*'"/> <!-- any sequence of 00+hexadecimal bytes until $qmgtbe --> <xsl:variable name= "noqmgtbe" select= "'(00[0-24-9A-F][0-9A-F]|003[0-9A-E]|003F00[0-24-9A-F][0-9A-F]|003f003[0-9A-DF])*'"/> <!-- match xml declaration --> <xsl:variable name= "xmldecl" select= "concat('^',$ltqm,$noqmgt,$qmgt)" /> <!-- match bom xml declaration --> <xsl:variable name= "bomxmldecl" select= "concat('^',$bom,$ltqm,$noqmgt,$qmgt)" /> <!-- match xml declaration (le) --> <xsl:variable name= "xmldeclle" select= "concat('^',$ltqmle,$noqmgtle,$qmgtle)" /> <!-- match bom xml declaration (le) --> <xsl:variable name= "bomxmldeclle" select= "concat('^',$bomle,$ltqmle,$noqmgtle,$qmgtle)" /> <!-- match xml declaration (be) --> <xsl:variable name= "xmldeclbe" select= "concat('^',$ltqmbe,$noqmgtbe,$qmgtbe)" /> <!-- match bom xml declaration (be) --> <xsl:variable name= "bomxmldeclbe" select= "concat('^',$bombe,$ltqmbe,$noqmgtbe,$qmgtbe)" /> <!-- base64 encoded FFFFFF prefix --> <xsl:variable name= "FFFFFF" select= "dp:radix-convert('FFFFFF',16,64)" /> <xsl:template match= "/"> <xsl:variable name= "input64" select= "dp:binary-encode(/object/message/node())"/> <xsl:variable name= "input16" select= "substring( dp:radix-convert(concat($FFFFFF,$input64),64,16), 7 ) " /> <object> <message> <!-- look for xml declaration in the input msg to see if the payload is xml vs bin, so we know what rule to call for processing the input. --> <xsl:choose> <xsl:when test= "regexp:test($input16,$xmldecl,'')"> <dp:set-variable name= "'var://context/txn/isXML'" value= "true()"/> </xsl:when> <xsl:when test= "regexp:test($input16,$bomxmldecl,'')"> <dp:set-variable name= "'var://context/txn/isXML'" value= "true()"/> </xsl:when> <xsl:when test= "regexp:test($input16,$xmldeclle,'')"> <dp:set-variable name= "'var://context/txn/isXML'" value= "true()"/> </xsl:when> <xsl:when test= "regexp:test($input16,$bomxmldeclle,'')"> <dp:set-variable name= "'var://context/txn/isXML'" value= "true()"/> </xsl:when> <xsl:when test= "regexp:test($input16,$xmldeclbe,'')"> <dp:set-variable name= "'var://context/txn/isXML'" value= "true()"/> </xsl:when> <xsl:when test= "regexp:test($input16,$bomxmldeclbe,'')"> <dp:set-variable name= "'var://context/txn/isXML'" value= "true()"/> </xsl:when> <xsl:otherwise> <dp:set-variable name= "'var://context/txn/isXML'" value= "false()"/> </xsl:otherwise> </xsl:choose> <xsl:copy-of select= "dp:binary-decode($input64)"/> </message> </object> </xsl:template> </xsl:stylesheet> </pre>

     
    Hermann<myXsltBlog/>
    testfiles.zip