Topic
  • 9 replies
  • Latest Post - ‏2013-10-21T19:59:59Z by Greenspan
Greenspan
Greenspan
23 Posts

Pinned topic UTF-8 on zOS

‏2013-05-08T13:39:36Z |

Hi ,

I am working on a project where we use WTX on zOS. We basically process files and feed the output to various downstream system. We build the maps on PC and port them to zOS via FTP and use DTXCMDSV to run them with overriding parametres . Now we are dealing with lot of files where UTF8 encoding is used. If the in put files use basic latin characters in UTF8 then there is no problem in mapping. as each character is mapped as one byte . However when the files have any other characters (Thai/Japanese/Korean) , each characters occupy multiple bytes(usually 3). So when we have to map say 40 characters MID function would give me 40 english characters but only 13 Thai characters and an extra byte that is not in any character set. What I want is some way the rule should map 40 character english as well as thai.

I am aware this works well on windows. If we make the Data language as UTF8. However it does not work if we port it to zOS.

Is there a way to handle this ?

 I have been searching this but could not get anything .

Edit:added some more details

Updated on 2013-05-13T04:37:38Z at 2013-05-13T04:37:38Z by Greenspan
  • Greenspan
    Greenspan
    23 Posts
    ACCEPTED ANSWER

    Re: UTF-8 on zOS

    ‏2013-10-21T19:59:59Z  
    • Greenspan
    • ‏2013-05-31T12:50:35Z

    I think your input is in ASCII. This is what allows the map to work corerctly in UTF8 characters. I tried with manually creating a file with ASCII hex values on Mainframe and it also works (yay !!). Although my original question reamains , still your help has given me one way to solve this. Many thanks for that. Also DISPLAY UTF8 does not work on our system. So I guess it is some system specific CLIST that your sys programmers has developed.

     

    I raised it with IBm and they confirmed that UTF8 does infact work even on Mainframe. However the hex values on zOS should match that of PC. In our system there is some kind of translation that prevents this from happening. So the solution is translate the characters back to UTF8 and then process via Map. Not elegant but it does answer the question.

  • Greenspan
    Greenspan
    23 Posts

    Re: UTF-8 on zOS

    ‏2013-05-23T06:29:20Z  

    Any one has any insights ?

  • RihRi
    RihRi
    7 Posts

    Re: UTF-8 on zOS

    ‏2013-05-28T06:04:12Z  

    Hello,

    How has you defined your fields size in TypeTree, bytes or characters ?

    When I have files which are UTF-8 in z/os, I define size as characters in TypeTree and it has worked fine.

    -- Riitta --  

  • Greenspan
    Greenspan
    23 Posts

    Re: UTF-8 on zOS

    ‏2013-05-30T05:02:09Z  
    • RihRi
    • ‏2013-05-28T06:04:12Z

    Hello,

    How has you defined your fields size in TypeTree, bytes or characters ?

    When I have files which are UTF-8 in z/os, I define size as characters in TypeTree and it has worked fine.

    -- Riitta --  

    Hi Riitta ,

     

    My input typetree has min length 1 byte and character and max 420 bytes and 140 characters. Output has no max or min. Both types are text and national language as western and UTF8. My desgn studion is 8.4.0.2 and zos command server  version is 8.4.0.3.

     

    What version are you running with ? and in your input what language characters do you have ?

    Thanks

    GS

  • RihRi
    RihRi
    7 Posts

    Re: UTF-8 on zOS

    ‏2013-05-30T05:36:05Z  

    Hello,

    I am running Design Studio and zos command server both version 8.3.0.4.

    I have National Language as Western and DataLanguage as UTF-8. In field size I have size only in characters and I do have size for both input and output.

    -- Riitta --  

  • Greenspan
    Greenspan
    23 Posts

    Re: UTF-8 on zOS

    ‏2013-05-30T10:20:05Z  
    • RihRi
    • ‏2013-05-30T05:36:05Z

    Hello,

    I am running Design Studio and zos command server both version 8.3.0.4.

    I have National Language as Western and DataLanguage as UTF-8. In field size I have size only in characters and I do have size for both input and output.

    -- Riitta --  

    So my design studio is is one version lower. That could be one the reasons it is different. One more question, what language characters are you using for the input. Can I get a screenshot/example  of a sample input and output on zOS ?

     

    many thanks

     

    GS

  • RihRi
    RihRi
    7 Posts

    Re: UTF-8 on zOS

    ‏2013-05-31T07:47:23Z  
    • Greenspan
    • ‏2013-05-30T10:20:05Z

    So my design studio is is one version lower. That could be one the reasons it is different. One more question, what language characters are you using for the input. Can I get a screenshot/example  of a sample input and output on zOS ?

     

    many thanks

     

    GS

    Actually your DesignStudio is one versio higher !! You have 8.4  and I have 8.3.

    My input is UTF-8 so it has NationalLanguage as Western and DataLanguae as UTF-8. 

    Picture named Input UTF8 shows my input when I do normal browse to zos-file. Picture named Input display shows part of my input after I gave zos-command DISPLAY UTF8. At bottom row you can see nordic letters ÅÖäöÄ, which all take two bytes in UTF8. I have tested this with that field.

    Picture named output shows my test output. First 5 character is  the input field as UTF8 and next 5 characters is input field as EBCDIC. I have tested this  by using MID-function MID(input_field,1,5) and without MID-function (straight drag and drop from input to output) and both gave  the same result.

    I don't have any Thai letters, only these Nordic letter, so maybe that is the difference between our cases.

    Good luck to you with this problem

    -- Riitta --

     

  • Greenspan
    Greenspan
    23 Posts

    Re: UTF-8 on zOS

    ‏2013-05-31T12:50:35Z  
    • RihRi
    • ‏2013-05-31T07:47:23Z

    Actually your DesignStudio is one versio higher !! You have 8.4  and I have 8.3.

    My input is UTF-8 so it has NationalLanguage as Western and DataLanguae as UTF-8. 

    Picture named Input UTF8 shows my input when I do normal browse to zos-file. Picture named Input display shows part of my input after I gave zos-command DISPLAY UTF8. At bottom row you can see nordic letters ÅÖäöÄ, which all take two bytes in UTF8. I have tested this with that field.

    Picture named output shows my test output. First 5 character is  the input field as UTF8 and next 5 characters is input field as EBCDIC. I have tested this  by using MID-function MID(input_field,1,5) and without MID-function (straight drag and drop from input to output) and both gave  the same result.

    I don't have any Thai letters, only these Nordic letter, so maybe that is the difference between our cases.

    Good luck to you with this problem

    -- Riitta --

     

    I think your input is in ASCII. This is what allows the map to work corerctly in UTF8 characters. I tried with manually creating a file with ASCII hex values on Mainframe and it also works (yay !!). Although my original question reamains , still your help has given me one way to solve this. Many thanks for that. Also DISPLAY UTF8 does not work on our system. So I guess it is some system specific CLIST that your sys programmers has developed.

  • Debnarayan
    Debnarayan
    17 Posts

    Re: UTF-8 on zOS

    ‏2013-07-11T06:47:45Z  

    Hello,

     

    I am also working on handling multi byte characters. Could you please help me find a fast and secure way of converting type tree element size property from Bytes to Characters?

     

     

    Thanks,

    Debnarayan

  • Greenspan
    Greenspan
    23 Posts

    Re: UTF-8 on zOS

    ‏2013-10-21T19:59:59Z  
    • Greenspan
    • ‏2013-05-31T12:50:35Z

    I think your input is in ASCII. This is what allows the map to work corerctly in UTF8 characters. I tried with manually creating a file with ASCII hex values on Mainframe and it also works (yay !!). Although my original question reamains , still your help has given me one way to solve this. Many thanks for that. Also DISPLAY UTF8 does not work on our system. So I guess it is some system specific CLIST that your sys programmers has developed.

     

    I raised it with IBm and they confirmed that UTF8 does infact work even on Mainframe. However the hex values on zOS should match that of PC. In our system there is some kind of translation that prevents this from happening. So the solution is translate the characters back to UTF8 and then process via Map. Not elegant but it does answer the question.