IC SunsetThe developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this forum will no longer be available. More details available on our FAQ.
Topic
  • 15 replies
  • Latest Post - ‏2019-10-18T18:20:55Z by AndrewPaier
Rao_DIID
Rao_DIID
42 Posts

Pinned topic non english characters showing as ???? in IBM BPM 8.5.6 advanced

‏2019-10-07T11:55:54Z | bpm

Hi there,

We have an application which was working just fine on IBM BPM 8.5.6 standard edition running on windows. We imported a snapshot of this application on IBM BPM 8.5.6 advanced edition, running on linux (rest everything remains same) and on new platform non english characters are displayed as ????? (question marks). Encoding on WAS is set to UTF-8.

 

Please advise what can be done to display these non-english characters correctly on IBM BPM 8.5.6 advanced running on linux. Any suggestions/pointers/help is much appreciated.

Thanks,

Rao

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-07T19:21:57Z  

    any help/suggestion on this please, thanks

  • Brajesh2004
    Brajesh2004
    10 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-08T13:24:54Z  
    • Rao_DIID
    • ‏2019-10-07T19:21:57Z

    any help/suggestion on this please, thanks

    Hi

     We did face an issue related to special characters few months back. That was resolved after setting $LANG parameter to appropriate value while restarting JVMs.

    By default system was taking different value whenever server was being restarted. You can check that.

    It seems to me that your issue is related to language parameter not being set properly.

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-08T18:48:42Z  

    thanks Brajesh,

     

    My problem is my application runs in few different locales including English. In English all characters appear fine, but CJK (Chinese/Japanese/Korean) character are shown as ????? (question marks) i.e CJK characters are not encoded/decoded properly.

    Has anyone faced such issue before? Please advise.

     

    Thanks,

    Rao

  • AndrewPaier
    AndrewPaier
    1200 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-09T14:09:09Z  
    • Rao_DIID
    • ‏2019-10-08T18:48:42Z

    thanks Brajesh,

     

    My problem is my application runs in few different locales including English. In English all characters appear fine, but CJK (Chinese/Japanese/Korean) character are shown as ????? (question marks) i.e CJK characters are not encoded/decoded properly.

    Has anyone faced such issue before? Please advise.

     

    Thanks,

    Rao

    I think the community would need a good bit more context to be able to really help you with this.  To start with, what characters are not showing correctly.  For example there is a big difference between the following scenarios - 

    • We are using the internationalization tools to show language appropriate labels in our coaches.  This is working for English and German languages but now failing for Korean, where it worked before.
    • Our users can enter data in different languages.  The when using Roman characters (including accents etc.) the data seems fine upon display but in Chinese it isn't rendering correctly.  We are using (select appropriate) |Heritage Coaches | Coach View based Coaches | Both.
    • Everything in our process execution looks correct but business data in the portal isn't rendering correctly when the business data is entered in Japanese

    It also matters what happens to the data after it is entered.  If it is only kept in the execution context then there is something going on with IBM BPM, but if you are storing it to a System of Record (SOR) then has that configuration changed with this environment change?

     

    Other data needed -

    • What database are you using?  
    • Did that change between your Windows and Linux installs?
    • If you call the REST API for the instances showing ?????? characters is the data correct there?  Is the windows system still around so you can try the same thing there and compare results?

    You have not actually told us what the JVM is set to for its LANG parameter and if that is different between the Windows and Linux environments.  You seem to have assumed that answer is wrong without supplying enough data to rule it out.

     

    -AP

     

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-09T15:15:16Z  

    Hello Andrew,

    many thanks for you kind response,

    First I couldn't see any $LANG parameter on WAS, so not really sure where to check and where to set and to what value in case this needs a change ? Please advise.

    However I ran locale command on OS and here's the output -

    [bpmadmin@ ~]$ locale
    LANG=en_US.UTF-8
    LC_CTYPE="en_US.UTF-8"
    LC_NUMERIC="en_US.UTF-8"
    LC_TIME="en_US.UTF-8"
    LC_COLLATE="en_US.UTF-8"
    LC_MONETARY="en_US.UTF-8"
    LC_MESSAGES="en_US.UTF-8"
    LC_PAPER="en_US.UTF-8"
    LC_NAME="en_US.UTF-8"
    LC_ADDRESS="en_US.UTF-8"
    LC_TELEPHONE="en_US.UTF-8"
    LC_MEASUREMENT="en_US.UTF-8"
    LC_IDENTIFICATION="en_US.UTF-8"
    LC_ALL=

    Regarding the context -

    we have 3 servers - S1/S2/S3, identical configuration and settings on all 3 servers to the best of our knowledge.

    S2/QA environment is working absolutely fine with no issues however DEV and PROD have issues in showing the CJK (Chinese/Japanese/Korean) characters, they are displayed as ???? instead of the actual value. Please note application uses only CJK characters in addition to English. The application code and other integrations remain identical across the servers. We have a web service which returns some data with CJK characters and we can debug and see that everything looks good in the context. few fields from this data ( in CJK characters) are used to set the subject of a human task in the process. While other characters appear okay, CJK characters in the task subject in process portal appear as ????.

     

    There are no interaction to SOR to this point, it's all in the context and displayed as task subject in process portal.

     

    Other data needed -

    • What database are you using?  <<Oracle 11g R2, same across working and non-working environments)
    • Did that change between your Windows and Linux installs? <<No>>
    • If you call the REST API for the instances showing ?????? characters is the data correct there?  Is the windows system still around so you can try the same thing there and compare results? <<Windows system, 8.5.6 standard edition is there and working fine, also the new QA environment is working fine. Trouble is only with DEV and PROD).

    Hope this adds more details to advise further on this please. thanks a million in advance.

     

    Cheers,

    Rao

     

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-10T08:22:31Z  

    Hi there,

    any insights/suggestion on this please ?

    1. where and how can we set $LANG parameter, what value should this be set to ?

     

    we have tried following but the issue still exists -

    a. setting encoding to UTF-8 in encoding.properties file

    b. setting generic JVM arguements as below -

    -Dclient.encoding.override=UTF-8
    -Dfile.encoding=UTF-8

     

    the issue still exists, please help.

     

    Thanks,

    Rao

  • AndrewPaier
    AndrewPaier
    1200 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-10T14:14:43Z  
    • Rao_DIID
    • ‏2019-10-10T08:22:31Z

    Hi there,

    any insights/suggestion on this please ?

    1. where and how can we set $LANG parameter, what value should this be set to ?

     

    we have tried following but the issue still exists -

    a. setting encoding to UTF-8 in encoding.properties file

    b. setting generic JVM arguements as below -

    -Dclient.encoding.override=UTF-8
    -Dfile.encoding=UTF-8

     

    the issue still exists, please help.

     

    Thanks,

    Rao

    Given that the parameter is the same on all 3 environments but one is working and the other 2 aren't I don't think that is the problem. I would suspect that the problem is in the DB configuration for Oracle.  I'm not an Oracle Guru, but I think there is a character set designation for the Oracle Databases.  This infocenter article tells you the settings you need for Oracle.  I would ask a DBA to check the values of those 2 settings in all of the DBs that have been setup and see if they are all matched.  If they are not then I think we have our suspect.

     

    If that doesn't prove out, then we need the answers to the ReST questions above, as that might give us another suspect.  If the problem isn't the DB, we know if really can't be the code, so need the ReST answer to see if the data is okay in the DB but somehow getting "corrupted" in the UI. 

     

    -Andrew Paier

     

    (FYI, I only get a daily digest of posts here, thus the lagged response)

    Updated on 2019-10-10T14:15:25Z at 2019-10-10T14:15:25Z by AndrewPaier
  • Brajesh2004
    Brajesh2004
    10 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-10T14:28:00Z  
    • Rao_DIID
    • ‏2019-10-10T08:22:31Z

    Hi there,

    any insights/suggestion on this please ?

    1. where and how can we set $LANG parameter, what value should this be set to ?

     

    we have tried following but the issue still exists -

    a. setting encoding to UTF-8 in encoding.properties file

    b. setting generic JVM arguements as below -

    -Dclient.encoding.override=UTF-8
    -Dfile.encoding=UTF-8

     

    the issue still exists, please help.

     

    Thanks,

    Rao

    Hi

      LANG is a OS parameter. We faced the issue because JVMs were started via crontab which used to take default LANG value.

    We used to face the issue when $LANG was et to en_US which gave a file.encoding = ANSI_X3.4-1968. Issue was resolved with setting $LANG = en_GB which gave file.encoding = UTF-8.

    In your case i see that file.encoding is set to UTF-8, so your issue seems different.

    Still i would suggest few actions:

    a. Raise a case with IBM, if you have not done so.

    b. Compare properties of servers running properly and having issues.

    c. Do you always face issues or is it intermittent.

     

    Thanks

    Brajesh

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-11T02:13:00Z  

    Given that the parameter is the same on all 3 environments but one is working and the other 2 aren't I don't think that is the problem. I would suspect that the problem is in the DB configuration for Oracle.  I'm not an Oracle Guru, but I think there is a character set designation for the Oracle Databases.  This infocenter article tells you the settings you need for Oracle.  I would ask a DBA to check the values of those 2 settings in all of the DBs that have been setup and see if they are all matched.  If they are not then I think we have our suspect.

     

    If that doesn't prove out, then we need the answers to the ReST questions above, as that might give us another suspect.  If the problem isn't the DB, we know if really can't be the code, so need the ReST answer to see if the data is okay in the DB but somehow getting "corrupted" in the UI. 

     

    -Andrew Paier

     

    (FYI, I only get a daily digest of posts here, thus the lagged response)

    Hello Andrew,

    Many thanks,

    I've checked the DB configurations and this is how they look like -

    DEV:

    AWPSD  : WE8ISO8859P15

    ebu11d  : AL32UTF8

     

    QA:

    AWPWQ  : UTF8

    AWPBQ    : UTF8

    EBU11Q   : AL32UTF8

     

    Prod:

    AWPWP : WE8ISO8859P1

    AWPBP   : WE8ISO8859P1

    EBU11P  : AL32UTF8

     

    do you think there's anything we can update here ? what should be the correct configuration? Please advise.

    Also , what are the REST questions I can answer for this. Please let me know and will share the details accordingly.

     

    Thanks a million, as always.

     

    Cheers,

    Rao

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-11T02:15:30Z  

    Hi

      LANG is a OS parameter. We faced the issue because JVMs were started via crontab which used to take default LANG value.

    We used to face the issue when $LANG was et to en_US which gave a file.encoding = ANSI_X3.4-1968. Issue was resolved with setting $LANG = en_GB which gave file.encoding = UTF-8.

    In your case i see that file.encoding is set to UTF-8, so your issue seems different.

    Still i would suggest few actions:

    a. Raise a case with IBM, if you have not done so.

    b. Compare properties of servers running properly and having issues.

    c. Do you always face issues or is it intermittent.

     

    Thanks

    Brajesh

    Hello Brajesh,

     

    many thanks,

    I've reviewed the properties files and they look consistent across working and non-working environments. Also the issue is consistent on non-working servers, say on Development environment.

     

    Please advise.

     

    Thanks,

    Pramod

  • Christiana Parker
    Christiana Parker
    1 Post

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-11T06:13:39Z  

    IBM provides periodic cumulative fixes for the IBM Business Process Manager products.Consider known issues and limitations for IBM Business Process Manager components. Right now trying to Change AOL Mail Password but got succeed in this by visiting the website.and I am glad you take pride in what you write.

  • AndrewPaier
    AndrewPaier
    1200 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-11T13:56:51Z  
    • Rao_DIID
    • ‏2019-10-11T02:13:00Z

    Hello Andrew,

    Many thanks,

    I've checked the DB configurations and this is how they look like -

    DEV:

    AWPSD  : WE8ISO8859P15

    ebu11d  : AL32UTF8

     

    QA:

    AWPWQ  : UTF8

    AWPBQ    : UTF8

    EBU11Q   : AL32UTF8

     

    Prod:

    AWPWP : WE8ISO8859P1

    AWPBP   : WE8ISO8859P1

    EBU11P  : AL32UTF8

     

    do you think there's anything we can update here ? what should be the correct configuration? Please advise.

    Also , what are the REST questions I can answer for this. Please let me know and will share the details accordingly.

     

    Thanks a million, as always.

     

    Cheers,

    Rao

    Like I said, I'm not an Oracle DBA, and I think now is the time you need to get one involved.  Show them the article I sent to you and the results above.  What I see is that the values for Dev and Prod match but QA is different.  You've also told us that QA is working fine but Dev and Production are not.  That tells me that the configuration of QA is correct and the configuration of Dev / Prod is not.  This is going to require a DBA as you will need someone who had administrative privileges on the Oracle box to make the changes.  

     

    This article seems to indicate that changing the character set on an existing database is a non-trivial exercise.  The community recommendations I saw seemed to recommend dropping and recreating the DB rather than convert.  Note that even after conversion, it is very likely that the data that is showing ???? may have been permanently damaged and will not recover.

     

    -Andrew Paier

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-12T10:12:43Z  

    Hello Andrew,

    Many thanks for this,

    I have got a DBA involved and have requested to configure the non-working database the way working database is configured. I'd update with further details and findings.

    Have a nice weekend!

    Cheers,

    Rao

  • Rao_DIID
    Rao_DIID
    42 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-17T10:43:25Z  

    Hello Andrew,

    Hope you are well,

    we configured Oracle database with Unicode character set and it's working fine now for non-English characters, many thanks for all the help.

     

    However as we discussed earlier these Non-English characters make a part of task subject, so any new tasks that are being created are showing the characters properly however the tasks which were created before the Oracle DB configuration were changed are still showing with ???? characters.

     

    We want the old instances/tasks to be rendered with proper character set, please could you suggest what can be done to done to render the old instances/tasks properly.

    We have tried full server restart so that clears cache and the other bits. Please advise what else can be tried.

     

    Many thanks for all the help.

     

    Thanks,

    Rao

  • AndrewPaier
    AndrewPaier
    1200 Posts

    Re: non english characters showing as ???? in IBM BPM 8.5.6 advanced

    ‏2019-10-18T18:20:55Z  
    • Rao_DIID
    • ‏2019-10-17T10:43:25Z

    Hello Andrew,

    Hope you are well,

    we configured Oracle database with Unicode character set and it's working fine now for non-English characters, many thanks for all the help.

     

    However as we discussed earlier these Non-English characters make a part of task subject, so any new tasks that are being created are showing the characters properly however the tasks which were created before the Oracle DB configuration were changed are still showing with ???? characters.

     

    We want the old instances/tasks to be rendered with proper character set, please could you suggest what can be done to done to render the old instances/tasks properly.

    We have tried full server restart so that clears cache and the other bits. Please advise what else can be tried.

     

    Many thanks for all the help.

     

    Thanks,

    Rao

    Glad the underlying problem has been solved.  

    For the data that is already broken, the bad news is that this is, in fact, data corruption.  Because of how the DB was configured the data got corrupted when you created your tasks.  Perhaps some Oracle DB Guru might know of a way to get back to the original value, but my assumption would be that what effectively happened is that some of the underlying bits/bytes were chopped off and that data is effectively forever gone.

    Long winded way of saying "There is no easy fix from the DB side that I know of".

    If you really need to fix these instances, assuming that the subject is derived from the execution context of the process you can create code that will re-set the task subjects to the correct value, but doing so will take some work.  There is a widget for Services called "Modify Task" that allows you to modify things about a task given its ID (e.g. the due date or subject).  So to make this work, if I had to fix this, I would do the following -

    1. Make a Call to the ReST API or search to pull the bad tasks.  If I really didn't care about maintainability I might query the LSW_TASK table directly pulling tasks where the task is still open and the subject contains a ? character.
    2. For each Task found, use the ReST API to get the full task context
    3. From the context figure out the correct subject.
    4. Use the "Modify Task" widget to set the subject appropriately.

    If there are large numbers of these, then for performance reasons I'd try to pick them up in batches of no more than 500 to avoid jamming too much data into memory.  Additionally in the past I've tested loops in IBM BPM and found that processing an array destructively (while myList.length >0, process item 0, delete item 0 from list) rather than using a for loop.

     

    Sorry that I don't have a quicker fix to offer you.

     

    -AP