Topic
  • 3 replies
  • Latest Post - ‏2014-02-16T23:39:25Z by DonaldN
JohnL
JohnL
4 Posts

Pinned topic Code page conversion at field level to allow cut/paste of emails

‏2014-02-11T23:11:09Z |

I have a CQ database using ISO Latin 1 code page and I get a lot of complains from people who try to cut/paste emails into one or two multiline text fields.   I looked at switching the CQ schema and backend Oracle database to UTF8 but this is not supported by Rational for existing schemas/databases.

So my other thought was to make add some form of validation hook code which would sanitize/convert the pasted data to ISO Latin 1 code page.   How have others worked around this issue?   Does anyone have hook code to do the sanitization/conversion?

TIA,

john

  • DonaldN
    DonaldN
    287 Posts

    Re: Code page conversion at field level to allow cut/paste of emails

    ‏2014-02-11T23:42:28Z  

    Hi John,

    I'm against the idea of sanitizing because it will be difficult to maintain the quality of the text.

    If you understand the reason why converting an existing schema/user database is unsupported, you will have more options available. This is due to the multibyte nature of the UTF-8 coding and it is very likely the conversion will end up failed. For example, let's say we have a short_string "title" field which is 50 characters long (or 50 bytes to be exact), and it is filled with a 50-byte string. When this string is transcoded to UTF-8, the length may become 51 or even more (particularly true for Chinese, Japanese and other Asian languages). The end result will be either the database refuses to fit the converted string or the string is truncated (data loss). In saying that, if the multiline_string is the main concern, it will be much easier to deal with since the backing CLOB type does not have such a tight restriction.

    Seeing that the original coding is ISO Latin 1, you may be lucky with a successful conversion. I would suggest you do a "move", meaning that you create a new Oracle database with the desired character set first and then "move" the existing ClearQuest database over. It is quite safe because you can revert back to your original database if needed.

    ClearQuest has some built-in tools to help you in this area.

    http://pic.dhe.ibm.com/infocenter/cqhelp/v8r0m0/topic/com.ibm.rational.clearquest.admin.doc/topics/codepage/c_working_w_codepages.htm

    http://pic.dhe.ibm.com/infocenter/cqhelp/v8r0m0/topic/com.ibm.rational.clearquest.admin.doc/topics/t_move_user_db._w_dsgnr.htm

    http://pic.dhe.ibm.com/infocenter/cqhelp/v8r0m0/topic/com.ibm.rational.clearquest.admin.doc/topics/t_mv_sch_repository_usng_mt.htm

     

  • JohnL
    JohnL
    4 Posts

    Re: Code page conversion at field level to allow cut/paste of emails

    ‏2014-02-14T23:13:23Z  
    • DonaldN
    • ‏2014-02-11T23:42:28Z

    Hi John,

    I'm against the idea of sanitizing because it will be difficult to maintain the quality of the text.

    If you understand the reason why converting an existing schema/user database is unsupported, you will have more options available. This is due to the multibyte nature of the UTF-8 coding and it is very likely the conversion will end up failed. For example, let's say we have a short_string "title" field which is 50 characters long (or 50 bytes to be exact), and it is filled with a 50-byte string. When this string is transcoded to UTF-8, the length may become 51 or even more (particularly true for Chinese, Japanese and other Asian languages). The end result will be either the database refuses to fit the converted string or the string is truncated (data loss). In saying that, if the multiline_string is the main concern, it will be much easier to deal with since the backing CLOB type does not have such a tight restriction.

    Seeing that the original coding is ISO Latin 1, you may be lucky with a successful conversion. I would suggest you do a "move", meaning that you create a new Oracle database with the desired character set first and then "move" the existing ClearQuest database over. It is quite safe because you can revert back to your original database if needed.

    ClearQuest has some built-in tools to help you in this area.

    http://pic.dhe.ibm.com/infocenter/cqhelp/v8r0m0/topic/com.ibm.rational.clearquest.admin.doc/topics/codepage/c_working_w_codepages.htm

    http://pic.dhe.ibm.com/infocenter/cqhelp/v8r0m0/topic/com.ibm.rational.clearquest.admin.doc/topics/t_move_user_db._w_dsgnr.htm

    http://pic.dhe.ibm.com/infocenter/cqhelp/v8r0m0/topic/com.ibm.rational.clearquest.admin.doc/topics/t_mv_sch_repository_usng_mt.htm

     

    Have you done this conversion like you described before? 

  • DonaldN
    DonaldN
    287 Posts

    Re: Code page conversion at field level to allow cut/paste of emails

    ‏2014-02-16T23:39:25Z  
    • JohnL
    • ‏2014-02-14T23:13:23Z

    Have you done this conversion like you described before? 

    Hi John,

    No I have not done it myself. What I know is that the only change made to the source database during the "move" is that it will be locked, and that's it. No other data, particularly user data will be altered.

    Of course you can do an in-place conversion if you can take a full backup of the Oracle database for the purpose of rollback. Since the IBM documents clearly state that converting an existing database is not supported, you may find yourself in an unsupported situation if something goes wrong with the in-place conversion.

    I suggest the conversion purely because it is reversible and worth a try.