Working with the Data Load utility
Mojca 1200008G2Y Comments (2) Visits (11394)
Data Load utility was introduced in WebSphere Commerce v7 and is getting more and more features with every feature pack. It is the recommended, highly customizable, and enhanced business object based loading utility, normally used during some of the most critical development or administrative tasks, such as during an initial setup, during regular operations for most of the WebSphere Commerce implementations, etc.
To load the information into your WebSphere Commerce database, the utility should do it in a single operation, as explained in the "Overview of the Data Load utility" section of the WebSphere Commerce Information Center. So it may be a very stressful moment for a WebSphere Commerce administrator to find out that Data Load is not working, for some of the entries to be loaded, and for no good reason apparently. Of course this might happen to anyone just few days and moments before going live. But here's how to solve this problem.
As an example, the following data needs to be loaded into WebSphere Commerce database for a product's short description, including the double quotation mark ("):
"Business", "casual", and "formal" occasions women's dress
But this results in Data Load utility failing, with an error in the log as following:
SEVERE: Token value is not properly delimited at line 3 for the Input Data File
This error points to the Input Data File, and is referring to the comma separated values (CSV) data reader which can be configured, see "Configuring the CSV data reader", however the default tokenValueDelimiter character is the double quotation mark ("). To use the tokenValueDelimiter character within the token, an administrator must use two tokenValueDelimiter characters, instead of one, and a string separator character is added also in the beginning and in the end of the string.
Thus the above data example should be modified to be used for a catalog entry short description as follows:
"""Business"", ""casual"", and ""formal"" occasions women's dress"
Still, it looks like an easy solution, but this is quite a lot of work for the administrator to figure it out and modify the input data on a daily basis according to the CSV data reader specifications, and when there are millions and millions of records to load, there is no way that the user should be happy to need to modify all the CSV data manually, even though another approach may also be to rectify this issue during data export from an external system preparing the data to be loaded into WebSphere Commerce database. Anyway, sometimes it is not possible and data is received "as-is".
So in that case, it would make much more sense (and make the user's life much easier) to reconfigure the CSV data reader, and set it up so it uses another tokenValueDelimiter character, which is not as likely to appear within the input CSV data to be loaded: