Modified on by Shweta Gupta IBM
In the PART1 of this series, I introduced the topic of security around eCommerce from a perspective of being in a Hot Air Balloon (which would be 2000 lifetime floor badge if you wear a Fitbit). You will want to read the Part 1 to get a glimpse of CISO’s role in your organization’s security, and introduction on IBM’s security services. However, if you want to directly dive into securing your WebSphere Commerce application, read on this article.
The WebSphere Commerce security model constitutes of setting up the access control, through authentication, authorization and policies. It provides through some of the main security standards applicable to the eCommerce industry. It also covers managing against common attack types. However, as outlined in the PART 1, the malicious attacks continuously innovate themselves, and therefore business needs to evaluate its advanced threat management capability on an ongoing basis.
In this PART2 of the series, I am going to cover 4 areas from the application perspective.
- Access control
- Hardening against common attacks types
- Data security
- Security Standards
Authentication: WebSphere Commerce issues an authentication token. This token is associated with a user on every subsequent request after the authentication process. There are account policies for password, account lockout, timeout that govern authentication.
Authorization: Access control policies is leveraged for enforcing authorization in Commerce.
Let us look at some some common access control areas that are prominent from en eCommerce perspective.
- Logon - Enforcing too many failed attempts: Set up an account lockout policy for different user roles in WebSphere Commerce. This is to enable enforcement of a lockout in case there is malicious attempt to break a password. There is also deterrents by setting up delays in between of consecutive unsuccessful attempts.
- Logon - Prevent privileged users from logging in externally: It is desirable to dis-allow access to the privileged users of WebSphere Commerce application, like the site administrator or a customer service representative. This need is primarily from a security perspective where one would not want a hacker to gain privileged access and act with malicious intent. WebSphere Commerce V 8 Mod Pack 1 has provided with a security configuration in wc-server.xml that can be enabled and then customer headers can be used to define roles to disallow the logon. There is further customization that one can do to address the security needs for logon. Earlier to this introduction of the feature, in earlier version of WebSphere Commerce (v7, v8), the article provides sample code to achieve the functionality.
- Securing the WebSphere Commerce search server: This can be achieved at two levels, WebSphere Application Server (WAS) Administrative Security or WebSphere Application Server (WAS) Application Security. The WAS Admin Administrative Security is recommended, as the application security is more expensive in terms of performance.
Protecting custom enterprise beans, data beans, controller commands, views: Primary resources should be protected. If a user is allowed to access a primary resource, the user would be allowed to access its dependent resources.
Access Control and Data Beans - The following article discusses the best practices for access control with WebSphere Commerce and how to use Data Beans securely
Hardening against common attacks types - A view on the common attacks types and guidance on handling them
- Changing URL params to break the system: WhiteList data validation on a URL is disabled by default. This restricts the running of URL such that it only allows processing of URLs that conform to the regular expression. This means you could restrict any malicious intent on the site by changing the params on the URL for store-id, catalog-id etc...
- Cross-site scripting protection, is enabled by default for the Stores web module. It is important that all the parameters are encoded by developers during the development phase and tested thoroughly. The prohibitedChars rule has been provided by default, however it cannot be comprehensive. The blacklist is used as fall back for cases when the store is not encoding the output properly (c:out). The problem with blacklists is that hackers keep finding ways of bypassing them by using different encodings and escaping characters. It's not possible to come up with a completely robust blacklist solution ( OWASP calls blacklist "fragile" ). This is more complex to apply to the REST body. It is recommended that there is a review on the OWASP Cross-site Scripting site for project implementation. The project management and security architect should provide guidelines on preventing, protecting and testing against Cross-site scripting.
- Mitigating threat of denial of service attacks: It is recommended to set the boundary for the product search results by specifying maximum page size and result size. Similarly, one can set allowable ranges for other business objects like the shopping cart.
- Enable ClickJacking Protection – Clickjacing is when an attacker is able to trick you into believing that you are on a particular site/page, while re-directing you to do something else. This can be achieved via several techniques, however we will focus on the mechanism where iFrames are exploited to overlay your desired site. The article talks about the X-Frame options and Content Security Policy.
- WebSphere Commerce database encryption to a stronger standard to reduce the chances of a successful brute force attack by Migrating from Triple DES to AES-128 encryption
- Data security practices for integrating eCommerce system with Order Management System
1 - Use of Cryptographic Algorithms and Key Lengths: NIST announced its Special Publication (SP) 800-131A in 2011, which recommends transitioning of the use of Cryptographic Algorithms and Key Lengths.
Standard requires adherence to
- Digital signatures must use at least SHA-2 hashing algorithm, and WebSphere Commerce v8 uses SHA-2
- Cryptographic keys adhere to a minimum key strength of 112 bits. WebSphere Commerce provides a detailed procedure for the migration of encrypted data in the database to use AES 128-bit encryption.
- Enable TLS 1.2 for SSL and disable protocols less than TLS 1.2 for WebServer, any integrations with LDAP, outbound email. WebSphere Application server must adhere to TLS 1.2
- All certificates with RSA or DSA keys must be 2048 bits or higher. Certificates with elliptic curve keys shorter than 160 bits must be replaced with longer keys. All certificates must be signed by an allowed signature algorithm. For example, SHA-256, SHA-384, or SHA-512
2 - Federal Information Processing Standards publication 140-2 (FIPS 140-2) covers the security standards that are required for cryptographic modules. The WebSphere Commerce documents the steps that are needed for commerce to runn on a WebSphere Application Server and on HTTP servers that are in FIPS 140-2 mode.
3 - The Payment Card Industry (PCI) Data Security Standard (DSS) - The PCI DSS Version 3.0 standard lists 12 requirements which retailers, online merchants, credit data processors, and other payment-related businesses must implement to help protect cardholders and their data. The requirements include technology controls (such as data encryption, user access control, and activity monitoring) and required procedures. This is required to be implemented when the cardholder data resides with the business. The WebSphere Commerce Knowledge Center documents the procedure ONLY for the WebSphere Commerce application to be PCI compliant, while there are many other aspects that the business needs to take care as part of the compliance.
The security of eCommerce is a broad topic that requires the Chief Information Security Officer (CISO) at the organization to work at various levels. This was covered in the PART 1
This article is an attempt to help the reader get a glimpse on the application security aspects for a commerce site. I have tried to categorize security from a view of access, vulnerabilities and data protection and help with a capability view on WebSphere Commerce.
I have also created a template spreadsheet that one can refer in their project as a starting point to track the various headings, status and desired status. This template is to be enhanced/tailored to suit your security posture.
2016-July-WCS-Security-Template-v1.xlsxView Details for Project Managers to capture actions. Do provide your feedback if you use it to enhance it further.
Modified on by Shweta Gupta IBM
Who owns and understands the security posture for the organization?
I would start with making an assumption that if you were a WebSphere Commerce architect, specialist, or a developer, you will agree with me that it will be foolish to attempt to respond to the loaded question – ‘Is our eCommerce site safe?’
This would be something that is under a purview of a Chief Information Security Officer (CISO). If you are interested to see a sample of the job responsibilities of a CISO profile in a commerce organization, have a look at this linkedin entry
The infographics on cyber-attacks from Threat Intelligent Report
IBM X-Force Threat Intelligent Report published by IBM, shows the 2015 security incidents and the retail industry is the second most attacked industry.
The components of a commerce shop that would typically need a CISO’s attention
There are several areas of a commerce shop that require to be at a desired level of security maturity model.
- To start with there is a need to have a Governance structure in place. This would include the processes established inorder to manage and report on the compliance
- Focus on the People aspect of the business, that would talk about identity and access management
- Infrastructure security would cover several pieces, networks, servers and the endpoints
- Protection of Data is key and this may entail industry standards and compliance
- Application security will include all the different applications and the security levels would be different based on their exposure to the internet
How does IBM help companies and businesses to be secure, and this would apply to eCommerce
IBM has a strong security product portfolio, business and product offerings. There is also an online set of Quiz Questions to help an organization evaluate their understanding of the security risk.
The following areas of focus, will help the security stakeholders to engage with IBM
Your organization’s security program - Understanding risks, establishing the right policies and programs and having a strong, cohesive team to implement changes is critical to building a next-generation security environment. IBM® Security can help you design an integrated framework to simplify the challenge of securely protecting the enterprise.
Tackle Advanced Threats – You are uncertain on the advanced threats and how to be prepared to manage them. IBM® Security can help you with a view the security landscape with a wide-angle lens to thoroughly understand the origins and distinctive features of attackers. Fraud, endpoint and data protection. Security intelligence and analytics. Incident response. They're all part of a comprehensive approach using extensive research and detective work to pinpoint, outsmart and stop them.
Protect critical assets – You have an internet facing business, with potentially millions of customers, partners, vendors accessing the systems. IBM® Security can help you leverage analytics and insight for rapid detection and response to protect critical systems and records.
Further, IBM provides services where the security experts will do an assessment of all the aspects of an organization from a security maturity model. This will allow the CISO to understand the security posture for the organization and the desired posture for each of the components mentioned above, like people, data, and infrastructure. This is an important step in securing organization and business against the cyber security threats.
Security of the WebSphere Commerce empowered storefront
The CISO’s governance and strategy will form the backbone of the eCommerce security. This will ensure that each of the applications follow the guidelines and demonstrate adherence to the security framework.
The eCommerce program manager will own the security aspects of the WebSphere Commerce implementation. It will therefore be the individual responsibility of every member involved in the implementation to understand the application guidelines for WebSphere Commerce and consider them at design, development, testing and deployment phases.
The product KnowledgeCenter has a topic in itself on ‘securing’, that details on the topics of authentication, authorization, session management. There is also section to cover security standards ‘National Institute of Standards and Technology’ (NIST) that provides guidance on the use of stronger cryptographic keys.
The Payment Card Industry (PCI) Data Security Standard (DSS) is applicable if your eCommerce system is capturing and holding credit card and payment-related data into the system. The product has created a summary of specific configuration that are required in the WebSphere Commerce implementation in order to comply with the PCI-DSS.
Having pointed the readers to the Knowledge Center for a lot of product documentation on the topic, I am also doing a PART 2 of this series to cover a simplified version of the aspects of the WebSphere Commerce product’s security considerations. Look out for the second part for a quick and clear insight areas of Access control, Hardening against common attacks types and Data security.
Credit: Thank You Sreekanth for your inputs and review on the topic.
Sreekanth Iyer is an Executive Architect with the IBM Cloud (CTO Office) team and works on defining the technical strategy and development of IBM Cloud Security. He has over 20 years of industry experience and has led several client solutions for Telco, Electronics, E&U, Govt, BPO & Banking industries. He is an Open Group Certified Distinguished Architect, IBM Master Inventor, Certified Ethical Hacker and Member of IBM Academy of Technology.
Modified on by Shweta Gupta IBM
The shoppers have loved the search bar on the shopping sites and come to expect them to be their first friend to take them close to the product they intend to shop on the site. This is akin to the quintessential ‘Shopping Assistants’ in the marts, supermarkets, our our big or small fashion outlets. The shop may be earmarked and categorized, however if we are not familiar that particular store, we tend to get awed by the array of aisles and have an instant liking towards the store assistant who can either guide you the aisle/corner of the store for your desired product or even better escort you to the product that you are seeking. You would agree that some of these assistants have converted how many bags you carried back home from your shopping expedition.
The current state of eCommerce has come a long way and with most search engines offering a set of standard technical capabilities, one would expect to find our shopping search experiences almost similar. However, we are actually at length from this experience. While I was looking for studies on the subject, I stumbled upon a 2014 report on smashingmagazine.com, where they benchmarked the search experience of the 50 top-grossing US e-commerce websites, rating each website across a set of 60 search usability parameters, which you would agree is rather very fine grained. Offcourse a study from 2014 would require a revisit in 2016 at the pace one would expect our eCommerce sites to launch capabilities in an agile way, to stay competitive.
What does this mean to you for as eCommerce Implementation Project Manager?
I have observed that even though search is an important functional feature on eCommerce sites, the search relevance testing is continued to be missed out of the plan. The traditional functional testing, includes search and browse feature test, where the test team is responsible for a set of use-cases for checking that the search functionality works as expected. However, this approach largely misses out on the desired shopper experience for the search results or/and the desired search results from the business point of view. The User Acceptance Testing (UAT) phase would typically have the business users look at search relevancy, however it remains more of an organic exercise, amongst the overall umbrella of UAT, not getting the due attention, time, effort and most importantly understanding of the approach, input and outcome.
What does this mean to you as the eCommerce Product Leadership for your business?
This is definite a very continual exercise for you, just as it is to deliver on the business requirements via your eCommerce and mCommerce channels. You will be doing analysis at the ‘search hits’ and ‘search misses’, hopefully on a weekly basis, and working towards tuning your results based on the data your shoppers are feeding into you. The strong point is that because you have invested into WebSphere Commerce, you will find that the Business Tools allow you with a lot of flexibility and ease to achieve your results.
Let us explore an approach towards search relevance, look at a proposal for the test strategy and guidelines for your WebSphere Commerce Search powered commerce site.
Outlining the search relevance testing proposal and approach
The Project Management of eCommerce sites must include a sprint (or more depending upon the catalog size) which focuses on the search relevance. I have had discussions with several test managers on the topic and found them wanting for guidance around the ‘search relevance testing’. This prompted me to create an approach document that can be used as starting guideline for your eCommerce site.
- Your list of search terms: Work with the business to identify a list of ‘n’ search terms which you would like to focus in your initial iterations. Where n could be 50, 100, 200 so on based on the width and depth of the catalog, product items.
- Variety in search terms: The search terms should include a variety, where you have single words, multi-words, words with units like weight (think grocery), phrases with attributes like color (think garments), variance in how shoppers search for certain products, brands which your catalog does not support, misspelled products …
- Get into the shopper’s shoes, sandals: Leverage analytics data from different sources to identify the changing trends and shoppers behavior. Go out there to your favorite, and not so favorite retailers with similar catalogs, and learn what you like and what you would wish did better.
- Product title/description recommendations: The quality of the search result highly depends upon the way the catalog data is created and maintained. The natural search results from the search engine is based on the query relevance as defined in the configuration. One of the important recommendation which would also come out of the relevance activity is on enriching the product descriptions and other searchable attributes.
- Tester training: Your army of testers may not be trained to think on the topic of search relevance. Plan for a training and get them to think and experience shopping around the domain. This may be simpler for certain industries like grocery and fashion, however this may slightly trickier incase your site is selling spare parts, or heavy industry tools.
- Search Developers: The search developer needs to be part of the iteration to support the relevance tuning. This will require to look at the relevancy scores, and providing analysis on the result. The developer will also be responsible for making any changes which come through search configuration changes.
- Inputs to the search relevance activity: At the start of the search activity, you will have the list of search terms, expected results, synonyms, and search replacement terms.
- Outputs of the search relevance activity: At the end of the search relevance activity, you will list of search terms, actual result-set, if it meets expected results and no what is the delta. You will also have a more expanded/modified list of synonyms, search replacement terms and search rules.
- Iterative: This exercise should be iterative based on how is your development life-cycle. If the catalog upload, products, brands are staggered towards different releases, plan the search relevance accordingly.
- Input to the performance cycle: The outcome from this exercise should also be fed into the performance system as the search performance should factor in the recommendations which will be applied on the production.
The search relevance is a very subjective criterion which varies by industry, business, catalog managers, merchandizers, the current promotions, and most of all your product catalog.
The search relevance activity needs to get under the skin of the shopper to be able to identify the patterns and results. The business needs to be engaged very closely in the process.
I would hope that the approach outlined in this article helps you to work towards the planning exercise and also gives a starting point. Do share your experience on search relevance, as I look forward to the different ways in which we are out to achieve the business outcome from our commerce sites – that is convert the searches into baskets and orders.
I am attaching a basic version of search template. If you think this is useful, do let me know so that I can share other versions on the same.
Search Relevancy Template - 2016-FEB-SearchRelevance.xlsxView Details
Previous related blog: Customizing search for shoppers and retaining them as their attention time dwindles
1 - Changing the relevancy of search index fields
2 - Tuning multiple-word search result relevancy by using minimum match and phrase slop in WCS Version 7
3 - Search Rules
Modified on by Shweta Gupta IBM
You may have been part of commerce implementation cycles where the functional stability pushes the timelines for the performance iterations. In such scenarios, the project teams may decide to start some level of application diagnosis while waiting for the isolated performance system with stable functional build.
The lower testing environments like SIT (System Integration) and UAT (User Acceptance) can make good platforms for the solution implementation and development team to do analysis on the application performance. These are some of the factors you would like to consider when you would like to use them for diagnosis and consider them in your analysis factor.
a) What is their deployment topology with respect to the production
b) What is the level in terms of the cumulative fixes, APARs wrt the production
c) What is the data load sizes with respect to the production
d) What are the management center business rules, activities, search rules present
e) What are the eSpots and promotions available
f) Can you get some isolated time slot to conduct your single user and some trivial load tests
g) What is the state of the integration systems like order management, payment, and others and how do you isolate your diagnosis
There are several monitoring tools that I have covered in my other blogs that can aid in your analysis. You may not have an APM (Application Performance Management) tool installed on these environments’, however you do have access to the Health Center tool which is packaged with WebSphere Application Server (WAS). I will use this specific article to focus on one of the IBM Monitoring and Diagnostic Tools, Health Center, which is an agent-based diagnostic tool for monitoring the status of a running Java applications.
The Health Center agent libraries are included with the WAS server Java SDK (Health Center 3.0 or later is required for WebSphere Commerce). This makes it an easy-to-use tool available on these environments which can be used by the developers, architects and performance analysts. The Health Center tool can be as easily used in the toolkit development environment to allow developers code for performance. It will show them the time spent and the thread stack. It is prescribed that all developers profile their code and make it performing prior during the implementation phase and work on the 80-20 rule, where if you are able to optimize the highest consumers, you will gain performance. When the code is in the SIT and UAT, the focus should be more from an integration perspective. However, I must admit, we often end up doing application level code tuning based on analysis till much later in the cycles.
Configuration, Setup and Running Health Center
1 – Check the version
# /opt/IBM/WebSphere/AppServer/java/bin/java -Xhealthcenter -version
[Thu 27 Aug 2015 11:55:33 AM IST] com.ibm.diagnostics.healthcenter.java INFO: Health Center 22.214.171.12441209
Aug 27, 2015 11:55:33 AM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean <init>
INFO: Agent version "126.96.36.19941209"
INFO: Health Center agent started on port 1972.
java version "1.6.0"
2 - Enabling the Health Center agent in profile or headless mode
In the headless mode, the agent starts collecting data immediately, and stores the data in files called healthcenterpid.hcd, where pid is the process ID of the agent. You can load this file into the client at a later date, to view the data. Headless mode is useful for systems where you cannot connect a client to the agent.
I am going to use the health center tool in the full mode for this article. The agent starts collecting data immediately and then you have to launch the client and connect to the server process for viewing the data.
3 – HealthCenter Settings
The Health Center agent can be configured as the JVM properties. Please refer to the KnowldegeCenter documentation for the configuration.
Knowledge Center Reference
Property (use -D for JVM arguments)
Use this format with the -Xhealthcenter option of the Java command when you start the agent and the application at the same time
Sets the JMX port number. For JMX client-agent connections (the default), the client uses port 1972 by default for communicating with the agent. If the client cannot use port 1972, it increments the port number and tries again, for up to 100 attempts. You can override the first port number that the agent tries to use.
Set to on to enable the client to communicate with the agent by using a JMX connection. This property is set to on by default.
Specifies headless mode. The agent starts collecting data immediately, and stores the data in files rather than sending it to the client. When the application ends, the agent creates a file called healthcenterpid.hcd, where pid is the process ID of the agent. You can load this file into the client at a later date, to view the data.
If headless mode is used, you can specify multiple other settings
I am starting the WebSphere Commerce Server using the following parameters for this illustration purpose:
-Xhealthcenter, -Dcom.ibm.java.diagnostics.healthcenter.agent.port=1980, -Dcom.ibm.diagnostics.healthcenter.jmx
These settings get saved to the <server.xml> under:
<profile directory >/config/cells/<cell>/nodes/<node>/servers/<servername>/server.xml and are picked up by startServer.sh
4 – Restart the server
Restart the WebSphere Commerce Server you are profiling. Confirm that the healthcenter agent shows up on the server. You will find the directory <healthcenter> created under the application server logs
Installing the Health Center Client to View Data
You may follow any one of the options to install the IBM Support Assistant.
Using a Health Center Client to View Data
1 – Connect the Health Center to the port mentioned while starting the application server.
The connection will start and you will be able to see the following data using the left navigation.
Example of a Profiling View for com.ibm.commerce.rest* filter:
You can use filters to view the classes and methods that you are watching. You can then turn on the trace for the method you want to drill. In Health Center, you have to enable method tracing when you start your application, as it cannot be enabled at run time. Enabling method tracing is intensive and negatively impacts application performance, so you want to do it based on what you are drilling.
Using health center profiler for your debugging
I am showing a few screenshots to help you see the traces for some of the commerce use-cases.
If your storefront has created extension to the Wish List and written custom code around it, you are looking to profile the user actions around adding to wish list, sharing the list and viewing it. You can chose a filter com.ibm.commerce.giftcenter.* and then look at the Invocation paths.
You may use the “Reset Data” button on the toolbar to get data for the current flow only. The second button from the right.
Search based operations:
IBM Health Center allows the commerce developers to monitor and profile their application in development and testing environments with an ease, no additional skill and resource required to install and use monitoring tools. The tool comes pre-installed with WAS and has a simple configuration to be setup in both profiling mode and headless mode.
The tool provides data like CPU usage, GC, I/O, Network, however the focus I have provided in this article is the profiling of live application, and using the method traces. This enables a developer to review the invocations and get into the details of the hot methods and drilling further to locate and then work on them. It will allow the analyst/architect/developer to analyze the path taken for a cold, not-warmed page versus and cached page. The developer can extract nuggets of information from the flows, which help with both troubleshooting functional issues, and performance issues.
When do you exercise load on the environment, the Health Center will allow you to view the system metrics and JVM metrics like the threads information. The Web Container threads are most valuable in understanding the behavior of application under load. I will come back to discuss the thread analysis topic again. If you are interested in some specific monitoring methodology or tool, feel free to talk about it using the comments section.
Modified on by Vani Mittal
A requirement that we sometimes come across is to have region based product assortment and pricing. FMCG and grocery retailers typically have different pricing and product assortment in their physical stores and the same needs to be replicated in the online channel. Running local promotions and offers is also quite common, for example, for a local festival or event.
There are multiple ways to achieve this requirement and the approach you choose to implement depends on the level of differentiation required among regions. Let’s explore some of these approaches.
The definition of region is kept intentionally loose here. Multiple countries could constitute a single region. Each country could form a region or one country could be made up of multiple regions.
The region to which a user belongs could be identified using many different approaches. It could either be through the user provided zip code or by using a geo-location service. This discussion is out of scope of this article. All we really care about is that before a user starts browsing the catalog, her region has been identified and saved somewhere, be it in a cookie or in the database.
Approach 1 - Multiple extended sites
This is probably the first approach that comes to one’s mind. The structure and concept of extended sites lends itself well to this requirement in general.
Each region can be represented using a separate e-site. Each e-site can have its own sales catalog that offers a subset of the products available in the master catalog in the catalog asset store, thus offering a different product assortment.
Each e-site could have its own price rule which could override the prices from the master catalog as needed.
The e-sites can even have their own UI differences and region specific marketing and promotions.
If the number of regions is very large then similar regions could be combined into a single e-site with the minor differences within those regions being implemented using other approaches discussed here.
Approach 2 - Multiple catalog filters in B2C store model
In the B2C store model, the store’s default contract is used to define the default entitlement of the store. All customers shop under this one contract which by default provides the same catalog and pricing to all users. There are ways to work within this default contract and offer differentiated catalog and pricing.
You can create multiple catalog filters and add them to the default contract using the CatalogFilterTC term. Each TC should have a TC level participant for a member group (region specific) which means that the TERMCOND_ID & MEMBER_ID columns in PARTICIPNT table will be populated while the TRADING_ID will be NULL. See the definition of PARTICIPNT table for details of these columns: http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.database.doc/database/participnt.htm?lang=en.
The default contract cannot be edited through WebSphere Commerce Accelerator, so to add the catalog filters to it you either need to use the contract XML import command (ContractImportApprovedVersion) or write custom data load configurations using TableObjectMediator to load the data into the required tables (TERMCOND, PARTICIPNT etc).
The default contract always has one price rule associated with it. The price rule can include different branches of pricing for different member groups thereby allowing for different pricing. Alternatively, you could create a custom price rule condition that checks for the user’s region (assuming it is saved somewhere – cookie or database) and use that to decide which path to follow.
Refer to this topic in info center on how to create custom price rule conditions:
Adding a new custom condition or action for a price rule
For this approach to work, the customers need to be added to region specific member groups. Only static member groups, i.e. member groups that have users added to them explicitly, can be used as TC level participants. Users can be added or removed from member groups dynamically, but you need to write custom code to map the shoppers with the appropriate member groups. This can become a performance bottleneck, particularly if the retailer has a very large user base.
Approach 3 - Multiple buyer contracts in B2B store model
If you are using the Enterprise edition of WebSphere Commerce, you can use the B2B store model in a B2C scenario with some adaptations.
Starting from Feature Pack 8, a single Aurora based storefront for both B2B and B2C capabilities is offered. When publishing the Aurora store archive you can choose to publish it as a B2B storefront and then disable the B2B features that you don’t require, for example, requisition lists, saved orders etc. Some of these features can be disabled using the Store Management tool in Management Center, while some might require JSP changes.
The benefit of basing your B2C store on the B2B store model is that you get the B2B business user functions; for example, you have the ability to manage the buyer contracts using WebSphere Commerce Accelerator.
In Feature Pack 7 and earlier, you can use the Elite starter store as your store’s base and again adapt it for B2C, i.e. enable guest user shopping and disable the features you don’t need.
If you don’t need the B2B business user functions, you can base your store on the Aurora B2C model too. In this case, you can manage the contracts either through contract XML import command (ContractImportApprovedVersion) or write custom data load configurations using TableObjectMediator to load the data into the required tables. Data load is particularly useful when managing a large number of contracts.
Whatever store model you choose to go with, the rest of the approach described here will remain the same.
Product assortment and pricing
Each region can be mapped to a buyer contract with each contract being associated with a unique catalog filter and price rule. A catalog filter can be assigned to a contract through a TC of type CatalogFilterTC and a price rule through a TC of type PriceRuleTC.
For a given contract, only a single price rule can be in effect at one time. Therefore, the price rule you assign must generate prices for every customer and for every catalog entry available to customers under the contract.
The contracts need to be applicable for all users. This is handled through contract participation where one row in PARTICIPNT table with MEMBER_ID value NULL having a ‘buyer’ participant role in each regional contract is created. This allows for any user to shop under the contract.
Once the customer’s region is identified, the corresponding contract can be set in the session using ContractSetInSession URL. Once this is done, the customer will see and be entitled to the region specific product assortment and pricing.
For contracts and catalog filter to be used at runtime, search entitlement needs to be enabled. This is required since from Feature Pack 7 onwards, browsing and searching as non-transactional requests were offloaded to the search server. As a result, the WebSphere Commerce server acts as a transaction server, and the search server acts as a non-transactional server that can be separately deployed and scaled. Entitlement was moved to the search server to apply entitlement checks before and after search results are returned in the storefront. If you use the Aurora storefront from Feature Pack 7, you will need to update the product and category REST calls in the JSPs to pass in the contractId that is set in session, so that it is used by the search server to enforce the entitlement.
By default, search entitlement is disabled for B2C stores and enabled for B2B stores. You can insert or update an entry with the name ‘wc.search.entitlement’ and value 0 (disable) or 1 (enable) in STORECONF table.
If the regions have simple UI differences, then those could be managed through contract specific style sheets where you can follow a naming convention for the files so that the appropriate one is picked up. For example, the name of the CSS file could be same as the contract name. Alternatives could be designed using the same concept (map the UI differences to the contract) if the UI differences are significant, but this would require customization.
Marketing and promotions
If region specific marketing and promotions are a requirement, then region specific customer segments can be created to target the customers. The customer segments can be created as static member groups, i.e. member groups that have users added to them explicitly. Users can be added or removed from member groups dynamically, but you need to write custom code to map the shoppers with the appropriate member groups. And as discussed in the previous approach this can become a performance bottleneck.
An alternative is to not include any explicit members or implicit rules of membership when creating the customer segments. So, all customer segments are actually identical copies of each other with only the names being different. To identify a specific customer segment as belonging to a particular region, the relationship between a region’s customer segment and contract can be stored in a custom table. You can customize the member group checking logic to return true when evaluating the customer segment of the region that the customer is currently browsing. Essentially, every user will be treated as a member of the regional customer segment only when they are browsing the contract of that region.
You can then use the customer segment while defining a promotion or a marketing activity. Refer to the screenshot below for an example of a region specific product recommendation.
This approach of using multiple buyer contracts works well when the number of regions is not too high. If you have a large number of regions, then that would mean a large number of eligible contracts for each user since each user is eligible to shop under any contract. WebSphere Commerce server and search server scan through all eligible contracts of a customer at multiple points. Therefore, having a very large number of eligible contracts can impact the performance of the storefront.
WebSphere Commerce provides a flexible and powerful architecture and runtime framework. As a result, there can often be multiple ways to implement a given requirement. In this blog post, we discussed the approaches that can be implemented to offer region specific products, pricing, UI, marketing and promotions. You should be able to evaluate the approaches given here and map them to your business requirements to select the best approach that works for you.
DEV/QA out of sync with PROD
Most often solution architects see their role wind down once the retail website they help architect, design and implement has been successfully launched. However, architect's deep expertise is needed for some day-to-day operations also. Often I have seen projects that fight through defects and scope creep to launch, but soon find that the development and test teams are way out of sync with how the system is being used by the business. Core of the sync-problem is data itself.
Let me elaborate that problem a bit here: It is typical for development teams to have fair bit of ownership of a "Dev" integration server environment (I will refer to that as DEV - not to be confused with development toolkit environment owned by individual developer, which I shall refer as TOOLKIT). Then test team would have their own environment to perform functional and system integration tests - I shall refer to that as QA. Next we typically have a preproduction environment that closely resembles production in terms of data and configuration - typically user acceptance testing may happen here, sometimes perf test may happen here.... I shall refer to that as PREPROD. Finally we have production environment that shoppers and business users actually use - referred as PROD. Note that each of DEV, QA, PREPROD and PROD will have their own authoring instance for business tooling, dataload etc and live/runtime instances for shopping activities. During initial launch, QA is often the environment where 90% of all data and configuration gets trialed and tested. PREPROD is where that reaches 99% with PROD having 100% of what is actually live. The percentage difference is also skewed towards marketing, promotion and content than with catalog. So in many ways, QA is a fair representation of what business users and shoppers experience in PROD. But once the site is launched, this equation changes drastically. PROD becomes the hot bed of activity - business users may go over and beyond processes defined by the solution team, adopt new processes....and end up creating data and configuration that lower environments never get any inkling about. If the technical leadership is not engaged, within the first year, QA drops from 90% to 66% as a representative of PROD. Soon development teams will see PROD defects where developers and testers swear that they have thoroughly tested but fails in PROD because that environment is now supporting new type of data that the code had not anticipated.
Problem is with communication. Technical teams needs to be actively engaged with business teams to understand ever changing landscape of retail processes. New attributes may be flowing into dataload - which none or if lucky, only a portion of IT team may be aware of. New price rules, new contracts, new layouts.... While most of it is supposed to be transparent to development teams, it is often not! Teams may have implemented some customization that get affected or affects work done by business user. Worst case scenario, as I observed with one customer, is for the business team to start coming up with their own independent processes, working with non-e-commerce IT teams to create solutions in other parts of the organization such as ERP systems. And end up choosing solutions that deviate from WebSphere Commerce recommended patterns of solution.
How do we deal with this
There should be a monthly interlock with business analysts and technical architects - where requirements are continually re-evaluated and validated. While meetings can solve one side of the problem, often truth is in data! Hence there should be some process put in place to ensure QA and PREPROD environments are updated with any non-transactional change that is introduced in PROD - in that sense almost anything that is affected through Commerce management center is a good candidate for replication in lower environment.
One way to ensure this replication is automated is via the use of Data Extract utility. Folks that are familiar with Data load, knows how the framework is geared to read up comma separated values in a specific format being mapped through an XML and used to created BOD objects to be loaded into WCS DB. Extract is just a reverse of that process - from DB data is read into BOD and fields from BOD are mapped to field columns that are written into CSV. The framework is exactly the same and the XML file used are very similar. In FEP8 - there is ability to add custom SQLs to the extract process by just inserting them within relevant tags - use of that for extracting promotions is documented in the knowledgecenter and a similar approach should work well for other parts of the DB : http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.data.doc/tasks/tmlextractpromotiondata.htm?lang=en.
Things to Watch out
One caveat is that attachments require an associated process to collect and FTP images/htmls that they may refer to -- these may have been directly deployed to web server or CDN - hence a separate process is needed to collect them and bring them down to suitable location for lower environments. Another hurdle I faced was in handling custom catalog associations that were implemented through custom code than through data model or BOD definition - so, the extract would need to mimic the same customization to ensure right information is extracted. Another was related to store and catalog organization model - especially if sales catalog and categories were directly created in PROD - then extra care needs to be taken to replicate those first. Lastly, inspite of the utility giving great capability, some data massaging may still be required.
It is worth investing in such a solution for any ecommerce project. For one, it helps to keep lower environments refreshed. But same process can be used to provide outward feeds from WebSphere Commerce - for example, sending catalog data to your ratings and reviews provider or your analytics provider.
Modified on by Anbu Ponniah
Performance metrics available for WebSphere Commerce
Need for Agile performance preview
The solution architecture is done and the iterations are on their way churning out designs and code. In projects executed following agile methodologies, features are designed and added in increments. While it is not too hard to get the whole picture on functional capabilities of the solution, it is relatively difficult to get a handle on how all this incremental design and code will eventually perform untill the last of the iterations are done and dusted. That often means, projects shift from iterative mode to waterfall mode with a performance test phase planned right at the end of development cycle, leaving little leeway for development teams to correct any major performance bottlenecks identified during performance testing.
Can performance testing be made iterative? I can imagine my performance specialist rolling her eyes. Problem is that we cannot certify the system for performance unless a test can be run in a controlled environment. And it takes time - which is premium in short iterations demanded by agile projects. So, is there hope? If we compromise our scope to not "certify" the system for performance and instead go for "flushing out major performance problems upfront", we may have a way around this conundrum.
Performance Measurement tool and framework
From fix pack 7 onwards performance logger feature supports a trace string "com.ibm.commerce.performance=fine" to trace response time of key external calls from WC to OMS. Through this a development team can understand the SLA when checking availability, reserving and cancelling inventory and getting order details from OMS. With Fix pack 9, the performance measurement logger feature of WebSphere Commerce allows teams to understand response time, impact of caching and size of result on each call at various application layers. It replaces the old ServiceLogger tracing with a package of logging that can be analyzed through the performance measurement tool.
How to enable and use performance measurement capabilities is well documented in the knowledge center. But here are the essentials of how a development team can use these capabilities to get a preview of performance capabilities of the solution before official performance test is due:
1. Plan a day during your test runs in your iterations for collecting performance metrics.
2. Work with your WCS/WAS administrator to enable the following traces:
3. Have your testers execute a full suite of tests that represents typical browsing behavior of the live site - include search, promotional messages/pricing, navigation, registered/guest and checkout flows.
4. Collect the files in WAS_profiledir/logs/ folder of your server.
5. Use your favorite tool to project the analysis -- for example, basic performance reports are in CSV format and can be simply charted in spreadsheets
Once a base line is formed, further iterations can focus on deviations from previous measurement.
What is measured
Development team gets a preview of where the heaviness is and where either a caching strategy or re-design is required. The reports are detailed enough to point out the following:
Average call duration and Execution time (performance reports)
Size of results (performance reports)
Cache hit/miss (performance reports)
Stack of operations with time taken by each child in the stack (stack reports)
Get information on a full execution cycle (execution report)
Identify source of calls (caller report)
A note of Caution
While this test is not a replacement for proper performance load test, this type of testing can point out where contention is likely to occur and take corrective design decisions early in the development cycle. If this was so easy, then why isn't everyone doing it - simply because there are likely to be false positives and a few false negatives in the findings. After all, the test environment is unlikely to be modeled to represent the scale of prod or perf environment. The environment itself may have some variability due to multiple testers accessing it and development teams delivering fixes. And test execution is subject to manual variations - think time cannot be controlled, sequence may vary etc. So, teams should take findings of such a testing with a grain of salt. So, such an approach requires the performance architect to calculate probabilities against the findings to determine which of them represent real problems. But in the hands of an expert, this mechanism can help create a solution that is functionally devoid of performance bottlenecks.
The price pressure
Twenty first century shopper has so many options. Unless a retailer is a niche player, it is a shopper's world. While retailer has many tricks up their sleeve to bring eyeballs to their e-commerce website, keep them there, convert into a sale and bring them back, the de-facto strategy that all retailer's adopt is the ubiquitous price strike through to give the shopper a sense of getting a good deal on the goods. Price used to be decided by business teams pouring over their analytics reports, industrial demand/supply reports and peering into the crystal ball of future trends. It still is done like that in many places. But what has changed is that price gets revised (usually downwards) more frequently than before. What used to take months and weeks earlier now happens matter of days and hours.
Let us say your client has a pricing strategy in place - and have the experts coming up the right prices to show for a product each hour of the day and change that every two hours . What would be your strategy as a technical architect and adviser?
Options for price updates
In technology, every thing is possible but may not be practical within the constraints a particular solutions may be operating within. Understanding the when and what to compromise on is the trick.
So what is possible in WebSphere Commerce?
If you get your price as CSV from your pricing engine, then refer here for how to load up your price: . If your pricing engine is IBM DemandTec, then see DemandTec with WebSphere Commerce.
For more control BODL service asset could be used to create a custom load handler.
Or if CSV file is not your cup of tea - say you need this as a true "feed", a look at the current "Content Management system web feed" utility provides some clues. At the end, same data load framework will be used to load information into WC database, but ability to establish connection to the feed provider and transform the feed into WC specific business objects is the real work here (how to do this will be taken up in a future post).
Or may be old style (now deprecated) massload may be used.
Some may use a form of database export/import.
Of course, WebSphere Commerce may be the price master so that business users create/manage price using Commerce Management Center.
Any which way we load the price - the question considered in this blog is : How often should we propagate updated price to our live site?
Real question: How many times?
A change in price needs the following to happen:
1. Change must be propagated from authoring DB to live DB.
2. Search index must be rebuilt (if using priceMode = 1 or 2)
3. Dynacache must be invalidated and warmed
4. CDN cache must be invalidated and warmed
All the above takes time. I don't mean hours - but we need to be cognizant of the number of minutes each of this takes.
If the price to be shown is dependent on conditions (price rules) or if the retailer wants to define price reductions via promotions because that is how they see price discounts, then some pre-processing will be necessary to decide which price to index and how to cache your price by associating the correct combination of customer segment, promotions and price rules.
And cache invalidation and warming takes time unless you have a caching appliance like WebSphere eXtreme Scale solution that has excellent cold-cache capabilities that renders warming unnecessary.
The price change also affects in-flight transactions. A robust business logic should inform shopper of the change in price up to a point in their shopping journey and otherwise accept the order with previous price. Visible price change during checkout affects shopper's trust on the retailer. Frequent switching of price may just make people wait longer and delay their buying decision. But not changing the price for long could expose the retailer to competitors under cutting them. Finding the right balance is more of an art than science.
One customer agreed for twice a day price change. Another was OK with hourly updates. Most challenging one was a customer who insisted on instant change and that too on average 50 times a day. Things we need to work with retailers is batching as many updates together as possible and helping the retailers work out some form of prioritization within their process. For example, price is wrong (missing a digit altogether) warrants instant reaction. In other cases, it may not be that urgent. Also, solution could be found by a different strategy altogether. For example, reaction to under cutting could just be advertising a policy of price match with reputed retailers rather than getting into a no-win war on price. Winning the retail race on core differentiators such as engaging shopping experience, after sales service etc is more profitable.
Random closing thought
One random, but relevant thought: What is the deal with strike through price? Some retailer may just want all their prices to be struck through with a lower price right next to them. And the "offer" price may just be one cent or one rupee less than the "list" price. Such tricks may just train the shoppers to filter out the struck-out price altogether, robbing the emotional benefit of scoring a deal. Often the actual value of saving in numbers is more transparent than percentage saving.
Modified on by Anbu Ponniah
E-Commerce in India is valued at USD $10 Billion in 2014 and expected to cross USD $15 Billion in 2015 (Report from IAMAI). India has barely scratched the surface of web ecommerce potential but is poised to go ballistic (in a good way) in mobile commerce. An interesting tidbit here is that over 5.5 million 4G enabled mobile devices have been sold but less than 100K 4G subscribers are there, mainly because of the limited availability of 4G network in India. But with reports suggesting that 4G Internet will be launched countrywide in 2015, it is not far fetched to think mobiles may soon monopolize shopping in India.
But rules of old world no longer apply. (Old = 5 years ago.... and I may be too generous here.) While there is general trend of preferring native apps over mobile web or hybrid apps, retailers really care about nailing just one thing : "Engaging User Experience".
What exactly makes a shopping experience "engaging"? The way I think, it is the complete shopping experience that presses all the right psychological, emotional, ethical and rational buttons to make shopping a gratifying experience at conscious and subconscious levels. The experience starts with how I hear about a retailer, my first impression on visiting the retailer, rich information centered around not just products, but also how products gel with my lifestyle (or my aspirations), ability to navigate, locate and visualize what I want, non intrusive but always on hand sales associate attuned to my level of knowledge of what I want to purchase, not having to shop-hop.... list goes on.
Yes, we need cool UI to give the initial wow factor, but we could do with functional UI that does not get in the way of the shopper. Functional does not mean run-of-the-mill, factory cut UI. It should still stamp the retail store's individuality. Beauty is important - world does not take the retailer seriously otherwise, but as mobile shoppers mature, skin deep beauty will ring in hollow. The days of having cookie cutter approach to providing shopping experience are numbered and coming up with custom cookie cutter pattern alone will no longer set a retailer apart. When I visit an e-commerce retailer in my mobile, I expect the product recommendations to match my previous browsing and purchase history. I only notice it if they don't. Rules of engagement are evolving and rules based engagement, any which way we package it, is no longer dynamic enough. The rules engine now needs to think. Not only think, but reason! And this intelligence must be available for both shoppers (as a guidance/advisor/sales associate) and for retailers (to enable marketers and merchandisers digest large quantities of data and discover patterns in a meaningful manner).
IBM has forayed in to the world of cognitive computing and has thrown open Watson analytics with natural language supported exploratory visualization for business associates that takes analytics to new heights without the steep learning curve. The free trial is available for everyone to give this a try. Free version has some limitations : For example data storage is capped at 500MB, record size cannot be more than 100K rows and 50 columns. But it will give a great insight into the promise that this new age solution holds. Imagine being able to ingest scores of spreadsheets from your ERP systems and other analytics tools and gain insights using natural language queries.
How do you do it? Visit IBM Watson Analytics, use your IBM id (register one if you don't have one - it is free). You will receive an email to validate your account. Once validated, you are taken to a landing page where you can upload your data - for example recent sales data. Once uploaded it suggests starting points for exploring your data. You can also view the useful sample tutorials for tips on analyzing your sales or marketing data. These insights can now lead you to make decisions that go over and beyond regular formulaic decisions. Once these insights are available, WebSphere Commerce solution can now factor these relations into their recommendation or guidance engine.
I played around with just a dump from a sample ORDERS table of WebSphere Commerce. My intention was to find my order spread by currency (aka region) in a given time frame. It was as simple as dumping a report about all sales order into a CSV, dragging and dropping that CSV onto IBM Watson Analytics and going through various recommended and custom visualization schemes.
I could type in natural language questions and it responded with graphs. As an example, I could make out that total sales value indicates that my store's sales in Japan (presumably due to value of Yen) far outstrips my sales anywhere, but number of orders say my USA sales was easily more than sales of all other countries put together. I could then drill more into the data to understand average value of sales from each region and type of products selling. This helps me tailor my recommendations, promotions and campaigns more effectively. Many analytics tools may do this in one manner or the other, but IBM Watson does this while supporting natural language queries putting it within reach of any retail business user and not requiring a PhD in data science.
I have barely scratched the surface of this tool's capabilities. Next time you wonder why you are able to navigate to a page showing your preferred brand, color and size of t-shirt within a couple of taps without ever disclosing this information to the retailer, you may have to thank Watson for it.
Has anyone played around with Watson analytics? What are your thoughts? I welcome you all to share your experience.
In part 1 of this blog we had seen how data that is handled in an ecommerce system could be bucketed into various perspectives and categorized into the role it plays or supports. And we discussed two use cases which I shall restate here:
UC1: A shopper must be allowed to define and manage any number of personal information about them.
UC2: A shopper must be shown personalized messages based on personal information they have shared
In this blog let us get down to the solutioning decisions and see how the puzzle pieces fall into place.
For properly solutioning this we must revisit the data that we will end up capturing and understand what qualities the data store must support. Apart from the obvious needs of storing, retrieving and updating data the choice should allow for discovery and exploration of the data over and beyond the relationship established through the data model. Let us discuss this need through a couple of examples of the second use case.
Beyond saved relations
Example 1: If the retailer wants to show a personalized message such as "Your favorite sports team is the most popular team among shoppers in our stores". For this we must be able to discover if indeed the sports team entered by the current shopper is mentioned by most of the users. A simple document store with some faceting capability can give us this information.
Example 2: On the weekend before Valentine's day every female shopper who have one of their favorite colors as red, pink, dark brown or chocolate will be shown an marketing widget that lists a set of select apparel accessories.
The above examples may or may not make sense as real use case examples, but are taken to illustrate two distinct problems - one is that of aggregation and other is that of segmentation. Similarly, from the retailer's perspective, this design should also support deduplication and standardization --- let me call this using a term that is superset of both: categorization. Without getting a view of categorization, merchandisers and marketers won't be able to utilize the rich data gathered. And we need robust analytics to help us translate the data into trends and relate them to behaviors that a retailer can target for their merchandising and marketing decisions.
Now NoSQL can help us solve the shopper perspective well and provide some help regarding categorization. It can integrate with an analytics solution for the last bit. The data we gather regarding shopper's personal profile should now be used to personalize his/her shopping experience. And the rules of influence could be self contained within the profile information or may need to be executed in the context or in combination of information from catalog, orders or marketing sub-systems. This is actually one of the reasons why we traditionally choose a relational data model solution for this.
And this leads to the first decision point - the data must be available for direct queries from client systems and from ecommerce business logic systems and should be map-able with existing relational data model. This is just to say that when I save information in my NoSQL data store, I will include a few key values such as member_id, store_id, catalog_id, address_id and language_id.
Second decision point is to mirror relevant promotional, marketing and search rules information from relational data store to the NoSQL data store. This is particularly easy since this information is available in XML format CLOBs in WebSphere commerce and lends itself to easy import into data bases like Cloudant or MongoDB. Obviously, every time a stageprop happens, a process should push updates from relational model to NoSQL model.
Third decision point is to extract information from NoSQL data store for analytics and feeding this back into the system of record for factoring it in promotional, pricing, search or merchandising rules. Now it may be attractive for analytics to actually run directly on the NoSQL store itself - but worst case we need a process to export, FTP to analytics system and import into its data store. I advocate the analytics to instead run on the NoSQL data store to become real time (subject to performance considerations) and create outcome of the analytics also in the same data store.
Fourth decision point is to keep authorization, contract entitlement and organizational logic away from this NoSQL data store. These can be enforced at a different layer.
The priorities for the CIO
The priorities of CIO of an organization include delivering on business solutions, business intelligence and analytics and improving the customer experience with the brand. The IT operations team within the CIO is responsible for the IT systems, which include,
development and innovation of business solutions
production management and uptime
operational efficiency, providing with the analytics from operational and transactional data for the consumption of both IT and business
The priorities for the retail CIO
Additionally, the retail CIO is grappling with the world of omni-channel, building a single customer experience, and working towards delivering personalized services. This involves building a mobile strategy as the statistics continue to show higher exploration on mobile and higher conversions on desktops/tablets. This also means the focus on the Big Data paradigm, to gain competitive advantage.
In the following blog, I will outline the layers of monitoring that an IT Operations team will need to setup as the commerce site is ready to Go-Live. The key operations data is required as soon as the system becomes available to the customers. The retailer must get their data in view and the thresholds and alerts set out prior to the launch. The alerts should be set based on the performance cycle for the launch. The other business data can be configured based on the priorities and focus of the business.
The metrics and measurement
1 – The key performance indicators (KPI) which a retailer wants from their digital channels are
Visitor summary: page views, average session length and time spent
Visitor behavior: bounce rate, new visitors and conversions
Product views, top search terms, search hits and misses, cart abandonment rate
Paid search sessions, marketing, referring site sessions
Browser types, devices
User behavior across different channels
Performance against industry trends
There are several analytics tools which offer integrations with websites. IBM’s Digital Analytics is the SaaS offering which provides insights on this data and also publishes industry trends across holiday seasons and events. Some of the reports that any retailer will find very insightful to plan for upcoming 2015 events and holidays are the 2014 Black Friday report, 2014 Cyber Monday Report, Worldwide Online Holiday Mobile Shopping Report 2014, UK Retail Online Christmas Shopping Recap Report 2014.
2 – Retailers have to manage their operations and deliver to business based on the uptimes, scalability and non-functional requirements (NFR). Some of the metrics which are must-monitor for a site to deliver client satisfaction are:
Application performance metrics like CPU usage, JVM heap usage, garbage collection, Web Container threads
Backend performance and response times from 3rd party integration systems like payment gateway, or external applications like inventory, availability, address verification
Response times for the various operations like browse, search, cart, order completion
Requests per second for various incoming requests
Number of orders, order-size, order-value
The Application Performance Management tools are specifically focused on monitoring and management of performance of the software applications. The tools provide a view on most of the above parameters, by either using agents or are based on log analytics. The out-of-the-box features have to be augmented by providing visibility to other metrics, like the order information will be extracted from the database. The agent-based tools need to be configured correctly and tested to ensure that they do not impact the performance of the host application. We will explore some of the tools under APM in future blog topics.
3 – Application specific monitoring aspects
Java applications have standard mechanism for managing application resources using Java Management Extensions (JMX), is managed bean (MBean). The WebSphere Application Server can be extended with custom JMX MBeans. One of the examples to demonstrate a key performance resource for WebSphere Commerce is the caching efficiency. The performance of the WebSphere Commerce can be optimized by building in a strong caching methodology at various levels. The application level caching is built upon the WebSphere Application Server Dynacache which caches JSP/servlets and data object. The cache statistics of cache hit, cache miss, hit/miss ratio, the cache eviction, invalidation, provide insights into the site usage and its performance. The dynamic cache service provides an MBean interface to access cache statistics. The Extended Cache Monitor tool pulls the mbean statistics for cache instances and displays the following data:
Display contents of servlet and object cache instances
Display the Dynamic Cache mbean statistics for cache instances
Display the dependency ids of the cache instances
Display the disk offload statistics and contents
There is a lot of data in the commerce system, and the IT Operations can harness the information for insights. The data ranges from the application performance metrics to the business data and analytics for business insights. The downtime, slow site, unknown issues in the infrastructure will catch an operations team blind-sided, impacting in loses and brand dilution. The monitoring and alert enables the operations team to manage and troubleshoot the system in a predictable context. The operations team should evaluate and chose the monitoring tool based on the data they want to view. The tool should be tailored with correct thresholds, alerts, messaging to provide the effective and actionable insights.
As an architect/practitioner growing up in a world of enterprise architecture where data is rarely seen outside the ambit of relational schemas, NoSQL brings with it a breath of fresh air - simultaneously challenging previous beliefs, promising exciting opportunities and bringing its load of inevitable baggages. In this blog post, I shall walk down the discovery path of how, as an architect, I see NoSQL complementing or supplementing or replacing traditional solutions. The perspective is that of an architect trained in a world of stack architecture and transactions, so this may not resonate well with advocates of either technology. However it highlights opportunities and challenges that would translate into solutions that everyone can relate to. This will be a two part series - part 1 (this blog) will set the context and part 2 will detail one solution possibility.
Defining NoSQL as a comparison of SQL databases is no longer straightforward. Lines are blurring. For this blog, NoSQL databases are those that have flexible schema capabilities, store their data as one of the following formats:
and primarily support a non-SQL query interface. They may or may not support some or all of ACID (Atomicity, Consistency, Isolation, Durability). Internet is filled with information on NoSQL, their features and extensions, so I will try not to replicate this information. Instead let us focus on what it means for ecommerce - starting with WebSphere Commerce.
Let us first set some background here. First order of business is to view information managed in an ecommerce solution in a perspective unconstrained by traditional views. For example, one may view data handled by WebSphere commerce from the following perspectives
Catalog perspective -- Attributes, structure and administrative information related to products. Here the term attributes is used in the traditional sense of the word and not restricted to WebSphere Commerce's definition of attributes. Hence, the information included in this set of data are information that describes products for external consumption, information about products for retailer's own management, information about structure and relationship between products and information that are needed to administer or manage it all. Entire Catalog data model of WebSphere commerce falls under this bucket. Importantly, this does not include price and inventory.
Order perspective -- Attributes, structure and administrative information about shopping carts and orders. Same definition here too - includes information from shopper, retailer and administrative perspectives. This, in most cases, constitutes the primary transactional data for a retailer. Information about orders will need information from other sets of data such as catalog, payments, pricing, marketing etc. It may be tempting to already think relational (foreign keys) here. If we hold off the temptation to bucket perspectives of data into relational concepts, we may find it easier to see alternative views.
Customer perspective -- Information about shoppers in our retail solution. It includes login and password for registered shoppers. It includes addresses and payment information for all shoppers. Again, it will have overlaps with orders and catalog perspectives.
Marketing perspective -- Information needed to market or promote one or more products. It includes information that defines the rules, information that should be displayed to shoppers and business users and information needed for business logic to evaluate and apply them.
Above is just a short list that this blog will use for illustrating a point of view and is not an exaustive or correct list.
For each perspective, we see the following are immediately apparent:
some data is needed to display to external world versus some information is for internal use.
some data is static, others not so static
some data needs high degree of ACID support others can live with cursory checks
With this backdrop, let us consider the following use cases:
UC1: A shopper must be allowed to define and manage any number of personal information about themselves.
UC2: A shopper must be shown personalized messages based on personal information they have shared
Traditional solution requires some innovation to allow users to create their own free-form information - so we may end up with a lot of duplicates or near-duplicates. This information can soon grow out of hand - imagine if you anticipate a million customers and dozen attributes on average for each. So natural inclination is to push back on such a requirement and instead box the attributes to retailer defined values such as hobbies, favorite XYZ (sports, color, place...) etc.
Then, in WC, one could define a service to adding information to Member attributes part of the Member data model in WebSphere Commerce, customize promotions/espots logic to allow targeting based on member attributes and then during customer interactions check if member attribute driven espots and/or promotions can be shown. This is one way (and one of our future blogs will detail this). Other way would be to define some custom table and store these as key-value pairs (in fact storing in MBRATTR has a key-value sense to it).
NoSQL gives us an opportunity to think beyond the constraints of data model here. After all if the information naturally fits the key value pair model OR can be modeled as a "Personalization Document" of a customer, why not think NoSQL here?
The solution must be capable of creating this information, allowing shoppers to maintain it and for internal commerce logic (such as promotion/pricing components) to use it in their transactions, specifically as part of existing commerce transactions. And it need not require remodelling entire WebSphere Commerce database and business logic to use NoSQL. I will be advocating a layered approach where each type of data model is used to take advantage of their core strengths while minimizing the custom code needed as much as we can.
We will explore that solution in my next blog.
What are the benchmark reports saying
The IBM benchmark report, 2014 US Online Retail Holiday Readiness Report, showed that an average shopper’s attention time is dwindling as they are making quick and abbreviated visits to retailer sites. The average time on the site was measured to be ~7 minutes which is about a minute less over past 2 years, and the data showed that there is reduced trend on page views per session. These metrics become pertinent as the retailers have to continue their focus on bringing the right products to the customers in fewer number of browses and searches.
The WebSphere Commerce search, which empowers the search on the storefront, can be customized, extended and tuned for a retailer’s business. The retailers require to consistently monitor the performance of their search through the search hits, misses, conversion ratio, site abandonment, and accordingly tune and refine based on what the shoppers want and marry it with what the retailers want to sell on their website. We explore some of the use-cases which retailers will find useful in order to achieve a high search to sale conversion ratio.
The foremost design decision is around the search index schema. The retailer would want to add more fields that can be searched, these additional fields would be added to the index schema, for example, a sport clothes retailer will want include to include the sport-type as an indexed and searchable field. This new field sport-type can be then be used for impacting the search results, say for example, the order for the search results.
Search for conversions
One of the ways to increase the search hits and help site users find what they are looking for, is for the business build a strong set of vocabulary which represents their catalog set. This is made available by what is called the ‘Synonyms and Replacement terms’. The synonyms would help a customer say who is looking for a ‘fitbit’, show the different wearable, gadgets, activity tracker, pedometer that the store sells. The replacement would be used for terms, in the same example, if a shopper searches for ‘bracelet’ in a sports store, this will likely be replaced with ‘activity tracker’, ‘pedometer’ for the retailer.
The retailer would further want to refine the search results returned to the shopper based on inventory or brand partnerships. The WebSphere Commerce Management Center tool has a store preview feature which displays the relevancy score in the search results. This data provides the insights on what adjustments the user would make to have the desired result displayed. The retailer can accordingly tweak the result by using the search rule ‘Change Search Result Order’ and then iteratively get the desired set by viewing the relevancy information. Continuing with the example of keyword search ‘activity tracker’, if the business would like the fitbit brand to show up higher than other brands, then it will need to set its ranking order higher.
The retailer can exercise more influential rules than changing order using pre-existing options in the Management Center. For instance, they can specify that the ‘Top Results’ where on keyword search for ‘activity tracker’, the business can decide to show the latest fitbit product – Fitbit Surge as the Top Result. Retailers can infact influence a result where say the search result contains a specific product, and then move it to the top of the result. So with the example, if the search result contain ‘Fitbit’, then specify the Top Result as ‘Fitbit Surge’.
The retailer should use search statistics and experiment data to evaluate the impact and continue refining the search rules to have the maximum return on their shopper behavior.
Concluding with search extensions
Influencing search results is an important extension to the marketing and promotion tool. We have seen several ways in which the retailers can achieve this with ease. We will continue to delve into search extensions in the future posts as my colleagues will share specific extensions to derive enhanced benefits from the storefront search.
Modified on by Anbu Ponniah
In part 1 of this blog post, we discussed subscriptions and recurring orders, two features provided out of box by IBM Commerce solutions. In this 2nd part, let us discuss another popular order capture mechanism – pre-orders.
A pre-order is an order placed for an item which has not yet been released. These days many retailers use pre-orders to allow for shoppers to reserve their own personal copy of a highly anticipated product. Apple iPhone is a great example of such a product for which there can be a huge initial demand on release of new models.
Pre-orders allow consumers to guarantee immediate shipment on release, manufacturers can gauge how much demand there will be and hence how large initial production runs should be, and sellers can be assured of minimum sales. Additionally, high pre-order rates can be used to further increase sales.
Retailers can provide incentives like discounts, associated exclusive merchandise etc. to encourage shoppers to pre-order.
On the face of it, you might think that IBM WebSphere Commerce does not support pre-orders, but it provides enough flexibility to allow for support of pre-orders to be added with some customization. But first, let us explore the concept of pre-order in a little more detail.
The pre-order feature can be implemented in various forms and this can dictate the complexity of the processes involved. Let us discuss two of these variations:
Pre-order without payment
Some retailers allow for a product to be pre-ordered without capturing any payment. When the product becomes available, the customer can be asked to complete the checkout flow and make the payment. In this case, pre-order is really just a reservation process where a customer’s intent to purchase is captured with bare minimum personal details like email address, so that the customer can be notified when the product becomes available. There are some points to consider about this form of pre-order:
The reservation process can be implemented as a bespoke feature in WebSphere Commerce where in there can be a simple form to capture the customer’s details and a notification mechanism to inform the customer when the product is available to order. The customer details can be stored in a custom data model and a custom scheduled job can send out notifications either through the WebSphere Commerce messaging system or through an external email/SMS provider.
Since the customer is not paying for the product at the time of reservation, there is no guarantee that he/she will actually buy the product when it becomes available. Hence, this does not provide an accurate picture of the initial demand.
Again, since payment has not been captured, the retailer is not bound to offer the product at a certain price. The price may change when the product actually becomes available.
Pre-order with payment
An alternative approach can be to capture the pre-order in a way similar to how normal orders are captured. That is, allow the customer to complete the checkout flow and authorize the customer’s credit card for the amount. Once the product is ready to ship, then the customer’s credit card can be charged. One important point to note is that credit card pre-authorizations typically expire after 5-7 days, after which the hold placed on the funds is released. The customer needs to be contacted again for a fresh pre-authorization to take place. So, this approach is ideal if the time difference between when a product is available for pre-order and when the product is available to ship is between 5-7 days.
In some countries it may be OK to capture funds immediately for a pre-order product, as long as it is clearly communicated to the customers that they are being charged in advance for an item that will ship at a later date. However, there is a higher risk of chargebacks in this case. If you make people wait before you ship, a certain share of customers always end up asking for their money back.
Now, let us see how the various subsystems of WebSphere Commerce can be adopted to handle pre-orders.
Catalog – Products that are available for pre-order need to be marked as such. This can be done by using any of the customizable fields (for e.g. FIELDx in CATENTRY table) to store the flag. On the UI, the “Add to Cart” button can then be shown as “Preorder Now”.
Inventory – A retailer can decide to offer either a fixed quantity of a product or unlimited quantity for pre-order. When using the non-ATP inventory model, INVENTORY.INVENTORYFLAGS column can be set to 3 which means ‘do no check inventory, do not update inventory’, thereby allowing for unlimited quantities of the product to be ordered with no inventory in stock. When using the ATP inventory model, ‘expected inventory’ can be used to track inventory for a pre-order product. This is similar to the back-order concept available out of box in WebSphere Commerce where in an order is captured against expected inventory and the fulfillment is placed on hold till inventory becomes available and can be allocated to the order.
Order management – Orders having pre-order products can be marked as pre-orders by using any of the customizable fields (FIELDx) in ORDERS table. This flag is used to indicate that the fulfillment of such orders will happen at a later stage. When the business is ready to deliver pre-order products, then the inventory status of such orders can be updated and their fulfillment can be started.
Marketing – The marketing strategy for pre-order products is critical. Pre-orders are only successful if there are customers that are excited about the newly launched product and are willing to buy it without waiting for reviews and feedback. WebSphere Commerce has powerful precision marketing features that can be used to advertise the pre-order products and help build the buzz around them.
Pre-orders, subscriptions and recurring orders can be powerful order capture mechanisms which when implemented properly can greatly increase a retailer’s revenue. As we have seen, the business processes related to these features can be much more complex and nuanced when compared to normal orders. So, apart from the technical aspects, it is equally important to plan and build the right business processes for these features to be successful. It will take a joint effort between the technical solution architect and the business analyst to get there.
eBay does it. Amazon does it. Alibaba does it. Walmart, Best Buy, Flipkart... list keeps growing every quarter. Quite a few clients I have talked to think marketplace is a "low hanging fruit" and are compelled to try it. This is especially true in what is traditionally referred to as "growth markets". On paper, the allure is irresistible - after all who wouldn't love to make a commission for providing real estate for sellers and buyers to meet and transact. In an era of cheap computing resources, this sounds almost like free lunch. Only it isn't. I will explore some aspects of a marketplace strategy that are worth considering before going live with a marketplace solution.
First of all - the millennial retailers often don't realize that marketplace model has been around for ages. Focus has always been to bring together sellers and buyers on a online portal. For example, manufacturing units would set up portal to bring suppliers together and bid for contracts. Metal junction is a successful auction based marketplace for coal industry in India. Even a stock exchange is a marketplace - with stock brokers playing the role of the portal to allow sellers and buyers to transact. Why then we see a renewed interest in marketplace? Why are retailers queuing up to try this model now?
Reason for adoption
Most consultants and retailers agree that decision to go for a marketplace is based on following reasons:
- Survival or expansion strategy based on whether retailer does it to counter erosion of customer base to large online retailers OR reaching out to new customer base/market
- expand the catalog that a retailer offers through their online channel without investing in larger warehouses and logistics
- Only the sale transaction need to be hosted by the marketplace provider - procurement and fulfillment is executed by the seller.
- no risk of stale inventory or need for mark down - inventory risk is with the seller
- sometimes the legal, political, social and cultural restrictions makes it easier for companies to launch marketplaces to circumvent those restrictions
- earn fees/commissions -- though this is relatively small, this is almost all profit when compared the cost of hosting transactions
- shopper may feel they got more choices in a single portal(psychological angle) hence popular expectation is that it leads to customer stickiness
I am sure there are more reasons, but no major surprises here.
But retailers overlook or don't give enough importance to some important aspects. I 'll talk about four such aspects in this blog.
Four aspects retailers mustn't overlook
Main among them is that shoppers often don't differentiate between the seller and marketplace retailer. A poor experience with a seller often translates to poor experience with the marketplace and lead to a lost footfall. In other words, established retailers are putting their reputation on the line on behalf of sellers. So they better have a way to enforce discipline among sellers. One way could be to do random spot checks. I have heard of employees posing as shoppers to get proof of malpractice against sellers with questionable track record. But that is reactive approach. Marketplace provider would need to take up more responsibility. How about a tie up a logistics provider and have random quality check at pick up points? Companies often dismiss this as a risk that needs to be written off but overlook the intangible value of brand reputation. Differentiating on trust often wins long lasting clientele beyond finicky deal-hunters.
Other thing is standardization of catalog information - quality of information from many sellers is below par. They are probably managing all information in spreadsheets and have systemic problems of duplication, inconsistency or general lack of information. Unless the marketplace strategy has a solution to address this problem, shoppers are going to find anemic product details pages.
Sellers have a mix of mature and amateur back end systems. So the marketplace solution must be able to expose robust APIs for mature sellers to integrate with their back end systems while allowing gateways for upstart sellers to directly manage their business on the marketplace itself.
Fourth is differentiation. When marketplaces are dime a dozen, what makes a shopper select a particular solution. This is where opinions are as diverse as people. Some say cool User Interface, others say cool features, yet others rich content, SEO, social integration etc. Most impactful ecommerce solutions have a compelling reason for shoppers and sellers to stay with you. Companies that target specific market segments have better success than generic retailers. If the marketplace is targeting office supplies, home furnishings or industrial tools, it is more likely to succeed than a low-margin driven competitive general retail.
There are many more - a quick search on the Internet should reveal a more comprehensive list. But above four is what I found as prerequisites a retailer must think before opening up a marketplace.
The tool for success
IBM Commerce portfolio has a rich set of products that enables a end-to-end ecommerce solution to be built to custom order. It comes with a rich set of off the shelf features, however its real value is in building on top of the vast and deep foundation it provides. At face value people often dismiss it as not suitable for marketplace solution. However its data model is rich in its catalog, contract and organizational capabilities, almost all of its services are available as REST APIs and WebSphere Commerce and Order management tandem provides the deep catalog and order management capabilities that is par none. IBM's deep analytics portfolio is well known and is available as real time as-a-service solutions. What I would like is a unified business tool that can bring together and expose all these deep capabilities to enable execution of a marketplace strategy. With all essential backend services available as web services and REST APIs it is just a matter of exposing them over relevant UI