Modified on by Vani Mittal
WebSphere Commerce search provides enhanced search functionality in starter stores by enabling enriched search engine capabilities such as automatic search term suggestions and spelling correction, while influencing store search results by using search term associations, and search-based merchandising rules.
Starting from Feature Pack 7, search server is a new WebSphere Application Server runtime instance with an embedded Solr runtime, which includes a published RESTful based API to interact with the WebSphere Commerce runtime instance. It de-couples the store pages into the presentation layer and business logic layer of search solution. With this approach, the search and browse-only traffic from the storefront can be offloaded away from the WebSphere Commerce server (transactional server), to the search server. The B2C store component pages use the RESTful API to retrieve necessary data from the Search Server instance and present it to an online store visitor. With this architecture, the search servers can be scaled separately and therefore the search and browse traffic can be handled independently, creating a flexible and scalable deployment model that can adjust to various storefront browsing traffic patterns at different times or shopping seasons.
A well designed cache is an important aspect of a well performing search solution in WebSphere Commerce. In this blog post, we will discuss the different types of caching that can be implemented pertaining to search.
The following diagram gives an overview of the caching configurations, triggers and invalidations with respect to the search solution of FEP7+.
There are multiple ways caching can be implemented in the WC and search servers. Typically, in Commerce server, we have:
1. JSP cache
2. Servlet cache (REST)
3. Command cache
4. Data cache
In the search server, we have:
1. Servlet cache (REST)
2. Data cache
3. Solr native cache
Let us discuss the cache configurations that are relevant for the search solution in FEP7+.
Search is used in the storefront in multiple places, mainly for category navigation, faceted navigation, keyword search and auto-suggest.
In keyword search and faceted navigation, there is a lower possibility of reuse of cache entries since there can be a large number of possible keywords and facet selections. Hence, the general recommendation is to not cache full page responses when keyword search or faceted navigation is used. In this case, caching at a lower level can provide a better cache hit ratio. For example, instead of caching the entire search results page, it makes sense to cache the individual product thumbnails which can be reused across different facet selections and search results.
On the other hand, the default version of category navigation pages (that is, without facet selections) can be full page cached. As mentioned above, for facet selections, it is beneficial to cache the individual product thumbnails as fragments. This would have the thumbnails cached for reuse across transactions but when the category page is repeatedly hit, the page would render without having to fetch/read all the individual product entries again.
As discussed earlier, with FEP7+, there is a separation between WC server and search server. The catalog pages, for example, category and product pages have REST calls which hit the search server directly. In this scenario, there can be two possible options for caching category navigation pages:
1) If WC JSPs are being used for rendering the UI then full page and fragment caching can be used as discussed above.
2) If a different technology is used as the front-end (for example, a content management system) and WC is used only as the back-end, then it is a common practice to use the REST API provided by WC for the business logic. In this case, the response of the REST calls can be cached in the search server (discussed in next point). This keeps the separation of browse and transactional traffic intact and as a result one can fully benefit from the ability to scale up search/browse traffic separately.
When using JSP caching, there are well documented invalidation techniques which can be implemented. Details can be found in IBM Knowledge Center:
Setting up cache invalidation for WebSphere Commerce search
Caching and invalidation in WebSphere Commerce search
REST services use both server-side caching and client-side caching. Server-side caching is achieved using dynacache, and client-side caching is achieved using cache directives in the response header.
The topic of REST caching is well documented, so I will not go into detail and instead just provide resources that can be useful for understanding how REST caching works. This topic in IBM Knowledge Center provides information about REST caching strategies.
To see an example of how to cache a REST request, you can refer to this blog post by Marco Fabbri.
From FEP7+, the search infrastructure can be deployed in a separated WAS cluster and then the dynacache service can be enabled on every cluster member the same way you do on Commerce servers. The response of the search REST calls can be cached using dynacache on the search server itself. This is an excellent blog post by Marco Fabbri that describes how to do so.
The default invalidation technique used on the search server is different from the WC server. Since there is no scheduler running on the search server, there is no DynaCacheInvalidation job that runs in the background to perform invalidations. The invalidation on the search server is managed by the cache manager instead. The search server periodically checks for pending invalidations (pending invalidation events in CACHEIVL table) before executing each request. What this means is that each search server instance is responsible for invalidation of its own (local) cache instances.
WebSphere Commerce contains code to use the WebSphere Dynamic Cache facility to perform more caching of database query results. This additional caching code is referred to as the WC data cache. Similar to this, there exists a data cache on the search server that is used for storing internal runtime properties that are used by certain search features to improve overall performance. These data object caches are configured to use DistributedMap object caches.
This Knowledge Center link provides a list of search data caches.
As with the base cache, the invalidations in the search data cache are managed by the cache manager on the search server. This blog entry by Andres Voldman can be referred to for more details about the search data cache configuration.
Solr native cache
To obtain maximum query performance, Solr stores several different pieces of information using in-memory caches. Result sets, filters and document fields are all cached so that subsequent, similar searches can be handled quickly. It is important to understand what they do and to tune them to suit the actual environment and search traffic.
You can refer to Apache Solr documentation for more information about Solr native cache.
For additional guidance, you can refer to the "Optimizing Solr native caching performance" section in this article and the "Search caching" section in this Knowledge Center article.
There are multiple ways caching can be implemented in the WC and search servers. In this blog post, we discussed the cache configurations that are relevant for the search solution that uses the new search architecture introduced in Feature Pack 7.
Modified on by Vani Mittal
There are many order capture scenarios prevalent in the eCommerce space today. Some of the commonly used are – pre-orders, recurring orders & subscriptions.
These order capture mechanisms are appropriate for different situations and different kind of products. For example, pre-orders are great for allowing shoppers to reserve their own personal copy of a highly anticipated product.
In this 2 part blog post, we will explore the above mentioned order capture mechanisms and also cover the various points that need to be kept in mind while implementing them using the IBM Commerce portfolio of products. Part 1 (aka this one) will cover subscriptions and recurring orders, while part 2 (coming next week) will cover pre-orders.
Subscriptions & Recurring Orders
Subscription is a business model where a customer must pay a subscription price to have access to the product/service. The model was pioneered by magazines and newspapers, but is now used by many businesses and websites. Recurring orders are similar to subscriptions in that they also allow shoppers to buy the same items on a periodic basis without having to place orders repeatedly.
In case of subscriptions, the time period and frequency is determined by the retailer. For example, a publisher decides that a magazine will be offered on a monthly basis and at 1 or 2 year subscriptions. While in case of recurring orders, the time period and frequency is usually decided by the shopper. For example, a shopper decides that she wants 1 litre of milk to be delivered to her every day for the next 3 months.
There are multiple benefits of subscriptions and recurring orders, some of which are mentioned below:
Retailers benefit as they are assured of a predictable and constant revenue stream from the subscribed shoppers.
It increases customer loyalty by allowing customers to become greatly attached to using the product or service and, therefore, be more likely to renew when the time comes.
It reduces customer acquisition costs and allows for personalized marketing.
Subscriptions and recurring orders are supported out of the box by IBM Commerce solutions, namely, WebSphere Commerce and Sterling Order Management. However, there are certain aspects that a retailer must think through before implementing this feature as it can have an impact across almost all of the eCommerce subsystems.
Merchandising & Marketing
Before a product can be bought as a subscription or be part of a recurring order, it needs to be marked as such. This can be done using IBM Management Center for WebSphere Commerce or via the catalog data load process. The retailer needs to prepare this data and ensure that the additional properties, for e.g. subscription frequency and time period, are included in the catalog data.
Another aspect to consider is product life-cycle management. If a product is withdrawn from sale, what happens to the subscriptions & recurring orders that it is part of? One option could be to proactively cancel such orders and email the customers giving them a choice to order similar products (an opportunity for cross-selling and up-selling).
The payment for a subscription is usually captured upfront for the entire duration. On the other hand, for a recurring order the payment may be captured when each instance of the recurring order is processed. This introduces many complexities in the management of payment. For example, how do you handle the case where the credit card provided at the time of placing the original recurring order expires? In WebSphere Commerce, the recurring order will be cancelled if the payment for one of its instances fails during order processing. The customer can then be informed through an email or SMS and be asked to place a new recurring order. Additionally, there are 3rd party recurring billing management systems available that handle the many subtle complexities of subscription based payments, such as automatically emailing customers about credit cards expiring. Such systems could theoretically be integrated with WebSphere Commerce to handle credit card expiry detection and notification proactively.
Pricing & shipping
As mentioned in the previous point, payment for a subscription is usually captured upfront. So, the total price being charged to the customer includes the price for all the items for the entire duration of the subscription. Any future price changes will not affect the customer during the period of subscription. While for recurring orders, it is common to capture the payment as and when each instance is processed. In this case, it is up to the business to decide whether the original price applicable at the time of placing the recurring order should be honored or should the current price be taken into account.
When a customer places a recurring order, the retailer is in effect promising the customer that the items will be available for the entire time period chosen by the customer. However, it may happen that the items are out of stock at the time of fulfillment of an instance of a recurring order. What should be the business process in this case? In WebSphere Commerce, the out of box behavior is for the recurring order to be automatically cancelled and the customer to be notified of the fact via email or SMS. An alternative approach could be to delay the fulfillment of the particular instance of the recurring order till inventory becomes available. This may be a good option if inventory is expected to be available soon and the customer can accept a delay. The customer could also be given a choice to accept similar products that are in stock providing an opportunity for cross-selling and up-selling. These alternative approaches do not come out of box but could possibly be implemented with some customization.
For normal orders, different retailers have different policies on whether they allow an order to be modified by the customer once it has been submitted. A common practice is to provide a short window of time (maybe a couple of hours), before an order is sent for fulfillment, in which the customer can request for changes in an order via the retailer’s call center. Once the order is sent to the backend system for processing then only cancellations may be allowed. In case of recurring orders there is further thought to be given, since a customer may want to edit one of the instances of the recurring order or maybe the remaining unfulfilled instances of the order at some point of time in the future. In this case, it may be simpler to just cancel the recurring order and allow the customer to place a new one.
Returns, refunds & cancellations
The return process for a normal order itself can be complex depending on the process and business policies followed by a retailer, but there can be additional intricacies involved in case of subscriptions and recurring orders. Consider a scenario where a customer has subscribed to a monthly magazine for a year and has paid the total subscription cost upfront. A few months in to the subscription the customer decides to cancel it. How much refund is the customer entitled to? You would need to work out the refund by calculating the proportionate cost of each magazine issue and then deducting the cost of the issues that the customer has already received.
Providing customer service via the phone and email can be expensive. It is always a good idea to empower customers to manage their account and order information through the online store. Features like order history are pretty much standard these days. But in case of recurring orders and subscriptions, some additional functionality may be required to make self-service truly effective. Customers should be allowed to track and manage their subscriptions. They should be able to view the history of not just the order originally placed but also the recurring instances of the order. They should be allowed to cancel or edit the orders (within the boundaries of the business policies governing it of course). They should receive appropriate notifications of actions to be taken, for example, if a subscription is up for renewal.
In this blog post, we discussed the main business processes involved in subscriptions and recurring orders. In next week’s post, we will explore pre-orders and how they can be leveraged by retailers.
Modified on by Anbu Ponniah
In part 1 of this blog post, we discussed subscriptions and recurring orders, two features provided out of box by IBM Commerce solutions. In this 2nd part, let us discuss another popular order capture mechanism – pre-orders.
A pre-order is an order placed for an item which has not yet been released. These days many retailers use pre-orders to allow for shoppers to reserve their own personal copy of a highly anticipated product. Apple iPhone is a great example of such a product for which there can be a huge initial demand on release of new models.
Pre-orders allow consumers to guarantee immediate shipment on release, manufacturers can gauge how much demand there will be and hence how large initial production runs should be, and sellers can be assured of minimum sales. Additionally, high pre-order rates can be used to further increase sales.
Retailers can provide incentives like discounts, associated exclusive merchandise etc. to encourage shoppers to pre-order.
On the face of it, you might think that IBM WebSphere Commerce does not support pre-orders, but it provides enough flexibility to allow for support of pre-orders to be added with some customization. But first, let us explore the concept of pre-order in a little more detail.
The pre-order feature can be implemented in various forms and this can dictate the complexity of the processes involved. Let us discuss two of these variations:
Pre-order without payment
Some retailers allow for a product to be pre-ordered without capturing any payment. When the product becomes available, the customer can be asked to complete the checkout flow and make the payment. In this case, pre-order is really just a reservation process where a customer’s intent to purchase is captured with bare minimum personal details like email address, so that the customer can be notified when the product becomes available. There are some points to consider about this form of pre-order:
The reservation process can be implemented as a bespoke feature in WebSphere Commerce where in there can be a simple form to capture the customer’s details and a notification mechanism to inform the customer when the product is available to order. The customer details can be stored in a custom data model and a custom scheduled job can send out notifications either through the WebSphere Commerce messaging system or through an external email/SMS provider.
Since the customer is not paying for the product at the time of reservation, there is no guarantee that he/she will actually buy the product when it becomes available. Hence, this does not provide an accurate picture of the initial demand.
Again, since payment has not been captured, the retailer is not bound to offer the product at a certain price. The price may change when the product actually becomes available.
Pre-order with payment
An alternative approach can be to capture the pre-order in a way similar to how normal orders are captured. That is, allow the customer to complete the checkout flow and authorize the customer’s credit card for the amount. Once the product is ready to ship, then the customer’s credit card can be charged. One important point to note is that credit card pre-authorizations typically expire after 5-7 days, after which the hold placed on the funds is released. The customer needs to be contacted again for a fresh pre-authorization to take place. So, this approach is ideal if the time difference between when a product is available for pre-order and when the product is available to ship is between 5-7 days.
In some countries it may be OK to capture funds immediately for a pre-order product, as long as it is clearly communicated to the customers that they are being charged in advance for an item that will ship at a later date. However, there is a higher risk of chargebacks in this case. If you make people wait before you ship, a certain share of customers always end up asking for their money back.
Now, let us see how the various subsystems of WebSphere Commerce can be adopted to handle pre-orders.
Catalog – Products that are available for pre-order need to be marked as such. This can be done by using any of the customizable fields (for e.g. FIELDx in CATENTRY table) to store the flag. On the UI, the “Add to Cart” button can then be shown as “Preorder Now”.
Inventory – A retailer can decide to offer either a fixed quantity of a product or unlimited quantity for pre-order. When using the non-ATP inventory model, INVENTORY.INVENTORYFLAGS column can be set to 3 which means ‘do no check inventory, do not update inventory’, thereby allowing for unlimited quantities of the product to be ordered with no inventory in stock. When using the ATP inventory model, ‘expected inventory’ can be used to track inventory for a pre-order product. This is similar to the back-order concept available out of box in WebSphere Commerce where in an order is captured against expected inventory and the fulfillment is placed on hold till inventory becomes available and can be allocated to the order.
Order management – Orders having pre-order products can be marked as pre-orders by using any of the customizable fields (FIELDx) in ORDERS table. This flag is used to indicate that the fulfillment of such orders will happen at a later stage. When the business is ready to deliver pre-order products, then the inventory status of such orders can be updated and their fulfillment can be started.
Marketing – The marketing strategy for pre-order products is critical. Pre-orders are only successful if there are customers that are excited about the newly launched product and are willing to buy it without waiting for reviews and feedback. WebSphere Commerce has powerful precision marketing features that can be used to advertise the pre-order products and help build the buzz around them.
Pre-orders, subscriptions and recurring orders can be powerful order capture mechanisms which when implemented properly can greatly increase a retailer’s revenue. As we have seen, the business processes related to these features can be much more complex and nuanced when compared to normal orders. So, apart from the technical aspects, it is equally important to plan and build the right business processes for these features to be successful. It will take a joint effort between the technical solution architect and the business analyst to get there.
What are the benchmark reports saying
The IBM benchmark report, 2014 US Online Retail Holiday Readiness Report, showed that an average shopper’s attention time is dwindling as they are making quick and abbreviated visits to retailer sites. The average time on the site was measured to be ~7 minutes which is about a minute less over past 2 years, and the data showed that there is reduced trend on page views per session. These metrics become pertinent as the retailers have to continue their focus on bringing the right products to the customers in fewer number of browses and searches.
The WebSphere Commerce search, which empowers the search on the storefront, can be customized, extended and tuned for a retailer’s business. The retailers require to consistently monitor the performance of their search through the search hits, misses, conversion ratio, site abandonment, and accordingly tune and refine based on what the shoppers want and marry it with what the retailers want to sell on their website. We explore some of the use-cases which retailers will find useful in order to achieve a high search to sale conversion ratio.
The foremost design decision is around the search index schema. The retailer would want to add more fields that can be searched, these additional fields would be added to the index schema, for example, a sport clothes retailer will want include to include the sport-type as an indexed and searchable field. This new field sport-type can be then be used for impacting the search results, say for example, the order for the search results.
Search for conversions
One of the ways to increase the search hits and help site users find what they are looking for, is for the business build a strong set of vocabulary which represents their catalog set. This is made available by what is called the ‘Synonyms and Replacement terms’. The synonyms would help a customer say who is looking for a ‘fitbit’, show the different wearable, gadgets, activity tracker, pedometer that the store sells. The replacement would be used for terms, in the same example, if a shopper searches for ‘bracelet’ in a sports store, this will likely be replaced with ‘activity tracker’, ‘pedometer’ for the retailer.
The retailer would further want to refine the search results returned to the shopper based on inventory or brand partnerships. The WebSphere Commerce Management Center tool has a store preview feature which displays the relevancy score in the search results. This data provides the insights on what adjustments the user would make to have the desired result displayed. The retailer can accordingly tweak the result by using the search rule ‘Change Search Result Order’ and then iteratively get the desired set by viewing the relevancy information. Continuing with the example of keyword search ‘activity tracker’, if the business would like the fitbit brand to show up higher than other brands, then it will need to set its ranking order higher.
The retailer can exercise more influential rules than changing order using pre-existing options in the Management Center. For instance, they can specify that the ‘Top Results’ where on keyword search for ‘activity tracker’, the business can decide to show the latest fitbit product – Fitbit Surge as the Top Result. Retailers can infact influence a result where say the search result contains a specific product, and then move it to the top of the result. So with the example, if the search result contain ‘Fitbit’, then specify the Top Result as ‘Fitbit Surge’.
The retailer should use search statistics and experiment data to evaluate the impact and continue refining the search rules to have the maximum return on their shopper behavior.
Concluding with search extensions
Influencing search results is an important extension to the marketing and promotion tool. We have seen several ways in which the retailers can achieve this with ease. We will continue to delve into search extensions in the future posts as my colleagues will share specific extensions to derive enhanced benefits from the storefront search.
DEV/QA out of sync with PROD
Most often solution architects see their role wind down once the retail website they help architect, design and implement has been successfully launched. However, architect's deep expertise is needed for some day-to-day operations also. Often I have seen projects that fight through defects and scope creep to launch, but soon find that the development and test teams are way out of sync with how the system is being used by the business. Core of the sync-problem is data itself.
Let me elaborate that problem a bit here: It is typical for development teams to have fair bit of ownership of a "Dev" integration server environment (I will refer to that as DEV - not to be confused with development toolkit environment owned by individual developer, which I shall refer as TOOLKIT). Then test team would have their own environment to perform functional and system integration tests - I shall refer to that as QA. Next we typically have a preproduction environment that closely resembles production in terms of data and configuration - typically user acceptance testing may happen here, sometimes perf test may happen here.... I shall refer to that as PREPROD. Finally we have production environment that shoppers and business users actually use - referred as PROD. Note that each of DEV, QA, PREPROD and PROD will have their own authoring instance for business tooling, dataload etc and live/runtime instances for shopping activities. During initial launch, QA is often the environment where 90% of all data and configuration gets trialed and tested. PREPROD is where that reaches 99% with PROD having 100% of what is actually live. The percentage difference is also skewed towards marketing, promotion and content than with catalog. So in many ways, QA is a fair representation of what business users and shoppers experience in PROD. But once the site is launched, this equation changes drastically. PROD becomes the hot bed of activity - business users may go over and beyond processes defined by the solution team, adopt new processes....and end up creating data and configuration that lower environments never get any inkling about. If the technical leadership is not engaged, within the first year, QA drops from 90% to 66% as a representative of PROD. Soon development teams will see PROD defects where developers and testers swear that they have thoroughly tested but fails in PROD because that environment is now supporting new type of data that the code had not anticipated.
Problem is with communication. Technical teams needs to be actively engaged with business teams to understand ever changing landscape of retail processes. New attributes may be flowing into dataload - which none or if lucky, only a portion of IT team may be aware of. New price rules, new contracts, new layouts.... While most of it is supposed to be transparent to development teams, it is often not! Teams may have implemented some customization that get affected or affects work done by business user. Worst case scenario, as I observed with one customer, is for the business team to start coming up with their own independent processes, working with non-e-commerce IT teams to create solutions in other parts of the organization such as ERP systems. And end up choosing solutions that deviate from WebSphere Commerce recommended patterns of solution.
How do we deal with this
There should be a monthly interlock with business analysts and technical architects - where requirements are continually re-evaluated and validated. While meetings can solve one side of the problem, often truth is in data! Hence there should be some process put in place to ensure QA and PREPROD environments are updated with any non-transactional change that is introduced in PROD - in that sense almost anything that is affected through Commerce management center is a good candidate for replication in lower environment.
One way to ensure this replication is automated is via the use of Data Extract utility. Folks that are familiar with Data load, knows how the framework is geared to read up comma separated values in a specific format being mapped through an XML and used to created BOD objects to be loaded into WCS DB. Extract is just a reverse of that process - from DB data is read into BOD and fields from BOD are mapped to field columns that are written into CSV. The framework is exactly the same and the XML file used are very similar. In FEP8 - there is ability to add custom SQLs to the extract process by just inserting them within relevant tags - use of that for extracting promotions is documented in the knowledgecenter and a similar approach should work well for other parts of the DB : http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.data.doc/tasks/tmlextractpromotiondata.htm?lang=en.
Things to Watch out
One caveat is that attachments require an associated process to collect and FTP images/htmls that they may refer to -- these may have been directly deployed to web server or CDN - hence a separate process is needed to collect them and bring them down to suitable location for lower environments. Another hurdle I faced was in handling custom catalog associations that were implemented through custom code than through data model or BOD definition - so, the extract would need to mimic the same customization to ensure right information is extracted. Another was related to store and catalog organization model - especially if sales catalog and categories were directly created in PROD - then extra care needs to be taken to replicate those first. Lastly, inspite of the utility giving great capability, some data massaging may still be required.
It is worth investing in such a solution for any ecommerce project. For one, it helps to keep lower environments refreshed. But same process can be used to provide outward feeds from WebSphere Commerce - for example, sending catalog data to your ratings and reviews provider or your analytics provider.
Modified on by Shweta Gupta IBM
The shoppers have loved the search bar on the shopping sites and come to expect them to be their first friend to take them close to the product they intend to shop on the site. This is akin to the quintessential ‘Shopping Assistants’ in the marts, supermarkets, our our big or small fashion outlets. The shop may be earmarked and categorized, however if we are not familiar that particular store, we tend to get awed by the array of aisles and have an instant liking towards the store assistant who can either guide you the aisle/corner of the store for your desired product or even better escort you to the product that you are seeking. You would agree that some of these assistants have converted how many bags you carried back home from your shopping expedition.
The current state of eCommerce has come a long way and with most search engines offering a set of standard technical capabilities, one would expect to find our shopping search experiences almost similar. However, we are actually at length from this experience. While I was looking for studies on the subject, I stumbled upon a 2014 report on smashingmagazine.com, where they benchmarked the search experience of the 50 top-grossing US e-commerce websites, rating each website across a set of 60 search usability parameters, which you would agree is rather very fine grained. Offcourse a study from 2014 would require a revisit in 2016 at the pace one would expect our eCommerce sites to launch capabilities in an agile way, to stay competitive.
What does this mean to you for as eCommerce Implementation Project Manager?
I have observed that even though search is an important functional feature on eCommerce sites, the search relevance testing is continued to be missed out of the plan. The traditional functional testing, includes search and browse feature test, where the test team is responsible for a set of use-cases for checking that the search functionality works as expected. However, this approach largely misses out on the desired shopper experience for the search results or/and the desired search results from the business point of view. The User Acceptance Testing (UAT) phase would typically have the business users look at search relevancy, however it remains more of an organic exercise, amongst the overall umbrella of UAT, not getting the due attention, time, effort and most importantly understanding of the approach, input and outcome.
What does this mean to you as the eCommerce Product Leadership for your business?
This is definite a very continual exercise for you, just as it is to deliver on the business requirements via your eCommerce and mCommerce channels. You will be doing analysis at the ‘search hits’ and ‘search misses’, hopefully on a weekly basis, and working towards tuning your results based on the data your shoppers are feeding into you. The strong point is that because you have invested into WebSphere Commerce, you will find that the Business Tools allow you with a lot of flexibility and ease to achieve your results.
Let us explore an approach towards search relevance, look at a proposal for the test strategy and guidelines for your WebSphere Commerce Search powered commerce site.
Outlining the search relevance testing proposal and approach
The Project Management of eCommerce sites must include a sprint (or more depending upon the catalog size) which focuses on the search relevance. I have had discussions with several test managers on the topic and found them wanting for guidance around the ‘search relevance testing’. This prompted me to create an approach document that can be used as starting guideline for your eCommerce site.
- Your list of search terms: Work with the business to identify a list of ‘n’ search terms which you would like to focus in your initial iterations. Where n could be 50, 100, 200 so on based on the width and depth of the catalog, product items.
- Variety in search terms: The search terms should include a variety, where you have single words, multi-words, words with units like weight (think grocery), phrases with attributes like color (think garments), variance in how shoppers search for certain products, brands which your catalog does not support, misspelled products …
- Get into the shopper’s shoes, sandals: Leverage analytics data from different sources to identify the changing trends and shoppers behavior. Go out there to your favorite, and not so favorite retailers with similar catalogs, and learn what you like and what you would wish did better.
- Product title/description recommendations: The quality of the search result highly depends upon the way the catalog data is created and maintained. The natural search results from the search engine is based on the query relevance as defined in the configuration. One of the important recommendation which would also come out of the relevance activity is on enriching the product descriptions and other searchable attributes.
- Tester training: Your army of testers may not be trained to think on the topic of search relevance. Plan for a training and get them to think and experience shopping around the domain. This may be simpler for certain industries like grocery and fashion, however this may slightly trickier incase your site is selling spare parts, or heavy industry tools.
- Search Developers: The search developer needs to be part of the iteration to support the relevance tuning. This will require to look at the relevancy scores, and providing analysis on the result. The developer will also be responsible for making any changes which come through search configuration changes.
- Inputs to the search relevance activity: At the start of the search activity, you will have the list of search terms, expected results, synonyms, and search replacement terms.
- Outputs of the search relevance activity: At the end of the search relevance activity, you will list of search terms, actual result-set, if it meets expected results and no what is the delta. You will also have a more expanded/modified list of synonyms, search replacement terms and search rules.
- Iterative: This exercise should be iterative based on how is your development life-cycle. If the catalog upload, products, brands are staggered towards different releases, plan the search relevance accordingly.
- Input to the performance cycle: The outcome from this exercise should also be fed into the performance system as the search performance should factor in the recommendations which will be applied on the production.
The search relevance is a very subjective criterion which varies by industry, business, catalog managers, merchandizers, the current promotions, and most of all your product catalog.
The search relevance activity needs to get under the skin of the shopper to be able to identify the patterns and results. The business needs to be engaged very closely in the process.
I would hope that the approach outlined in this article helps you to work towards the planning exercise and also gives a starting point. Do share your experience on search relevance, as I look forward to the different ways in which we are out to achieve the business outcome from our commerce sites – that is convert the searches into baskets and orders.
I am attaching a basic version of search template. If you think this is useful, do let me know so that I can share other versions on the same.
Search Relevancy Template - 2016-FEB-SearchRelevance.xlsxView Details
Previous related blog: Customizing search for shoppers and retaining them as their attention time dwindles
1 - Changing the relevancy of search index fields
2 - Tuning multiple-word search result relevancy by using minimum match and phrase slop in WCS Version 7
3 - Search Rules
Modified on by Anbu Ponniah
E-Commerce in India is valued at USD $10 Billion in 2014 and expected to cross USD $15 Billion in 2015 (Report from IAMAI). India has barely scratched the surface of web ecommerce potential but is poised to go ballistic (in a good way) in mobile commerce. An interesting tidbit here is that over 5.5 million 4G enabled mobile devices have been sold but less than 100K 4G subscribers are there, mainly because of the limited availability of 4G network in India. But with reports suggesting that 4G Internet will be launched countrywide in 2015, it is not far fetched to think mobiles may soon monopolize shopping in India.
But rules of old world no longer apply. (Old = 5 years ago.... and I may be too generous here.) While there is general trend of preferring native apps over mobile web or hybrid apps, retailers really care about nailing just one thing : "Engaging User Experience".
What exactly makes a shopping experience "engaging"? The way I think, it is the complete shopping experience that presses all the right psychological, emotional, ethical and rational buttons to make shopping a gratifying experience at conscious and subconscious levels. The experience starts with how I hear about a retailer, my first impression on visiting the retailer, rich information centered around not just products, but also how products gel with my lifestyle (or my aspirations), ability to navigate, locate and visualize what I want, non intrusive but always on hand sales associate attuned to my level of knowledge of what I want to purchase, not having to shop-hop.... list goes on.
Yes, we need cool UI to give the initial wow factor, but we could do with functional UI that does not get in the way of the shopper. Functional does not mean run-of-the-mill, factory cut UI. It should still stamp the retail store's individuality. Beauty is important - world does not take the retailer seriously otherwise, but as mobile shoppers mature, skin deep beauty will ring in hollow. The days of having cookie cutter approach to providing shopping experience are numbered and coming up with custom cookie cutter pattern alone will no longer set a retailer apart. When I visit an e-commerce retailer in my mobile, I expect the product recommendations to match my previous browsing and purchase history. I only notice it if they don't. Rules of engagement are evolving and rules based engagement, any which way we package it, is no longer dynamic enough. The rules engine now needs to think. Not only think, but reason! And this intelligence must be available for both shoppers (as a guidance/advisor/sales associate) and for retailers (to enable marketers and merchandisers digest large quantities of data and discover patterns in a meaningful manner).
IBM has forayed in to the world of cognitive computing and has thrown open Watson analytics with natural language supported exploratory visualization for business associates that takes analytics to new heights without the steep learning curve. The free trial is available for everyone to give this a try. Free version has some limitations : For example data storage is capped at 500MB, record size cannot be more than 100K rows and 50 columns. But it will give a great insight into the promise that this new age solution holds. Imagine being able to ingest scores of spreadsheets from your ERP systems and other analytics tools and gain insights using natural language queries.
How do you do it? Visit IBM Watson Analytics, use your IBM id (register one if you don't have one - it is free). You will receive an email to validate your account. Once validated, you are taken to a landing page where you can upload your data - for example recent sales data. Once uploaded it suggests starting points for exploring your data. You can also view the useful sample tutorials for tips on analyzing your sales or marketing data. These insights can now lead you to make decisions that go over and beyond regular formulaic decisions. Once these insights are available, WebSphere Commerce solution can now factor these relations into their recommendation or guidance engine.
I played around with just a dump from a sample ORDERS table of WebSphere Commerce. My intention was to find my order spread by currency (aka region) in a given time frame. It was as simple as dumping a report about all sales order into a CSV, dragging and dropping that CSV onto IBM Watson Analytics and going through various recommended and custom visualization schemes.
I could type in natural language questions and it responded with graphs. As an example, I could make out that total sales value indicates that my store's sales in Japan (presumably due to value of Yen) far outstrips my sales anywhere, but number of orders say my USA sales was easily more than sales of all other countries put together. I could then drill more into the data to understand average value of sales from each region and type of products selling. This helps me tailor my recommendations, promotions and campaigns more effectively. Many analytics tools may do this in one manner or the other, but IBM Watson does this while supporting natural language queries putting it within reach of any retail business user and not requiring a PhD in data science.
I have barely scratched the surface of this tool's capabilities. Next time you wonder why you are able to navigate to a page showing your preferred brand, color and size of t-shirt within a couple of taps without ever disclosing this information to the retailer, you may have to thank Watson for it.
Has anyone played around with Watson analytics? What are your thoughts? I welcome you all to share your experience.
As an architect/practitioner growing up in a world of enterprise architecture where data is rarely seen outside the ambit of relational schemas, NoSQL brings with it a breath of fresh air - simultaneously challenging previous beliefs, promising exciting opportunities and bringing its load of inevitable baggages. In this blog post, I shall walk down the discovery path of how, as an architect, I see NoSQL complementing or supplementing or replacing traditional solutions. The perspective is that of an architect trained in a world of stack architecture and transactions, so this may not resonate well with advocates of either technology. However it highlights opportunities and challenges that would translate into solutions that everyone can relate to. This will be a two part series - part 1 (this blog) will set the context and part 2 will detail one solution possibility.
Defining NoSQL as a comparison of SQL databases is no longer straightforward. Lines are blurring. For this blog, NoSQL databases are those that have flexible schema capabilities, store their data as one of the following formats:
and primarily support a non-SQL query interface. They may or may not support some or all of ACID (Atomicity, Consistency, Isolation, Durability). Internet is filled with information on NoSQL, their features and extensions, so I will try not to replicate this information. Instead let us focus on what it means for ecommerce - starting with WebSphere Commerce.
Let us first set some background here. First order of business is to view information managed in an ecommerce solution in a perspective unconstrained by traditional views. For example, one may view data handled by WebSphere commerce from the following perspectives
Catalog perspective -- Attributes, structure and administrative information related to products. Here the term attributes is used in the traditional sense of the word and not restricted to WebSphere Commerce's definition of attributes. Hence, the information included in this set of data are information that describes products for external consumption, information about products for retailer's own management, information about structure and relationship between products and information that are needed to administer or manage it all. Entire Catalog data model of WebSphere commerce falls under this bucket. Importantly, this does not include price and inventory.
Order perspective -- Attributes, structure and administrative information about shopping carts and orders. Same definition here too - includes information from shopper, retailer and administrative perspectives. This, in most cases, constitutes the primary transactional data for a retailer. Information about orders will need information from other sets of data such as catalog, payments, pricing, marketing etc. It may be tempting to already think relational (foreign keys) here. If we hold off the temptation to bucket perspectives of data into relational concepts, we may find it easier to see alternative views.
Customer perspective -- Information about shoppers in our retail solution. It includes login and password for registered shoppers. It includes addresses and payment information for all shoppers. Again, it will have overlaps with orders and catalog perspectives.
Marketing perspective -- Information needed to market or promote one or more products. It includes information that defines the rules, information that should be displayed to shoppers and business users and information needed for business logic to evaluate and apply them.
Above is just a short list that this blog will use for illustrating a point of view and is not an exaustive or correct list.
For each perspective, we see the following are immediately apparent:
some data is needed to display to external world versus some information is for internal use.
some data is static, others not so static
some data needs high degree of ACID support others can live with cursory checks
With this backdrop, let us consider the following use cases:
UC1: A shopper must be allowed to define and manage any number of personal information about themselves.
UC2: A shopper must be shown personalized messages based on personal information they have shared
Traditional solution requires some innovation to allow users to create their own free-form information - so we may end up with a lot of duplicates or near-duplicates. This information can soon grow out of hand - imagine if you anticipate a million customers and dozen attributes on average for each. So natural inclination is to push back on such a requirement and instead box the attributes to retailer defined values such as hobbies, favorite XYZ (sports, color, place...) etc.
Then, in WC, one could define a service to adding information to Member attributes part of the Member data model in WebSphere Commerce, customize promotions/espots logic to allow targeting based on member attributes and then during customer interactions check if member attribute driven espots and/or promotions can be shown. This is one way (and one of our future blogs will detail this). Other way would be to define some custom table and store these as key-value pairs (in fact storing in MBRATTR has a key-value sense to it).
NoSQL gives us an opportunity to think beyond the constraints of data model here. After all if the information naturally fits the key value pair model OR can be modeled as a "Personalization Document" of a customer, why not think NoSQL here?
The solution must be capable of creating this information, allowing shoppers to maintain it and for internal commerce logic (such as promotion/pricing components) to use it in their transactions, specifically as part of existing commerce transactions. And it need not require remodelling entire WebSphere Commerce database and business logic to use NoSQL. I will be advocating a layered approach where each type of data model is used to take advantage of their core strengths while minimizing the custom code needed as much as we can.
We will explore that solution in my next blog.
In part 1 of this blog we had seen how data that is handled in an ecommerce system could be bucketed into various perspectives and categorized into the role it plays or supports. And we discussed two use cases which I shall restate here:
UC1: A shopper must be allowed to define and manage any number of personal information about them.
UC2: A shopper must be shown personalized messages based on personal information they have shared
In this blog let us get down to the solutioning decisions and see how the puzzle pieces fall into place.
For properly solutioning this we must revisit the data that we will end up capturing and understand what qualities the data store must support. Apart from the obvious needs of storing, retrieving and updating data the choice should allow for discovery and exploration of the data over and beyond the relationship established through the data model. Let us discuss this need through a couple of examples of the second use case.
Beyond saved relations
Example 1: If the retailer wants to show a personalized message such as "Your favorite sports team is the most popular team among shoppers in our stores". For this we must be able to discover if indeed the sports team entered by the current shopper is mentioned by most of the users. A simple document store with some faceting capability can give us this information.
Example 2: On the weekend before Valentine's day every female shopper who have one of their favorite colors as red, pink, dark brown or chocolate will be shown an marketing widget that lists a set of select apparel accessories.
The above examples may or may not make sense as real use case examples, but are taken to illustrate two distinct problems - one is that of aggregation and other is that of segmentation. Similarly, from the retailer's perspective, this design should also support deduplication and standardization --- let me call this using a term that is superset of both: categorization. Without getting a view of categorization, merchandisers and marketers won't be able to utilize the rich data gathered. And we need robust analytics to help us translate the data into trends and relate them to behaviors that a retailer can target for their merchandising and marketing decisions.
Now NoSQL can help us solve the shopper perspective well and provide some help regarding categorization. It can integrate with an analytics solution for the last bit. The data we gather regarding shopper's personal profile should now be used to personalize his/her shopping experience. And the rules of influence could be self contained within the profile information or may need to be executed in the context or in combination of information from catalog, orders or marketing sub-systems. This is actually one of the reasons why we traditionally choose a relational data model solution for this.
And this leads to the first decision point - the data must be available for direct queries from client systems and from ecommerce business logic systems and should be map-able with existing relational data model. This is just to say that when I save information in my NoSQL data store, I will include a few key values such as member_id, store_id, catalog_id, address_id and language_id.
Second decision point is to mirror relevant promotional, marketing and search rules information from relational data store to the NoSQL data store. This is particularly easy since this information is available in XML format CLOBs in WebSphere commerce and lends itself to easy import into data bases like Cloudant or MongoDB. Obviously, every time a stageprop happens, a process should push updates from relational model to NoSQL model.
Third decision point is to extract information from NoSQL data store for analytics and feeding this back into the system of record for factoring it in promotional, pricing, search or merchandising rules. Now it may be attractive for analytics to actually run directly on the NoSQL store itself - but worst case we need a process to export, FTP to analytics system and import into its data store. I advocate the analytics to instead run on the NoSQL data store to become real time (subject to performance considerations) and create outcome of the analytics also in the same data store.
Fourth decision point is to keep authorization, contract entitlement and organizational logic away from this NoSQL data store. These can be enforced at a different layer.
Musings on WebSphere Commerce and stack options
Most of us know this - WebSphere Commerce is an enterprise Java application that runs on top of WebSphere Application Server that runs on an Unix OS and uses DB2 or Oracle as its persistence layer. Technology space is constantly evolving and new choices are available for each layer that WebSphere Commerce depends on. There is often doubts in the mind of retail IT departments on whether they can safely adopt an alternative technology in any of the stacks. This blog is a collection of my musings on that topic:
By default, everyone assumes that WebSphere Commerce (WCS) runs on an IBM stack! But a quick look at detailed requirements page reveals some non-IBM components in the stack: Windows and Linux on Intel based processor systems are two key differences in the operating system layer (bringing in flavors of windows server, Red Hat Enterprise linux and SuSE into the mix), while Oracle is a key difference at the database layer. Otherwise, the list looks like a pure IBM list. However, I have seen customers experiment with more choices than the above. Let us look at some variations.
1. Web Server Options
IBM HTTP Server v18.104.22.168 corresponds to Apache web server 2.2.29. Some customers choose Microsoft Internet Information Services (IIS) because they are using that solution in other parts of their IT enterprise. Similarly, other choices such as NGinx or Apache web server is not unknown. Of course, the default choice of IBM HTTP Server comes with plugin-configuration, predefined configurations for SSL communication between WAS and web server layers and supporting both unmanaged and managed WAS nodes. So using a different web server solution would require IT team to perform some manual configuration. Reasons for choosing an alternative web server is usually because they already use a different web server for their all enterprise needs or are after specific capabilities of a web server (for example, Apache web server version 2.4's bandwidth limits for clients is needed).
2. Hosting and OS options
WebSphere Commerce is supported on popular enterprise *nix options such as AIX and Linux on x86, AMD and PPC architectures and virtualization options such as PowerVM. However, customers often wonder if they can deploy WCS solutions on their choice of cloud providers. The short answer is "WCS is functionally compatible to most cloud providers". Cloud providers differ on the virtualization technology they use - so product will provide a full set of functional features, but the sizing needed to run the solution may differ between the providers. As long as the IT team works with that provider to adjust their techline sizing, this should work. Of course, going with IBM Softlayer means the techline sizing from WCS is straight forward and support is at one place. IBM Commerce on Cloud DevOps services has hardened reference architectures and devops patterns used in multiple client engagements. We also have IBM Urbancode and Chef based setup, configuration and deployment management solution that automates many of the mundane tasks in addition to supporting common solution architectures. And as per http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.install.doc/refs/rigsupportSCCI.htm: "Depending on your virtualization configuration and use case scenarios, the cost of operating in virtualized environments could be more than 30% of your overall native, non-CCI IBM WebSphere Commerce available capacity".
On operating systems, if the database and WebSphere Application Server runs on that OS, theoretically WCS also runs. However, from WCS product literature, it is clear that only a handful of operating systems have been certified (tested) for production use. So, while WCS "should" work (and probably would) and knowing IBM, it will probably provide a best efforts help for solving any problems reported on those operating systems, companies adopting any unsupported flavors of Linux should have a plan in place to handle such situations. For example, having at least a small farm of machines in supported OS flavor helps in differentiating a problem as specific to OS or configuration vs WCS product. I haven't personally seen clients experiment much here.
3. DB options
DB2 and Oracle are very popular amongst WCS customers. What if someone wants to experiment with some NoSQL or in-memory DB options? Again, typically with IBM, these will not be supported but if you open a PMR you would find that IBM will help you on a "best efforts" basis - note that this is the same level of support provided for your custom code. I have seen customers with good IT teams experiment with such options. One of the earlier posts in this blog discusses a pattern where some of the information is moved to a NoSQL DB. Similar exploration on using alternatives for parts of transaction processing - for example if you keep all registered shopper information in an elastic search solution.
4. Queue, Transformation, ESB etc
WCS integrates with multitude of IBM and third party products - IBM OMS (Sterling), CPQ... and other OMS and ERP systems are common. Point to point queue based or ESB based integration patters are popular. IBM MQ and IIB are supported. IBM has a long history of supporting integration with SAP based systems, in fact some say better than Hybris. What if you want to explore something like Apache Kafka? Like any solution built on loose coupling - by implementing a few Java Adaptors one can integrate over such technologies too.
5. Monitoring and managing utilities
Any OS level monitoring and managing utilities are of course supported - that is a no-brainer. WCS tracing and logging framework is built on WAS's jRAS framework. While any log monitoring or log analytics tool that reads those logs is easy to support, if you need to replace the complete logging solution with some custom logging framework, you would need to switch out some foundation jars from WCS and WAS. This will impact your support statement. One option to be "in support" is to extend needed classes and override the loggers at all critical places. WCS provides many extension points which can come in handy here.
It should be mentioned that another layer that often pops up during solution discussion is "Search". As we know WebSphere Commerce Search has Solr 4.7 (lucene) as its engine, but there are many add-ons that integrates search into mainstream e-commerce. So not just usual search and catalog navigation uses Solr, but price and inventory are available there, rules to modify search are now bubbled up to business tools and search is integral part of precision marketing too. However, some customers want to improve the core search capability itself and that is possible. For example customers may want to upgrade to a later version of Solr to take advantage of a specific filter or add custom extensions to support specific language or capability (such as phonetic search). Since the architecture is loosely coupled, and since Solr configuration (as kept in solrconfig.xml) is visible to IT teams, modifying that layer is straightforward.
7. Utility libraries
Lastly, I would like to mention about multiple third party utility or special purpose libraries that become part of the solution - these are typically located in WC_TOOLKIT/workspace/WC/lib directory in our toolkit. I know of instances where projects needed string processing capability available in a later version of a third party library. That is also the place for adding solution specific additional third party libraries.
I have just shared my musings about some of the common alternatives that come up during discussions with IT experts involved in e-commerce solutions. There are numerous other layers Note that none of the above should be interpreted as a support statement from IBM - these were just examples of options explored to various degree in various contexts. As always, check with your IBM support personnel before venturing into something different.
The price pressure
Twenty first century shopper has so many options. Unless a retailer is a niche player, it is a shopper's world. While retailer has many tricks up their sleeve to bring eyeballs to their e-commerce website, keep them there, convert into a sale and bring them back, the de-facto strategy that all retailer's adopt is the ubiquitous price strike through to give the shopper a sense of getting a good deal on the goods. Price used to be decided by business teams pouring over their analytics reports, industrial demand/supply reports and peering into the crystal ball of future trends. It still is done like that in many places. But what has changed is that price gets revised (usually downwards) more frequently than before. What used to take months and weeks earlier now happens matter of days and hours.
Let us say your client has a pricing strategy in place - and have the experts coming up the right prices to show for a product each hour of the day and change that every two hours . What would be your strategy as a technical architect and adviser?
Options for price updates
In technology, every thing is possible but may not be practical within the constraints a particular solutions may be operating within. Understanding the when and what to compromise on is the trick.
So what is possible in WebSphere Commerce?
If you get your price as CSV from your pricing engine, then refer here for how to load up your price: . If your pricing engine is IBM DemandTec, then see DemandTec with WebSphere Commerce.
For more control BODL service asset could be used to create a custom load handler.
Or if CSV file is not your cup of tea - say you need this as a true "feed", a look at the current "Content Management system web feed" utility provides some clues. At the end, same data load framework will be used to load information into WC database, but ability to establish connection to the feed provider and transform the feed into WC specific business objects is the real work here (how to do this will be taken up in a future post).
Or may be old style (now deprecated) massload may be used.
Some may use a form of database export/import.
Of course, WebSphere Commerce may be the price master so that business users create/manage price using Commerce Management Center.
Any which way we load the price - the question considered in this blog is : How often should we propagate updated price to our live site?
Real question: How many times?
A change in price needs the following to happen:
1. Change must be propagated from authoring DB to live DB.
2. Search index must be rebuilt (if using priceMode = 1 or 2)
3. Dynacache must be invalidated and warmed
4. CDN cache must be invalidated and warmed
All the above takes time. I don't mean hours - but we need to be cognizant of the number of minutes each of this takes.
If the price to be shown is dependent on conditions (price rules) or if the retailer wants to define price reductions via promotions because that is how they see price discounts, then some pre-processing will be necessary to decide which price to index and how to cache your price by associating the correct combination of customer segment, promotions and price rules.
And cache invalidation and warming takes time unless you have a caching appliance like WebSphere eXtreme Scale solution that has excellent cold-cache capabilities that renders warming unnecessary.
The price change also affects in-flight transactions. A robust business logic should inform shopper of the change in price up to a point in their shopping journey and otherwise accept the order with previous price. Visible price change during checkout affects shopper's trust on the retailer. Frequent switching of price may just make people wait longer and delay their buying decision. But not changing the price for long could expose the retailer to competitors under cutting them. Finding the right balance is more of an art than science.
One customer agreed for twice a day price change. Another was OK with hourly updates. Most challenging one was a customer who insisted on instant change and that too on average 50 times a day. Things we need to work with retailers is batching as many updates together as possible and helping the retailers work out some form of prioritization within their process. For example, price is wrong (missing a digit altogether) warrants instant reaction. In other cases, it may not be that urgent. Also, solution could be found by a different strategy altogether. For example, reaction to under cutting could just be advertising a policy of price match with reputed retailers rather than getting into a no-win war on price. Winning the retail race on core differentiators such as engaging shopping experience, after sales service etc is more profitable.
Random closing thought
One random, but relevant thought: What is the deal with strike through price? Some retailer may just want all their prices to be struck through with a lower price right next to them. And the "offer" price may just be one cent or one rupee less than the "list" price. Such tricks may just train the shoppers to filter out the struck-out price altogether, robbing the emotional benefit of scoring a deal. Often the actual value of saving in numbers is more transparent than percentage saving.
We are a group of architects and subject matter experts who collectively have 25+ years of experience in executing ecommerce solutions over WebSphere Commerce across the world. This blog is our way of sharing this experience and learnings to a wider audience.
We would be covering a wide range of topics -- most of them will be in the context of interesting requirements we encountered over our services career along with a generous sprinkling of insights into the product's capabilities and limitations that enabled us to solution those requirements or overcome the limitations.
We are here to share and learn - so would love to hear back from you!
Modified on by Anbu Ponniah
Authored by : Raghuveer P Nagar, Senior Architect for IBM Order management
By going through the article, readers can expect to know what IBM UrbanCode Deploy (UCD) is, why one needs it for IBM Order Management deployments, how one can access it, how it is useful for day-to-day IBM Order Management deployments and few of our learning as an implementation team.
IBM UrbanCode Deploy
IBM UrbanCode Deploy is a tool used for automating application deployments (in both on-premise and on-cloud scenarios).It simplifies, and standardizes, the application deployment process across environments, throughout the development and maintenance of the application. Additionally, it offers clear visibility (into what is running where, differences across the environments, who changed what, etc) and rollbacks of the deployed applications.
IBM Order Management
IBM currently offers its multi channel commerce Order Management System (OMS) as both on-premise and on-cloud software. The on-premise software is commonly known as Sterling Order Management. IBM Order Management is software as a service (SaaS) offering, which is a cloud-based OMS service offering (on the IBM SoftLayer infrastructure). For managing IBM Order Management deployments, UCD is used as the deployment automation and self-servicing tool.
On-boarding and Access
IBM Order management Environment cannot be accessed directly. To access the environment, that is, to access UCD and manage the OMS application, you first need to connect to jump host. Jump host connection provides ability to bypass firewalls that can prohibit internet services. Refer here for detailed process of connecting to IBM Order Management environment using jump host.
Once you are connected to IBM Order Management environment, you can access UCD. In addition to deploying application, UCD is used for complete implementation process, such as deploying changes and fixes into the environment. Refer this link for more information on UCD access.
After accessing and logging into UCD, you need to download developer toolkit environment. Developer toolkit is replica of the IBM Order Management environment, which enables you to do required customizations on your development environment. Go here for step-by-step process on downloading the developer toolkit from UCD.
Key OMS Deployment Features in UCD
UCD provides insights into IBM Order Management deployment audits, status and history. It helps in deploying IBM Order Management application in no time and also provides wide variety of process management options to also administer the deployment. The following are few of the key OMS administration processes which can be executed through UCD:
- Deploying custom code, like deploying custom jar files (say, the jar files having custom User Exit implementation and classes for Web Services)
- Exporting Configurations/CDT, which includes exporting CDT from one environment (say, the master configuration environment) and comparing it with another environment (say, the development environment)
- Importing Configurations, that is, applying CDT exported from another environment
- Starting and stopping application server
- Managing Agents, including starting, stopping and triggering agents
- Managing Queues, such as configuring a JMS queue and clearing contents of a Queue
- Managing OMS Application Server Logs, like exporting and configuration driven archiving of the logs
- Installing 3rd Party Certificates, for various integrations and implementation features
- Importing Customer Overrides (in customer_override.properties file) to the database
The processes mentioned above remain same for OMS related product offerings like IBM Call Center for Commerce as well.
While working on UCD for IBM Order Management, we have experienced many common application deployment and management issues. The following is a list of few of those, with tips based on our experience:
- Automatic Process Monitoring - UCD has email configuration to inform the status of particular process through email. For example, it can send emails when agent startup fails and build failures. As one of the first things, one should ensure that this configuration is complete.
- Application/Code Build Failures – While building custom code in the development environment, you may get errors like “Not able to access the file” and “Not enough memory” (in the build logs), say when the build involves compilation of the custom Java classes. It is important to know and remember that these issues need access of the installation directory, which is with the cloud support team. So, one should raise a cloud support ticket on facing such issues.
- Infrastructure Related Errors – While running a process, an infrastructure issue is reported, say “Database connection parameters incorrect". This happens during a time window when the password is reset in the database server, but an implementation script is run before the same password is updated in the deployed application. One should raise cloud support in such scenarios also.
- UCD Options - For administrators who are new to UCD, it will be a good idea to spend some time in getting aware of the available OMS deployment options by exploring various user interface (UI) options of UCD. This may sound unusual, but, think of the productivity loss which you may have when you reach cloud support or go through the documentation just to know that the option is already available on the UCD UI!
This is what we have today on the IBM UCD tool for OMS deployments on cloud. We will share more experiential information in our upcoming blogs in the OMS-on-Cloud stream. Stay tuned for our next article in the series, which will be on Developer Toolkit.
Modified on by Shweta Gupta IBM
Who owns and understands the security posture for the organization?
I would start with making an assumption that if you were a WebSphere Commerce architect, specialist, or a developer, you will agree with me that it will be foolish to attempt to respond to the loaded question – ‘Is our eCommerce site safe?’
This would be something that is under a purview of a Chief Information Security Officer (CISO). If you are interested to see a sample of the job responsibilities of a CISO profile in a commerce organization, have a look at this linkedin entry
The infographics on cyber-attacks from Threat Intelligent Report
IBM X-Force Threat Intelligent Report published by IBM, shows the 2015 security incidents and the retail industry is the second most attacked industry.
The components of a commerce shop that would typically need a CISO’s attention
There are several areas of a commerce shop that require to be at a desired level of security maturity model.
- To start with there is a need to have a Governance structure in place. This would include the processes established inorder to manage and report on the compliance
- Focus on the People aspect of the business, that would talk about identity and access management
- Infrastructure security would cover several pieces, networks, servers and the endpoints
- Protection of Data is key and this may entail industry standards and compliance
- Application security will include all the different applications and the security levels would be different based on their exposure to the internet
How does IBM help companies and businesses to be secure, and this would apply to eCommerce
IBM has a strong security product portfolio, business and product offerings. There is also an online set of Quiz Questions to help an organization evaluate their understanding of the security risk.
The following areas of focus, will help the security stakeholders to engage with IBM
Your organization’s security program - Understanding risks, establishing the right policies and programs and having a strong, cohesive team to implement changes is critical to building a next-generation security environment. IBM® Security can help you design an integrated framework to simplify the challenge of securely protecting the enterprise.
Tackle Advanced Threats – You are uncertain on the advanced threats and how to be prepared to manage them. IBM® Security can help you with a view the security landscape with a wide-angle lens to thoroughly understand the origins and distinctive features of attackers. Fraud, endpoint and data protection. Security intelligence and analytics. Incident response. They're all part of a comprehensive approach using extensive research and detective work to pinpoint, outsmart and stop them.
Protect critical assets – You have an internet facing business, with potentially millions of customers, partners, vendors accessing the systems. IBM® Security can help you leverage analytics and insight for rapid detection and response to protect critical systems and records.
Further, IBM provides services where the security experts will do an assessment of all the aspects of an organization from a security maturity model. This will allow the CISO to understand the security posture for the organization and the desired posture for each of the components mentioned above, like people, data, and infrastructure. This is an important step in securing organization and business against the cyber security threats.
Security of the WebSphere Commerce empowered storefront
The CISO’s governance and strategy will form the backbone of the eCommerce security. This will ensure that each of the applications follow the guidelines and demonstrate adherence to the security framework.
The eCommerce program manager will own the security aspects of the WebSphere Commerce implementation. It will therefore be the individual responsibility of every member involved in the implementation to understand the application guidelines for WebSphere Commerce and consider them at design, development, testing and deployment phases.
The product KnowledgeCenter has a topic in itself on ‘securing’, that details on the topics of authentication, authorization, session management. There is also section to cover security standards ‘National Institute of Standards and Technology’ (NIST) that provides guidance on the use of stronger cryptographic keys.
The Payment Card Industry (PCI) Data Security Standard (DSS) is applicable if your eCommerce system is capturing and holding credit card and payment-related data into the system. The product has created a summary of specific configuration that are required in the WebSphere Commerce implementation in order to comply with the PCI-DSS.
Having pointed the readers to the Knowledge Center for a lot of product documentation on the topic, I am also doing a PART 2 of this series to cover a simplified version of the aspects of the WebSphere Commerce product’s security considerations. Look out for the second part for a quick and clear insight areas of Access control, Hardening against common attacks types and Data security.
Credit: Thank You Sreekanth for your inputs and review on the topic.
Sreekanth Iyer is an Executive Architect with the IBM Cloud (CTO Office) team and works on defining the technical strategy and development of IBM Cloud Security. He has over 20 years of industry experience and has led several client solutions for Telco, Electronics, E&U, Govt, BPO & Banking industries. He is an Open Group Certified Distinguished Architect, IBM Master Inventor, Certified Ethical Hacker and Member of IBM Academy of Technology.
Modified on by Shweta Gupta IBM
In the PART1 of this series, I introduced the topic of security around eCommerce from a perspective of being in a Hot Air Balloon (which would be 2000 lifetime floor badge if you wear a Fitbit). You will want to read the Part 1 to get a glimpse of CISO’s role in your organization’s security, and introduction on IBM’s security services. However, if you want to directly dive into securing your WebSphere Commerce application, read on this article.
The WebSphere Commerce security model constitutes of setting up the access control, through authentication, authorization and policies. It provides through some of the main security standards applicable to the eCommerce industry. It also covers managing against common attack types. However, as outlined in the PART 1, the malicious attacks continuously innovate themselves, and therefore business needs to evaluate its advanced threat management capability on an ongoing basis.
In this PART2 of the series, I am going to cover 4 areas from the application perspective.
- Access control
- Hardening against common attacks types
- Data security
- Security Standards
Authentication: WebSphere Commerce issues an authentication token. This token is associated with a user on every subsequent request after the authentication process. There are account policies for password, account lockout, timeout that govern authentication.
Authorization: Access control policies is leveraged for enforcing authorization in Commerce.
Let us look at some some common access control areas that are prominent from en eCommerce perspective.
- Logon - Enforcing too many failed attempts: Set up an account lockout policy for different user roles in WebSphere Commerce. This is to enable enforcement of a lockout in case there is malicious attempt to break a password. There is also deterrents by setting up delays in between of consecutive unsuccessful attempts.
- Logon - Prevent privileged users from logging in externally: It is desirable to dis-allow access to the privileged users of WebSphere Commerce application, like the site administrator or a customer service representative. This need is primarily from a security perspective where one would not want a hacker to gain privileged access and act with malicious intent. WebSphere Commerce V 8 Mod Pack 1 has provided with a security configuration in wc-server.xml that can be enabled and then customer headers can be used to define roles to disallow the logon. There is further customization that one can do to address the security needs for logon. Earlier to this introduction of the feature, in earlier version of WebSphere Commerce (v7, v8), the article provides sample code to achieve the functionality.
- Securing the WebSphere Commerce search server: This can be achieved at two levels, WebSphere Application Server (WAS) Administrative Security or WebSphere Application Server (WAS) Application Security. The WAS Admin Administrative Security is recommended, as the application security is more expensive in terms of performance.
Protecting custom enterprise beans, data beans, controller commands, views: Primary resources should be protected. If a user is allowed to access a primary resource, the user would be allowed to access its dependent resources.
Access Control and Data Beans - The following article discusses the best practices for access control with WebSphere Commerce and how to use Data Beans securely
Hardening against common attacks types - A view on the common attacks types and guidance on handling them
- Changing URL params to break the system: WhiteList data validation on a URL is disabled by default. This restricts the running of URL such that it only allows processing of URLs that conform to the regular expression. This means you could restrict any malicious intent on the site by changing the params on the URL for store-id, catalog-id etc...
- Cross-site scripting protection, is enabled by default for the Stores web module. It is important that all the parameters are encoded by developers during the development phase and tested thoroughly. The prohibitedChars rule has been provided by default, however it cannot be comprehensive. The blacklist is used as fall back for cases when the store is not encoding the output properly (c:out). The problem with blacklists is that hackers keep finding ways of bypassing them by using different encodings and escaping characters. It's not possible to come up with a completely robust blacklist solution ( OWASP calls blacklist "fragile" ). This is more complex to apply to the REST body. It is recommended that there is a review on the OWASP Cross-site Scripting site for project implementation. The project management and security architect should provide guidelines on preventing, protecting and testing against Cross-site scripting.
- Mitigating threat of denial of service attacks: It is recommended to set the boundary for the product search results by specifying maximum page size and result size. Similarly, one can set allowable ranges for other business objects like the shopping cart.
- Enable ClickJacking Protection – Clickjacing is when an attacker is able to trick you into believing that you are on a particular site/page, while re-directing you to do something else. This can be achieved via several techniques, however we will focus on the mechanism where iFrames are exploited to overlay your desired site. The article talks about the X-Frame options and Content Security Policy.
- WebSphere Commerce database encryption to a stronger standard to reduce the chances of a successful brute force attack by Migrating from Triple DES to AES-128 encryption
- Data security practices for integrating eCommerce system with Order Management System
1 - Use of Cryptographic Algorithms and Key Lengths: NIST announced its Special Publication (SP) 800-131A in 2011, which recommends transitioning of the use of Cryptographic Algorithms and Key Lengths.
Standard requires adherence to
- Digital signatures must use at least SHA-2 hashing algorithm, and WebSphere Commerce v8 uses SHA-2
- Cryptographic keys adhere to a minimum key strength of 112 bits. WebSphere Commerce provides a detailed procedure for the migration of encrypted data in the database to use AES 128-bit encryption.
- Enable TLS 1.2 for SSL and disable protocols less than TLS 1.2 for WebServer, any integrations with LDAP, outbound email. WebSphere Application server must adhere to TLS 1.2
- All certificates with RSA or DSA keys must be 2048 bits or higher. Certificates with elliptic curve keys shorter than 160 bits must be replaced with longer keys. All certificates must be signed by an allowed signature algorithm. For example, SHA-256, SHA-384, or SHA-512
2 - Federal Information Processing Standards publication 140-2 (FIPS 140-2) covers the security standards that are required for cryptographic modules. The WebSphere Commerce documents the steps that are needed for commerce to runn on a WebSphere Application Server and on HTTP servers that are in FIPS 140-2 mode.
3 - The Payment Card Industry (PCI) Data Security Standard (DSS) - The PCI DSS Version 3.0 standard lists 12 requirements which retailers, online merchants, credit data processors, and other payment-related businesses must implement to help protect cardholders and their data. The requirements include technology controls (such as data encryption, user access control, and activity monitoring) and required procedures. This is required to be implemented when the cardholder data resides with the business. The WebSphere Commerce Knowledge Center documents the procedure ONLY for the WebSphere Commerce application to be PCI compliant, while there are many other aspects that the business needs to take care as part of the compliance.
The security of eCommerce is a broad topic that requires the Chief Information Security Officer (CISO) at the organization to work at various levels. This was covered in the PART 1
This article is an attempt to help the reader get a glimpse on the application security aspects for a commerce site. I have tried to categorize security from a view of access, vulnerabilities and data protection and help with a capability view on WebSphere Commerce.
I have also created a template spreadsheet that one can refer in their project as a starting point to track the various headings, status and desired status. This template is to be enhanced/tailored to suit your security posture.
2016-July-WCS-Security-Template-v1.xlsxView Details for Project Managers to capture actions. Do provide your feedback if you use it to enhance it further.