We are a group of architects and subject matter experts who collectively have 25+ years of experience in executing ecommerce solutions over WebSphere Commerce across the world. This blog is our way of sharing this experience and learnings to a wider audience.
We would be covering a wide range of topics -- most of them will be in the context of interesting requirements we encountered over our services career along with a generous sprinkling of insights into the product's capabilities and limitations that enabled us to solution those requirements or overcome the limitations.
We are here to share and learn - so would love to hear back from you!
eBay does it. Amazon does it. Alibaba does it. Walmart, Best Buy, Flipkart... list keeps growing every quarter. Quite a few clients I have talked to think marketplace is a "low hanging fruit" and are compelled to try it. This is especially true in what is traditionally referred to as "growth markets". On paper, the allure is irresistible - after all who wouldn't love to make a commission for providing real estate for sellers and buyers to meet and transact. In an era of cheap computing resources, this sounds almost like free lunch. Only it isn't. I will explore some aspects of a marketplace strategy that are worth considering before going live with a marketplace solution.
First of all - the millennial retailers often don't realize that marketplace model has been around for ages. Focus has always been to bring together sellers and buyers on a online portal. For example, manufacturing units would set up portal to bring suppliers together and bid for contracts. Metal junction is a successful auction based marketplace for coal industry in India. Even a stock exchange is a marketplace - with stock brokers playing the role of the portal to allow sellers and buyers to transact. Why then we see a renewed interest in marketplace? Why are retailers queuing up to try this model now?
Reason for adoption
Most consultants and retailers agree that decision to go for a marketplace is based on following reasons:
- Survival or expansion strategy based on whether retailer does it to counter erosion of customer base to large online retailers OR reaching out to new customer base/market
- expand the catalog that a retailer offers through their online channel without investing in larger warehouses and logistics
- Only the sale transaction need to be hosted by the marketplace provider - procurement and fulfillment is executed by the seller.
- no risk of stale inventory or need for mark down - inventory risk is with the seller
- sometimes the legal, political, social and cultural restrictions makes it easier for companies to launch marketplaces to circumvent those restrictions
- earn fees/commissions -- though this is relatively small, this is almost all profit when compared the cost of hosting transactions
- shopper may feel they got more choices in a single portal(psychological angle) hence popular expectation is that it leads to customer stickiness
I am sure there are more reasons, but no major surprises here.
But retailers overlook or don't give enough importance to some important aspects. I 'll talk about four such aspects in this blog.
Four aspects retailers mustn't overlook
Main among them is that shoppers often don't differentiate between the seller and marketplace retailer. A poor experience with a seller often translates to poor experience with the marketplace and lead to a lost footfall. In other words, established retailers are putting their reputation on the line on behalf of sellers. So they better have a way to enforce discipline among sellers. One way could be to do random spot checks. I have heard of employees posing as shoppers to get proof of malpractice against sellers with questionable track record. But that is reactive approach. Marketplace provider would need to take up more responsibility. How about a tie up a logistics provider and have random quality check at pick up points? Companies often dismiss this as a risk that needs to be written off but overlook the intangible value of brand reputation. Differentiating on trust often wins long lasting clientele beyond finicky deal-hunters.
Other thing is standardization of catalog information - quality of information from many sellers is below par. They are probably managing all information in spreadsheets and have systemic problems of duplication, inconsistency or general lack of information. Unless the marketplace strategy has a solution to address this problem, shoppers are going to find anemic product details pages.
Sellers have a mix of mature and amateur back end systems. So the marketplace solution must be able to expose robust APIs for mature sellers to integrate with their back end systems while allowing gateways for upstart sellers to directly manage their business on the marketplace itself.
Fourth is differentiation. When marketplaces are dime a dozen, what makes a shopper select a particular solution. This is where opinions are as diverse as people. Some say cool User Interface, others say cool features, yet others rich content, SEO, social integration etc. Most impactful ecommerce solutions have a compelling reason for shoppers and sellers to stay with you. Companies that target specific market segments have better success than generic retailers. If the marketplace is targeting office supplies, home furnishings or industrial tools, it is more likely to succeed than a low-margin driven competitive general retail.
There are many more - a quick search on the Internet should reveal a more comprehensive list. But above four is what I found as prerequisites a retailer must think before opening up a marketplace.
The tool for success
IBM Commerce portfolio has a rich set of products that enables a end-to-end ecommerce solution to be built to custom order. It comes with a rich set of off the shelf features, however its real value is in building on top of the vast and deep foundation it provides. At face value people often dismiss it as not suitable for marketplace solution. However its data model is rich in its catalog, contract and organizational capabilities, almost all of its services are available as REST APIs and WebSphere Commerce and Order management tandem provides the deep catalog and order management capabilities that is par none. IBM's deep analytics portfolio is well known and is available as real time as-a-service solutions. What I would like is a unified business tool that can bring together and expose all these deep capabilities to enable execution of a marketplace strategy. With all essential backend services available as web services and REST APIs it is just a matter of exposing them over relevant UI
Modified on by Vani Mittal
There are many order capture scenarios prevalent in the eCommerce space today. Some of the commonly used are – pre-orders, recurring orders & subscriptions.
These order capture mechanisms are appropriate for different situations and different kind of products. For example, pre-orders are great for allowing shoppers to reserve their own personal copy of a highly anticipated product.
In this 2 part blog post, we will explore the above mentioned order capture mechanisms and also cover the various points that need to be kept in mind while implementing them using the IBM Commerce portfolio of products. Part 1 (aka this one) will cover subscriptions and recurring orders, while part 2 (coming next week) will cover pre-orders.
Subscriptions & Recurring Orders
Subscription is a business model where a customer must pay a subscription price to have access to the product/service. The model was pioneered by magazines and newspapers, but is now used by many businesses and websites. Recurring orders are similar to subscriptions in that they also allow shoppers to buy the same items on a periodic basis without having to place orders repeatedly.
In case of subscriptions, the time period and frequency is determined by the retailer. For example, a publisher decides that a magazine will be offered on a monthly basis and at 1 or 2 year subscriptions. While in case of recurring orders, the time period and frequency is usually decided by the shopper. For example, a shopper decides that she wants 1 litre of milk to be delivered to her every day for the next 3 months.
There are multiple benefits of subscriptions and recurring orders, some of which are mentioned below:
Retailers benefit as they are assured of a predictable and constant revenue stream from the subscribed shoppers.
It increases customer loyalty by allowing customers to become greatly attached to using the product or service and, therefore, be more likely to renew when the time comes.
It reduces customer acquisition costs and allows for personalized marketing.
Subscriptions and recurring orders are supported out of the box by IBM Commerce solutions, namely, WebSphere Commerce and Sterling Order Management. However, there are certain aspects that a retailer must think through before implementing this feature as it can have an impact across almost all of the eCommerce subsystems.
Merchandising & Marketing
Before a product can be bought as a subscription or be part of a recurring order, it needs to be marked as such. This can be done using IBM Management Center for WebSphere Commerce or via the catalog data load process. The retailer needs to prepare this data and ensure that the additional properties, for e.g. subscription frequency and time period, are included in the catalog data.
Another aspect to consider is product life-cycle management. If a product is withdrawn from sale, what happens to the subscriptions & recurring orders that it is part of? One option could be to proactively cancel such orders and email the customers giving them a choice to order similar products (an opportunity for cross-selling and up-selling).
The payment for a subscription is usually captured upfront for the entire duration. On the other hand, for a recurring order the payment may be captured when each instance of the recurring order is processed. This introduces many complexities in the management of payment. For example, how do you handle the case where the credit card provided at the time of placing the original recurring order expires? In WebSphere Commerce, the recurring order will be cancelled if the payment for one of its instances fails during order processing. The customer can then be informed through an email or SMS and be asked to place a new recurring order. Additionally, there are 3rd party recurring billing management systems available that handle the many subtle complexities of subscription based payments, such as automatically emailing customers about credit cards expiring. Such systems could theoretically be integrated with WebSphere Commerce to handle credit card expiry detection and notification proactively.
Pricing & shipping
As mentioned in the previous point, payment for a subscription is usually captured upfront. So, the total price being charged to the customer includes the price for all the items for the entire duration of the subscription. Any future price changes will not affect the customer during the period of subscription. While for recurring orders, it is common to capture the payment as and when each instance is processed. In this case, it is up to the business to decide whether the original price applicable at the time of placing the recurring order should be honored or should the current price be taken into account.
When a customer places a recurring order, the retailer is in effect promising the customer that the items will be available for the entire time period chosen by the customer. However, it may happen that the items are out of stock at the time of fulfillment of an instance of a recurring order. What should be the business process in this case? In WebSphere Commerce, the out of box behavior is for the recurring order to be automatically cancelled and the customer to be notified of the fact via email or SMS. An alternative approach could be to delay the fulfillment of the particular instance of the recurring order till inventory becomes available. This may be a good option if inventory is expected to be available soon and the customer can accept a delay. The customer could also be given a choice to accept similar products that are in stock providing an opportunity for cross-selling and up-selling. These alternative approaches do not come out of box but could possibly be implemented with some customization.
For normal orders, different retailers have different policies on whether they allow an order to be modified by the customer once it has been submitted. A common practice is to provide a short window of time (maybe a couple of hours), before an order is sent for fulfillment, in which the customer can request for changes in an order via the retailer’s call center. Once the order is sent to the backend system for processing then only cancellations may be allowed. In case of recurring orders there is further thought to be given, since a customer may want to edit one of the instances of the recurring order or maybe the remaining unfulfilled instances of the order at some point of time in the future. In this case, it may be simpler to just cancel the recurring order and allow the customer to place a new one.
Returns, refunds & cancellations
The return process for a normal order itself can be complex depending on the process and business policies followed by a retailer, but there can be additional intricacies involved in case of subscriptions and recurring orders. Consider a scenario where a customer has subscribed to a monthly magazine for a year and has paid the total subscription cost upfront. A few months in to the subscription the customer decides to cancel it. How much refund is the customer entitled to? You would need to work out the refund by calculating the proportionate cost of each magazine issue and then deducting the cost of the issues that the customer has already received.
Providing customer service via the phone and email can be expensive. It is always a good idea to empower customers to manage their account and order information through the online store. Features like order history are pretty much standard these days. But in case of recurring orders and subscriptions, some additional functionality may be required to make self-service truly effective. Customers should be allowed to track and manage their subscriptions. They should be able to view the history of not just the order originally placed but also the recurring instances of the order. They should be allowed to cancel or edit the orders (within the boundaries of the business policies governing it of course). They should receive appropriate notifications of actions to be taken, for example, if a subscription is up for renewal.
In this blog post, we discussed the main business processes involved in subscriptions and recurring orders. In next week’s post, we will explore pre-orders and how they can be leveraged by retailers.
Modified on by Anbu Ponniah
In part 1 of this blog post, we discussed subscriptions and recurring orders, two features provided out of box by IBM Commerce solutions. In this 2nd part, let us discuss another popular order capture mechanism – pre-orders.
A pre-order is an order placed for an item which has not yet been released. These days many retailers use pre-orders to allow for shoppers to reserve their own personal copy of a highly anticipated product. Apple iPhone is a great example of such a product for which there can be a huge initial demand on release of new models.
Pre-orders allow consumers to guarantee immediate shipment on release, manufacturers can gauge how much demand there will be and hence how large initial production runs should be, and sellers can be assured of minimum sales. Additionally, high pre-order rates can be used to further increase sales.
Retailers can provide incentives like discounts, associated exclusive merchandise etc. to encourage shoppers to pre-order.
On the face of it, you might think that IBM WebSphere Commerce does not support pre-orders, but it provides enough flexibility to allow for support of pre-orders to be added with some customization. But first, let us explore the concept of pre-order in a little more detail.
The pre-order feature can be implemented in various forms and this can dictate the complexity of the processes involved. Let us discuss two of these variations:
Pre-order without payment
Some retailers allow for a product to be pre-ordered without capturing any payment. When the product becomes available, the customer can be asked to complete the checkout flow and make the payment. In this case, pre-order is really just a reservation process where a customer’s intent to purchase is captured with bare minimum personal details like email address, so that the customer can be notified when the product becomes available. There are some points to consider about this form of pre-order:
The reservation process can be implemented as a bespoke feature in WebSphere Commerce where in there can be a simple form to capture the customer’s details and a notification mechanism to inform the customer when the product is available to order. The customer details can be stored in a custom data model and a custom scheduled job can send out notifications either through the WebSphere Commerce messaging system or through an external email/SMS provider.
Since the customer is not paying for the product at the time of reservation, there is no guarantee that he/she will actually buy the product when it becomes available. Hence, this does not provide an accurate picture of the initial demand.
Again, since payment has not been captured, the retailer is not bound to offer the product at a certain price. The price may change when the product actually becomes available.
Pre-order with payment
An alternative approach can be to capture the pre-order in a way similar to how normal orders are captured. That is, allow the customer to complete the checkout flow and authorize the customer’s credit card for the amount. Once the product is ready to ship, then the customer’s credit card can be charged. One important point to note is that credit card pre-authorizations typically expire after 5-7 days, after which the hold placed on the funds is released. The customer needs to be contacted again for a fresh pre-authorization to take place. So, this approach is ideal if the time difference between when a product is available for pre-order and when the product is available to ship is between 5-7 days.
In some countries it may be OK to capture funds immediately for a pre-order product, as long as it is clearly communicated to the customers that they are being charged in advance for an item that will ship at a later date. However, there is a higher risk of chargebacks in this case. If you make people wait before you ship, a certain share of customers always end up asking for their money back.
Now, let us see how the various subsystems of WebSphere Commerce can be adopted to handle pre-orders.
Catalog – Products that are available for pre-order need to be marked as such. This can be done by using any of the customizable fields (for e.g. FIELDx in CATENTRY table) to store the flag. On the UI, the “Add to Cart” button can then be shown as “Preorder Now”.
Inventory – A retailer can decide to offer either a fixed quantity of a product or unlimited quantity for pre-order. When using the non-ATP inventory model, INVENTORY.INVENTORYFLAGS column can be set to 3 which means ‘do no check inventory, do not update inventory’, thereby allowing for unlimited quantities of the product to be ordered with no inventory in stock. When using the ATP inventory model, ‘expected inventory’ can be used to track inventory for a pre-order product. This is similar to the back-order concept available out of box in WebSphere Commerce where in an order is captured against expected inventory and the fulfillment is placed on hold till inventory becomes available and can be allocated to the order.
Order management – Orders having pre-order products can be marked as pre-orders by using any of the customizable fields (FIELDx) in ORDERS table. This flag is used to indicate that the fulfillment of such orders will happen at a later stage. When the business is ready to deliver pre-order products, then the inventory status of such orders can be updated and their fulfillment can be started.
Marketing – The marketing strategy for pre-order products is critical. Pre-orders are only successful if there are customers that are excited about the newly launched product and are willing to buy it without waiting for reviews and feedback. WebSphere Commerce has powerful precision marketing features that can be used to advertise the pre-order products and help build the buzz around them.
Pre-orders, subscriptions and recurring orders can be powerful order capture mechanisms which when implemented properly can greatly increase a retailer’s revenue. As we have seen, the business processes related to these features can be much more complex and nuanced when compared to normal orders. So, apart from the technical aspects, it is equally important to plan and build the right business processes for these features to be successful. It will take a joint effort between the technical solution architect and the business analyst to get there.
What are the benchmark reports saying
The IBM benchmark report, 2014 US Online Retail Holiday Readiness Report, showed that an average shopper’s attention time is dwindling as they are making quick and abbreviated visits to retailer sites. The average time on the site was measured to be ~7 minutes which is about a minute less over past 2 years, and the data showed that there is reduced trend on page views per session. These metrics become pertinent as the retailers have to continue their focus on bringing the right products to the customers in fewer number of browses and searches.
The WebSphere Commerce search, which empowers the search on the storefront, can be customized, extended and tuned for a retailer’s business. The retailers require to consistently monitor the performance of their search through the search hits, misses, conversion ratio, site abandonment, and accordingly tune and refine based on what the shoppers want and marry it with what the retailers want to sell on their website. We explore some of the use-cases which retailers will find useful in order to achieve a high search to sale conversion ratio.
The foremost design decision is around the search index schema. The retailer would want to add more fields that can be searched, these additional fields would be added to the index schema, for example, a sport clothes retailer will want include to include the sport-type as an indexed and searchable field. This new field sport-type can be then be used for impacting the search results, say for example, the order for the search results.
Search for conversions
One of the ways to increase the search hits and help site users find what they are looking for, is for the business build a strong set of vocabulary which represents their catalog set. This is made available by what is called the ‘Synonyms and Replacement terms’. The synonyms would help a customer say who is looking for a ‘fitbit’, show the different wearable, gadgets, activity tracker, pedometer that the store sells. The replacement would be used for terms, in the same example, if a shopper searches for ‘bracelet’ in a sports store, this will likely be replaced with ‘activity tracker’, ‘pedometer’ for the retailer.
The retailer would further want to refine the search results returned to the shopper based on inventory or brand partnerships. The WebSphere Commerce Management Center tool has a store preview feature which displays the relevancy score in the search results. This data provides the insights on what adjustments the user would make to have the desired result displayed. The retailer can accordingly tweak the result by using the search rule ‘Change Search Result Order’ and then iteratively get the desired set by viewing the relevancy information. Continuing with the example of keyword search ‘activity tracker’, if the business would like the fitbit brand to show up higher than other brands, then it will need to set its ranking order higher.
The retailer can exercise more influential rules than changing order using pre-existing options in the Management Center. For instance, they can specify that the ‘Top Results’ where on keyword search for ‘activity tracker’, the business can decide to show the latest fitbit product – Fitbit Surge as the Top Result. Retailers can infact influence a result where say the search result contains a specific product, and then move it to the top of the result. So with the example, if the search result contain ‘Fitbit’, then specify the Top Result as ‘Fitbit Surge’.
The retailer should use search statistics and experiment data to evaluate the impact and continue refining the search rules to have the maximum return on their shopper behavior.
Concluding with search extensions
Influencing search results is an important extension to the marketing and promotion tool. We have seen several ways in which the retailers can achieve this with ease. We will continue to delve into search extensions in the future posts as my colleagues will share specific extensions to derive enhanced benefits from the storefront search.
As an architect/practitioner growing up in a world of enterprise architecture where data is rarely seen outside the ambit of relational schemas, NoSQL brings with it a breath of fresh air - simultaneously challenging previous beliefs, promising exciting opportunities and bringing its load of inevitable baggages. In this blog post, I shall walk down the discovery path of how, as an architect, I see NoSQL complementing or supplementing or replacing traditional solutions. The perspective is that of an architect trained in a world of stack architecture and transactions, so this may not resonate well with advocates of either technology. However it highlights opportunities and challenges that would translate into solutions that everyone can relate to. This will be a two part series - part 1 (this blog) will set the context and part 2 will detail one solution possibility.
Defining NoSQL as a comparison of SQL databases is no longer straightforward. Lines are blurring. For this blog, NoSQL databases are those that have flexible schema capabilities, store their data as one of the following formats:
and primarily support a non-SQL query interface. They may or may not support some or all of ACID (Atomicity, Consistency, Isolation, Durability). Internet is filled with information on NoSQL, their features and extensions, so I will try not to replicate this information. Instead let us focus on what it means for ecommerce - starting with WebSphere Commerce.
Let us first set some background here. First order of business is to view information managed in an ecommerce solution in a perspective unconstrained by traditional views. For example, one may view data handled by WebSphere commerce from the following perspectives
Catalog perspective -- Attributes, structure and administrative information related to products. Here the term attributes is used in the traditional sense of the word and not restricted to WebSphere Commerce's definition of attributes. Hence, the information included in this set of data are information that describes products for external consumption, information about products for retailer's own management, information about structure and relationship between products and information that are needed to administer or manage it all. Entire Catalog data model of WebSphere commerce falls under this bucket. Importantly, this does not include price and inventory.
Order perspective -- Attributes, structure and administrative information about shopping carts and orders. Same definition here too - includes information from shopper, retailer and administrative perspectives. This, in most cases, constitutes the primary transactional data for a retailer. Information about orders will need information from other sets of data such as catalog, payments, pricing, marketing etc. It may be tempting to already think relational (foreign keys) here. If we hold off the temptation to bucket perspectives of data into relational concepts, we may find it easier to see alternative views.
Customer perspective -- Information about shoppers in our retail solution. It includes login and password for registered shoppers. It includes addresses and payment information for all shoppers. Again, it will have overlaps with orders and catalog perspectives.
Marketing perspective -- Information needed to market or promote one or more products. It includes information that defines the rules, information that should be displayed to shoppers and business users and information needed for business logic to evaluate and apply them.
Above is just a short list that this blog will use for illustrating a point of view and is not an exaustive or correct list.
For each perspective, we see the following are immediately apparent:
some data is needed to display to external world versus some information is for internal use.
some data is static, others not so static
some data needs high degree of ACID support others can live with cursory checks
With this backdrop, let us consider the following use cases:
UC1: A shopper must be allowed to define and manage any number of personal information about themselves.
UC2: A shopper must be shown personalized messages based on personal information they have shared
Traditional solution requires some innovation to allow users to create their own free-form information - so we may end up with a lot of duplicates or near-duplicates. This information can soon grow out of hand - imagine if you anticipate a million customers and dozen attributes on average for each. So natural inclination is to push back on such a requirement and instead box the attributes to retailer defined values such as hobbies, favorite XYZ (sports, color, place...) etc.
Then, in WC, one could define a service to adding information to Member attributes part of the Member data model in WebSphere Commerce, customize promotions/espots logic to allow targeting based on member attributes and then during customer interactions check if member attribute driven espots and/or promotions can be shown. This is one way (and one of our future blogs will detail this). Other way would be to define some custom table and store these as key-value pairs (in fact storing in MBRATTR has a key-value sense to it).
NoSQL gives us an opportunity to think beyond the constraints of data model here. After all if the information naturally fits the key value pair model OR can be modeled as a "Personalization Document" of a customer, why not think NoSQL here?
The solution must be capable of creating this information, allowing shoppers to maintain it and for internal commerce logic (such as promotion/pricing components) to use it in their transactions, specifically as part of existing commerce transactions. And it need not require remodelling entire WebSphere Commerce database and business logic to use NoSQL. I will be advocating a layered approach where each type of data model is used to take advantage of their core strengths while minimizing the custom code needed as much as we can.
We will explore that solution in my next blog.
Modified on by Vani Mittal
WebSphere Commerce search provides enhanced search functionality in starter stores by enabling enriched search engine capabilities such as automatic search term suggestions and spelling correction, while influencing store search results by using search term associations, and search-based merchandising rules.
Starting from Feature Pack 7, search server is a new WebSphere Application Server runtime instance with an embedded Solr runtime, which includes a published RESTful based API to interact with the WebSphere Commerce runtime instance. It de-couples the store pages into the presentation layer and business logic layer of search solution. With this approach, the search and browse-only traffic from the storefront can be offloaded away from the WebSphere Commerce server (transactional server), to the search server. The B2C store component pages use the RESTful API to retrieve necessary data from the Search Server instance and present it to an online store visitor. With this architecture, the search servers can be scaled separately and therefore the search and browse traffic can be handled independently, creating a flexible and scalable deployment model that can adjust to various storefront browsing traffic patterns at different times or shopping seasons.
A well designed cache is an important aspect of a well performing search solution in WebSphere Commerce. In this blog post, we will discuss the different types of caching that can be implemented pertaining to search.
The following diagram gives an overview of the caching configurations, triggers and invalidations with respect to the search solution of FEP7+.
There are multiple ways caching can be implemented in the WC and search servers. Typically, in Commerce server, we have:
1. JSP cache
2. Servlet cache (REST)
3. Command cache
4. Data cache
In the search server, we have:
1. Servlet cache (REST)
2. Data cache
3. Solr native cache
Let us discuss the cache configurations that are relevant for the search solution in FEP7+.
Search is used in the storefront in multiple places, mainly for category navigation, faceted navigation, keyword search and auto-suggest.
In keyword search and faceted navigation, there is a lower possibility of reuse of cache entries since there can be a large number of possible keywords and facet selections. Hence, the general recommendation is to not cache full page responses when keyword search or faceted navigation is used. In this case, caching at a lower level can provide a better cache hit ratio. For example, instead of caching the entire search results page, it makes sense to cache the individual product thumbnails which can be reused across different facet selections and search results.
On the other hand, the default version of category navigation pages (that is, without facet selections) can be full page cached. As mentioned above, for facet selections, it is beneficial to cache the individual product thumbnails as fragments. This would have the thumbnails cached for reuse across transactions but when the category page is repeatedly hit, the page would render without having to fetch/read all the individual product entries again.
As discussed earlier, with FEP7+, there is a separation between WC server and search server. The catalog pages, for example, category and product pages have REST calls which hit the search server directly. In this scenario, there can be two possible options for caching category navigation pages:
1) If WC JSPs are being used for rendering the UI then full page and fragment caching can be used as discussed above.
2) If a different technology is used as the front-end (for example, a content management system) and WC is used only as the back-end, then it is a common practice to use the REST API provided by WC for the business logic. In this case, the response of the REST calls can be cached in the search server (discussed in next point). This keeps the separation of browse and transactional traffic intact and as a result one can fully benefit from the ability to scale up search/browse traffic separately.
When using JSP caching, there are well documented invalidation techniques which can be implemented. Details can be found in IBM Knowledge Center:
Setting up cache invalidation for WebSphere Commerce search
Caching and invalidation in WebSphere Commerce search
REST services use both server-side caching and client-side caching. Server-side caching is achieved using dynacache, and client-side caching is achieved using cache directives in the response header.
The topic of REST caching is well documented, so I will not go into detail and instead just provide resources that can be useful for understanding how REST caching works. This topic in IBM Knowledge Center provides information about REST caching strategies.
To see an example of how to cache a REST request, you can refer to this blog post by Marco Fabbri.
From FEP7+, the search infrastructure can be deployed in a separated WAS cluster and then the dynacache service can be enabled on every cluster member the same way you do on Commerce servers. The response of the search REST calls can be cached using dynacache on the search server itself. This is an excellent blog post by Marco Fabbri that describes how to do so.
The default invalidation technique used on the search server is different from the WC server. Since there is no scheduler running on the search server, there is no DynaCacheInvalidation job that runs in the background to perform invalidations. The invalidation on the search server is managed by the cache manager instead. The search server periodically checks for pending invalidations (pending invalidation events in CACHEIVL table) before executing each request. What this means is that each search server instance is responsible for invalidation of its own (local) cache instances.
WebSphere Commerce contains code to use the WebSphere Dynamic Cache facility to perform more caching of database query results. This additional caching code is referred to as the WC data cache. Similar to this, there exists a data cache on the search server that is used for storing internal runtime properties that are used by certain search features to improve overall performance. These data object caches are configured to use DistributedMap object caches.
This Knowledge Center link provides a list of search data caches.
As with the base cache, the invalidations in the search data cache are managed by the cache manager on the search server. This blog entry by Andres Voldman can be referred to for more details about the search data cache configuration.
Solr native cache
To obtain maximum query performance, Solr stores several different pieces of information using in-memory caches. Result sets, filters and document fields are all cached so that subsequent, similar searches can be handled quickly. It is important to understand what they do and to tune them to suit the actual environment and search traffic.
You can refer to Apache Solr documentation for more information about Solr native cache.
For additional guidance, you can refer to the "Optimizing Solr native caching performance" section in this article and the "Search caching" section in this Knowledge Center article.
There are multiple ways caching can be implemented in the WC and search servers. In this blog post, we discussed the cache configurations that are relevant for the search solution that uses the new search architecture introduced in Feature Pack 7.
The priorities for the CIO
The priorities of CIO of an organization include delivering on business solutions, business intelligence and analytics and improving the customer experience with the brand. The IT operations team within the CIO is responsible for the IT systems, which include,
development and innovation of business solutions
production management and uptime
operational efficiency, providing with the analytics from operational and transactional data for the consumption of both IT and business
The priorities for the retail CIO
Additionally, the retail CIO is grappling with the world of omni-channel, building a single customer experience, and working towards delivering personalized services. This involves building a mobile strategy as the statistics continue to show higher exploration on mobile and higher conversions on desktops/tablets. This also means the focus on the Big Data paradigm, to gain competitive advantage.
In the following blog, I will outline the layers of monitoring that an IT Operations team will need to setup as the commerce site is ready to Go-Live. The key operations data is required as soon as the system becomes available to the customers. The retailer must get their data in view and the thresholds and alerts set out prior to the launch. The alerts should be set based on the performance cycle for the launch. The other business data can be configured based on the priorities and focus of the business.
The metrics and measurement
1 – The key performance indicators (KPI) which a retailer wants from their digital channels are
Visitor summary: page views, average session length and time spent
Visitor behavior: bounce rate, new visitors and conversions
Product views, top search terms, search hits and misses, cart abandonment rate
Paid search sessions, marketing, referring site sessions
Browser types, devices
User behavior across different channels
Performance against industry trends
There are several analytics tools which offer integrations with websites. IBM’s Digital Analytics is the SaaS offering which provides insights on this data and also publishes industry trends across holiday seasons and events. Some of the reports that any retailer will find very insightful to plan for upcoming 2015 events and holidays are the 2014 Black Friday report, 2014 Cyber Monday Report, Worldwide Online Holiday Mobile Shopping Report 2014, UK Retail Online Christmas Shopping Recap Report 2014.
2 – Retailers have to manage their operations and deliver to business based on the uptimes, scalability and non-functional requirements (NFR). Some of the metrics which are must-monitor for a site to deliver client satisfaction are:
Application performance metrics like CPU usage, JVM heap usage, garbage collection, Web Container threads
Backend performance and response times from 3rd party integration systems like payment gateway, or external applications like inventory, availability, address verification
Response times for the various operations like browse, search, cart, order completion
Requests per second for various incoming requests
Number of orders, order-size, order-value
The Application Performance Management tools are specifically focused on monitoring and management of performance of the software applications. The tools provide a view on most of the above parameters, by either using agents or are based on log analytics. The out-of-the-box features have to be augmented by providing visibility to other metrics, like the order information will be extracted from the database. The agent-based tools need to be configured correctly and tested to ensure that they do not impact the performance of the host application. We will explore some of the tools under APM in future blog topics.
3 – Application specific monitoring aspects
Java applications have standard mechanism for managing application resources using Java Management Extensions (JMX), is managed bean (MBean). The WebSphere Application Server can be extended with custom JMX MBeans. One of the examples to demonstrate a key performance resource for WebSphere Commerce is the caching efficiency. The performance of the WebSphere Commerce can be optimized by building in a strong caching methodology at various levels. The application level caching is built upon the WebSphere Application Server Dynacache which caches JSP/servlets and data object. The cache statistics of cache hit, cache miss, hit/miss ratio, the cache eviction, invalidation, provide insights into the site usage and its performance. The dynamic cache service provides an MBean interface to access cache statistics. The Extended Cache Monitor tool pulls the mbean statistics for cache instances and displays the following data:
Display contents of servlet and object cache instances
Display the Dynamic Cache mbean statistics for cache instances
Display the dependency ids of the cache instances
Display the disk offload statistics and contents
There is a lot of data in the commerce system, and the IT Operations can harness the information for insights. The data ranges from the application performance metrics to the business data and analytics for business insights. The downtime, slow site, unknown issues in the infrastructure will catch an operations team blind-sided, impacting in loses and brand dilution. The monitoring and alert enables the operations team to manage and troubleshoot the system in a predictable context. The operations team should evaluate and chose the monitoring tool based on the data they want to view. The tool should be tailored with correct thresholds, alerts, messaging to provide the effective and actionable insights.
In part 1 of this blog we had seen how data that is handled in an ecommerce system could be bucketed into various perspectives and categorized into the role it plays or supports. And we discussed two use cases which I shall restate here:
UC1: A shopper must be allowed to define and manage any number of personal information about them.
UC2: A shopper must be shown personalized messages based on personal information they have shared
In this blog let us get down to the solutioning decisions and see how the puzzle pieces fall into place.
For properly solutioning this we must revisit the data that we will end up capturing and understand what qualities the data store must support. Apart from the obvious needs of storing, retrieving and updating data the choice should allow for discovery and exploration of the data over and beyond the relationship established through the data model. Let us discuss this need through a couple of examples of the second use case.
Beyond saved relations
Example 1: If the retailer wants to show a personalized message such as "Your favorite sports team is the most popular team among shoppers in our stores". For this we must be able to discover if indeed the sports team entered by the current shopper is mentioned by most of the users. A simple document store with some faceting capability can give us this information.
Example 2: On the weekend before Valentine's day every female shopper who have one of their favorite colors as red, pink, dark brown or chocolate will be shown an marketing widget that lists a set of select apparel accessories.
The above examples may or may not make sense as real use case examples, but are taken to illustrate two distinct problems - one is that of aggregation and other is that of segmentation. Similarly, from the retailer's perspective, this design should also support deduplication and standardization --- let me call this using a term that is superset of both: categorization. Without getting a view of categorization, merchandisers and marketers won't be able to utilize the rich data gathered. And we need robust analytics to help us translate the data into trends and relate them to behaviors that a retailer can target for their merchandising and marketing decisions.
Now NoSQL can help us solve the shopper perspective well and provide some help regarding categorization. It can integrate with an analytics solution for the last bit. The data we gather regarding shopper's personal profile should now be used to personalize his/her shopping experience. And the rules of influence could be self contained within the profile information or may need to be executed in the context or in combination of information from catalog, orders or marketing sub-systems. This is actually one of the reasons why we traditionally choose a relational data model solution for this.
And this leads to the first decision point - the data must be available for direct queries from client systems and from ecommerce business logic systems and should be map-able with existing relational data model. This is just to say that when I save information in my NoSQL data store, I will include a few key values such as member_id, store_id, catalog_id, address_id and language_id.
Second decision point is to mirror relevant promotional, marketing and search rules information from relational data store to the NoSQL data store. This is particularly easy since this information is available in XML format CLOBs in WebSphere commerce and lends itself to easy import into data bases like Cloudant or MongoDB. Obviously, every time a stageprop happens, a process should push updates from relational model to NoSQL model.
Third decision point is to extract information from NoSQL data store for analytics and feeding this back into the system of record for factoring it in promotional, pricing, search or merchandising rules. Now it may be attractive for analytics to actually run directly on the NoSQL store itself - but worst case we need a process to export, FTP to analytics system and import into its data store. I advocate the analytics to instead run on the NoSQL data store to become real time (subject to performance considerations) and create outcome of the analytics also in the same data store.
Fourth decision point is to keep authorization, contract entitlement and organizational logic away from this NoSQL data store. These can be enforced at a different layer.
Why PIM and eCommerce?
Data is power. Information is power. An important aspect of any eCommerce solution is to present as much data and information as possible to the shoppers. Shoppers love to visit websites which contain useful information for their exploration, research, studies or whatever it takes to help them decide on a purchase. Product Information Management (PIM) systems play a crucial role in managing the enterprise data. Integration of an eCommerce system with PIM is part of overall eCommerce strategy. Ensure a proper PIM solution is in place before the start of any eCommerce solution. Single view of products enables a better multi-channel experience by providing consistent, complete, and accurate product information to customers. It also provides complete and consistent product information across all commerce channels e.g. eCommerce, store, kiosk, mobile, print catalog, Global Data Synchronization (GDSN). This consideration is all the more important when an enterprise has a wide array of products, frequently changing product characteristics and non-uniform IT infrastructure. eCommerce often needs data in different flavors depending on the channel of sale. For example, mobile app may not require detailed product information but a desktop browser might require as much information as possible.
What PIM contains?
Typically PIM holds product information such as
- product name
- short to long product descriptions
- attribute and its values
- images of different sizes and its meta data
- catalog taxonomy
- workflows to represent business processes
IBM Commerce and PIM
IBM WebSphere Commerce has a strong integration story with Advanced Catalog Management (ACM) which is part of InfoSphere MDM (Master Data Management) suite. Advanced Catalog Management provides easy-to-use tools for business users to centrally manage a cross-channel strategy. Business users can create and manage precision marketing campaigns, promotions, catalog, and merchandising across all sales channels. WebSphere Commerce consumes the output from ACM to present a single, unified platform which offers the ability to do business directly with consumers, directly with businesses, and indirectly through channel partners (indirect business models).
ACM has a 3-step data synchronization.
1. Generating the Collaboration Server data from the InfoSphere MDM Collaboration Server
2. Mapping the InfoSphere MDM Collaboration Server data to the WebSphere Commerce data, and
3. Uploading the WebSphere Commerce data to the WebSphere Commerce Server.
Utilize the pre-configured data model in InfoSphere MDM Collaboration Server which mirrors the data model of WebSphere Commerce, it has some sample workflows (like New Product Introduction, Manage Catalog Entry), predefined user roles with appropriate access privileges as well as integration with WebSphere Commerce to export the catalog entries. These features can be extended or modified as per an organizations requirement.
There are 2 integration approaches with WebSphere Commerce server
- Web service approach
- Batch load approach
For a small volume of data, ACM uses real time a web service call to publish data directly to IBM WebSphere Commerce system. This approach is used for catalog groups and attribute dictionary exports.
For a large volume of data, ACM starts an asynchronous batch load process. This approach is used for catalog entries. The batch process uses WebSphere MQ for notifications and an FTP server for large data file transfer.
For non-IBM PIM solutions, the synchronization concepts with WebSphere Commerce described above remains pretty much the same. Utilize the sample dataload files given in WebSphere Commerce information center to model the output from the PIM system and use the OOB dataload tool to load the WebSphere Commerce business objects. Other option is to use BODL (Business Object Data Load), a services asset from IBM, to load the WC business objects. Some implementations involve customizing BODL to transform and enrich the data from PIM. In some of the recent implementations, both OOB dataload tool and BODL were used to achieve the complex dataload requirements.
To conclude, integrating with a PIM should be one of the key solution architecture pieces for any eCommerce engagements. Dataload must be treated as a “mini-project” and must be taken up for implementation in the very beginning of the overall eCommerce journey!
Modified on by Anbu Ponniah
E-Commerce in India is valued at USD $10 Billion in 2014 and expected to cross USD $15 Billion in 2015 (Report from IAMAI). India has barely scratched the surface of web ecommerce potential but is poised to go ballistic (in a good way) in mobile commerce. An interesting tidbit here is that over 5.5 million 4G enabled mobile devices have been sold but less than 100K 4G subscribers are there, mainly because of the limited availability of 4G network in India. But with reports suggesting that 4G Internet will be launched countrywide in 2015, it is not far fetched to think mobiles may soon monopolize shopping in India.
But rules of old world no longer apply. (Old = 5 years ago.... and I may be too generous here.) While there is general trend of preferring native apps over mobile web or hybrid apps, retailers really care about nailing just one thing : "Engaging User Experience".
What exactly makes a shopping experience "engaging"? The way I think, it is the complete shopping experience that presses all the right psychological, emotional, ethical and rational buttons to make shopping a gratifying experience at conscious and subconscious levels. The experience starts with how I hear about a retailer, my first impression on visiting the retailer, rich information centered around not just products, but also how products gel with my lifestyle (or my aspirations), ability to navigate, locate and visualize what I want, non intrusive but always on hand sales associate attuned to my level of knowledge of what I want to purchase, not having to shop-hop.... list goes on.
Yes, we need cool UI to give the initial wow factor, but we could do with functional UI that does not get in the way of the shopper. Functional does not mean run-of-the-mill, factory cut UI. It should still stamp the retail store's individuality. Beauty is important - world does not take the retailer seriously otherwise, but as mobile shoppers mature, skin deep beauty will ring in hollow. The days of having cookie cutter approach to providing shopping experience are numbered and coming up with custom cookie cutter pattern alone will no longer set a retailer apart. When I visit an e-commerce retailer in my mobile, I expect the product recommendations to match my previous browsing and purchase history. I only notice it if they don't. Rules of engagement are evolving and rules based engagement, any which way we package it, is no longer dynamic enough. The rules engine now needs to think. Not only think, but reason! And this intelligence must be available for both shoppers (as a guidance/advisor/sales associate) and for retailers (to enable marketers and merchandisers digest large quantities of data and discover patterns in a meaningful manner).
IBM has forayed in to the world of cognitive computing and has thrown open Watson analytics with natural language supported exploratory visualization for business associates that takes analytics to new heights without the steep learning curve. The free trial is available for everyone to give this a try. Free version has some limitations : For example data storage is capped at 500MB, record size cannot be more than 100K rows and 50 columns. But it will give a great insight into the promise that this new age solution holds. Imagine being able to ingest scores of spreadsheets from your ERP systems and other analytics tools and gain insights using natural language queries.
How do you do it? Visit IBM Watson Analytics, use your IBM id (register one if you don't have one - it is free). You will receive an email to validate your account. Once validated, you are taken to a landing page where you can upload your data - for example recent sales data. Once uploaded it suggests starting points for exploring your data. You can also view the useful sample tutorials for tips on analyzing your sales or marketing data. These insights can now lead you to make decisions that go over and beyond regular formulaic decisions. Once these insights are available, WebSphere Commerce solution can now factor these relations into their recommendation or guidance engine.
I played around with just a dump from a sample ORDERS table of WebSphere Commerce. My intention was to find my order spread by currency (aka region) in a given time frame. It was as simple as dumping a report about all sales order into a CSV, dragging and dropping that CSV onto IBM Watson Analytics and going through various recommended and custom visualization schemes.
I could type in natural language questions and it responded with graphs. As an example, I could make out that total sales value indicates that my store's sales in Japan (presumably due to value of Yen) far outstrips my sales anywhere, but number of orders say my USA sales was easily more than sales of all other countries put together. I could then drill more into the data to understand average value of sales from each region and type of products selling. This helps me tailor my recommendations, promotions and campaigns more effectively. Many analytics tools may do this in one manner or the other, but IBM Watson does this while supporting natural language queries putting it within reach of any retail business user and not requiring a PhD in data science.
I have barely scratched the surface of this tool's capabilities. Next time you wonder why you are able to navigate to a page showing your preferred brand, color and size of t-shirt within a couple of taps without ever disclosing this information to the retailer, you may have to thank Watson for it.
Has anyone played around with Watson analytics? What are your thoughts? I welcome you all to share your experience.
The price pressure
Twenty first century shopper has so many options. Unless a retailer is a niche player, it is a shopper's world. While retailer has many tricks up their sleeve to bring eyeballs to their e-commerce website, keep them there, convert into a sale and bring them back, the de-facto strategy that all retailer's adopt is the ubiquitous price strike through to give the shopper a sense of getting a good deal on the goods. Price used to be decided by business teams pouring over their analytics reports, industrial demand/supply reports and peering into the crystal ball of future trends. It still is done like that in many places. But what has changed is that price gets revised (usually downwards) more frequently than before. What used to take months and weeks earlier now happens matter of days and hours.
Let us say your client has a pricing strategy in place - and have the experts coming up the right prices to show for a product each hour of the day and change that every two hours . What would be your strategy as a technical architect and adviser?
Options for price updates
In technology, every thing is possible but may not be practical within the constraints a particular solutions may be operating within. Understanding the when and what to compromise on is the trick.
So what is possible in WebSphere Commerce?
If you get your price as CSV from your pricing engine, then refer here for how to load up your price: . If your pricing engine is IBM DemandTec, then see DemandTec with WebSphere Commerce.
For more control BODL service asset could be used to create a custom load handler.
Or if CSV file is not your cup of tea - say you need this as a true "feed", a look at the current "Content Management system web feed" utility provides some clues. At the end, same data load framework will be used to load information into WC database, but ability to establish connection to the feed provider and transform the feed into WC specific business objects is the real work here (how to do this will be taken up in a future post).
Or may be old style (now deprecated) massload may be used.
Some may use a form of database export/import.
Of course, WebSphere Commerce may be the price master so that business users create/manage price using Commerce Management Center.
Any which way we load the price - the question considered in this blog is : How often should we propagate updated price to our live site?
Real question: How many times?
A change in price needs the following to happen:
1. Change must be propagated from authoring DB to live DB.
2. Search index must be rebuilt (if using priceMode = 1 or 2)
3. Dynacache must be invalidated and warmed
4. CDN cache must be invalidated and warmed
All the above takes time. I don't mean hours - but we need to be cognizant of the number of minutes each of this takes.
If the price to be shown is dependent on conditions (price rules) or if the retailer wants to define price reductions via promotions because that is how they see price discounts, then some pre-processing will be necessary to decide which price to index and how to cache your price by associating the correct combination of customer segment, promotions and price rules.
And cache invalidation and warming takes time unless you have a caching appliance like WebSphere eXtreme Scale solution that has excellent cold-cache capabilities that renders warming unnecessary.
The price change also affects in-flight transactions. A robust business logic should inform shopper of the change in price up to a point in their shopping journey and otherwise accept the order with previous price. Visible price change during checkout affects shopper's trust on the retailer. Frequent switching of price may just make people wait longer and delay their buying decision. But not changing the price for long could expose the retailer to competitors under cutting them. Finding the right balance is more of an art than science.
One customer agreed for twice a day price change. Another was OK with hourly updates. Most challenging one was a customer who insisted on instant change and that too on average 50 times a day. Things we need to work with retailers is batching as many updates together as possible and helping the retailers work out some form of prioritization within their process. For example, price is wrong (missing a digit altogether) warrants instant reaction. In other cases, it may not be that urgent. Also, solution could be found by a different strategy altogether. For example, reaction to under cutting could just be advertising a policy of price match with reputed retailers rather than getting into a no-win war on price. Winning the retail race on core differentiators such as engaging shopping experience, after sales service etc is more profitable.
Random closing thought
One random, but relevant thought: What is the deal with strike through price? Some retailer may just want all their prices to be struck through with a lower price right next to them. And the "offer" price may just be one cent or one rupee less than the "list" price. Such tricks may just train the shoppers to filter out the struck-out price altogether, robbing the emotional benefit of scoring a deal. Often the actual value of saving in numbers is more transparent than percentage saving.
Modified on by KittyKrishna
One of the requirements is to index the output of price rule computation as part of the solr catalog entry index. This ensures the prices that are shown on PDP, PLP and the price facets are from index. In order to achieve this, we will utilize the steps from indexing contract prices in WebSphere Commerce search.
Steps to set up the computed price into index
Set up a test price rule to test.
Open Management center and select the esite and go to Catalog Filter and Pricing tool.
Create a new price rule and view the same as shown in the following diagrams
c. After creating new price rule, you should see in the list as shown
Associate the above price rule to the eSite default contract. Use WebSphere Commerce accelerator to do the association
Open WebSphere Commerce Accelerator and select the eSite
Go to Merchandise > Catalog Filter and Price Rule
Select the Price Rule tab and in the Property Value drop down select the price rule created in the 1st step above
3. Change the price mode to wc.search.priceMode.compatiblePriceIndex in the STORECONF table.
insert into storeconf(storeent_id,name,value) values('storeent_id', 'wc.search.priceMode.compatiblePriceIndex', '1.0');
storeent_id is the store id of the eSite. For example, in my set up
insert into storeconf(storeent_id,name,value) values(10152, 'wc.search.priceMode.compatiblePriceIndex', '1.0');
Preprocess the WebSphere Commerce search index, for example,
Run the calculate price utility, for example,
di-calculateprice.bat -serverName localhost -masterCatalogId 10001 -siteAdminId wcsadmin -siteAdminPwd *****
Ensure that the utility runs successfully
Check for the message – “The calculate price utility completed successfully.”
Inspect the following file for errors - WCDE_installdir\logs\wc-dataimport-calculateprice.log
Build the WebSphere Commerce search index, for example,
di-buildindex.bat -masterCatalogId 10001
Verify the prices on PDP, PLP and price facets
Launch the Aurora eSite and navigate to a category as shown
Examine the prices of the products shown
In the above diagram, Budget Tablet price from the offer price list is $300. You can check this in the database table as shown. In the price displayed above there is a markup of 10% which is $330
Indexing catalog prices - http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.developer.doc/concepts/csdsearchindexprice.htm
Changing search configuration properties in the STORECONF table - http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.developer.doc/tasks/tsdsearchstoreconf.htm?lang=en
Modified on by Anbu Ponniah
Performance metrics available for WebSphere Commerce
Need for Agile performance preview
The solution architecture is done and the iterations are on their way churning out designs and code. In projects executed following agile methodologies, features are designed and added in increments. While it is not too hard to get the whole picture on functional capabilities of the solution, it is relatively difficult to get a handle on how all this incremental design and code will eventually perform untill the last of the iterations are done and dusted. That often means, projects shift from iterative mode to waterfall mode with a performance test phase planned right at the end of development cycle, leaving little leeway for development teams to correct any major performance bottlenecks identified during performance testing.
Can performance testing be made iterative? I can imagine my performance specialist rolling her eyes. Problem is that we cannot certify the system for performance unless a test can be run in a controlled environment. And it takes time - which is premium in short iterations demanded by agile projects. So, is there hope? If we compromise our scope to not "certify" the system for performance and instead go for "flushing out major performance problems upfront", we may have a way around this conundrum.
Performance Measurement tool and framework
From fix pack 7 onwards performance logger feature supports a trace string "com.ibm.commerce.performance=fine" to trace response time of key external calls from WC to OMS. Through this a development team can understand the SLA when checking availability, reserving and cancelling inventory and getting order details from OMS. With Fix pack 9, the performance measurement logger feature of WebSphere Commerce allows teams to understand response time, impact of caching and size of result on each call at various application layers. It replaces the old ServiceLogger tracing with a package of logging that can be analyzed through the performance measurement tool.
How to enable and use performance measurement capabilities is well documented in the knowledge center. But here are the essentials of how a development team can use these capabilities to get a preview of performance capabilities of the solution before official performance test is due:
1. Plan a day during your test runs in your iterations for collecting performance metrics.
2. Work with your WCS/WAS administrator to enable the following traces:
3. Have your testers execute a full suite of tests that represents typical browsing behavior of the live site - include search, promotional messages/pricing, navigation, registered/guest and checkout flows.
4. Collect the files in WAS_profiledir/logs/ folder of your server.
5. Use your favorite tool to project the analysis -- for example, basic performance reports are in CSV format and can be simply charted in spreadsheets
Once a base line is formed, further iterations can focus on deviations from previous measurement.
What is measured
Development team gets a preview of where the heaviness is and where either a caching strategy or re-design is required. The reports are detailed enough to point out the following:
Average call duration and Execution time (performance reports)
Size of results (performance reports)
Cache hit/miss (performance reports)
Stack of operations with time taken by each child in the stack (stack reports)
Get information on a full execution cycle (execution report)
Identify source of calls (caller report)
A note of Caution
While this test is not a replacement for proper performance load test, this type of testing can point out where contention is likely to occur and take corrective design decisions early in the development cycle. If this was so easy, then why isn't everyone doing it - simply because there are likely to be false positives and a few false negatives in the findings. After all, the test environment is unlikely to be modeled to represent the scale of prod or perf environment. The environment itself may have some variability due to multiple testers accessing it and development teams delivering fixes. And test execution is subject to manual variations - think time cannot be controlled, sequence may vary etc. So, teams should take findings of such a testing with a grain of salt. So, such an approach requires the performance architect to calculate probabilities against the findings to determine which of them represent real problems. But in the hands of an expert, this mechanism can help create a solution that is functionally devoid of performance bottlenecks.
DEV/QA out of sync with PROD
Most often solution architects see their role wind down once the retail website they help architect, design and implement has been successfully launched. However, architect's deep expertise is needed for some day-to-day operations also. Often I have seen projects that fight through defects and scope creep to launch, but soon find that the development and test teams are way out of sync with how the system is being used by the business. Core of the sync-problem is data itself.
Let me elaborate that problem a bit here: It is typical for development teams to have fair bit of ownership of a "Dev" integration server environment (I will refer to that as DEV - not to be confused with development toolkit environment owned by individual developer, which I shall refer as TOOLKIT). Then test team would have their own environment to perform functional and system integration tests - I shall refer to that as QA. Next we typically have a preproduction environment that closely resembles production in terms of data and configuration - typically user acceptance testing may happen here, sometimes perf test may happen here.... I shall refer to that as PREPROD. Finally we have production environment that shoppers and business users actually use - referred as PROD. Note that each of DEV, QA, PREPROD and PROD will have their own authoring instance for business tooling, dataload etc and live/runtime instances for shopping activities. During initial launch, QA is often the environment where 90% of all data and configuration gets trialed and tested. PREPROD is where that reaches 99% with PROD having 100% of what is actually live. The percentage difference is also skewed towards marketing, promotion and content than with catalog. So in many ways, QA is a fair representation of what business users and shoppers experience in PROD. But once the site is launched, this equation changes drastically. PROD becomes the hot bed of activity - business users may go over and beyond processes defined by the solution team, adopt new processes....and end up creating data and configuration that lower environments never get any inkling about. If the technical leadership is not engaged, within the first year, QA drops from 90% to 66% as a representative of PROD. Soon development teams will see PROD defects where developers and testers swear that they have thoroughly tested but fails in PROD because that environment is now supporting new type of data that the code had not anticipated.
Problem is with communication. Technical teams needs to be actively engaged with business teams to understand ever changing landscape of retail processes. New attributes may be flowing into dataload - which none or if lucky, only a portion of IT team may be aware of. New price rules, new contracts, new layouts.... While most of it is supposed to be transparent to development teams, it is often not! Teams may have implemented some customization that get affected or affects work done by business user. Worst case scenario, as I observed with one customer, is for the business team to start coming up with their own independent processes, working with non-e-commerce IT teams to create solutions in other parts of the organization such as ERP systems. And end up choosing solutions that deviate from WebSphere Commerce recommended patterns of solution.
How do we deal with this
There should be a monthly interlock with business analysts and technical architects - where requirements are continually re-evaluated and validated. While meetings can solve one side of the problem, often truth is in data! Hence there should be some process put in place to ensure QA and PREPROD environments are updated with any non-transactional change that is introduced in PROD - in that sense almost anything that is affected through Commerce management center is a good candidate for replication in lower environment.
One way to ensure this replication is automated is via the use of Data Extract utility. Folks that are familiar with Data load, knows how the framework is geared to read up comma separated values in a specific format being mapped through an XML and used to created BOD objects to be loaded into WCS DB. Extract is just a reverse of that process - from DB data is read into BOD and fields from BOD are mapped to field columns that are written into CSV. The framework is exactly the same and the XML file used are very similar. In FEP8 - there is ability to add custom SQLs to the extract process by just inserting them within relevant tags - use of that for extracting promotions is documented in the knowledgecenter and a similar approach should work well for other parts of the DB : http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.data.doc/tasks/tmlextractpromotiondata.htm?lang=en.
Things to Watch out
One caveat is that attachments require an associated process to collect and FTP images/htmls that they may refer to -- these may have been directly deployed to web server or CDN - hence a separate process is needed to collect them and bring them down to suitable location for lower environments. Another hurdle I faced was in handling custom catalog associations that were implemented through custom code than through data model or BOD definition - so, the extract would need to mimic the same customization to ensure right information is extracted. Another was related to store and catalog organization model - especially if sales catalog and categories were directly created in PROD - then extra care needs to be taken to replicate those first. Lastly, inspite of the utility giving great capability, some data massaging may still be required.
It is worth investing in such a solution for any ecommerce project. For one, it helps to keep lower environments refreshed. But same process can be used to provide outward feeds from WebSphere Commerce - for example, sending catalog data to your ratings and reviews provider or your analytics provider.