DEV/QA out of sync with PROD
Most often solution architects see their role wind down once the retail website they help architect, design and implement has been successfully launched. However, architect's deep expertise is needed for some day-to-day operations also. Often I have seen projects that fight through defects and scope creep to launch, but soon find that the development and test teams are way out of sync with how the system is being used by the business. Core of the sync-problem is data itself.
Let me elaborate that problem a bit here: It is typical for development teams to have fair bit of ownership of a "Dev" integration server environment (I will refer to that as DEV - not to be confused with development toolkit environment owned by individual developer, which I shall refer as TOOLKIT). Then test team would have their own environment to perform functional and system integration tests - I shall refer to that as QA. Next we typically have a preproduction environment that closely resembles production in terms of data and configuration - typically user acceptance testing may happen here, sometimes perf test may happen here.... I shall refer to that as PREPROD. Finally we have production environment that shoppers and business users actually use - referred as PROD. Note that each of DEV, QA, PREPROD and PROD will have their own authoring instance for business tooling, dataload etc and live/runtime instances for shopping activities. During initial launch, QA is often the environment where 90% of all data and configuration gets trialed and tested. PREPROD is where that reaches 99% with PROD having 100% of what is actually live. The percentage difference is also skewed towards marketing, promotion and content than with catalog. So in many ways, QA is a fair representation of what business users and shoppers experience in PROD. But once the site is launched, this equation changes drastically. PROD becomes the hot bed of activity - business users may go over and beyond processes defined by the solution team, adopt new processes....and end up creating data and configuration that lower environments never get any inkling about. If the technical leadership is not engaged, within the first year, QA drops from 90% to 66% as a representative of PROD. Soon development teams will see PROD defects where developers and testers swear that they have thoroughly tested but fails in PROD because that environment is now supporting new type of data that the code had not anticipated.
Problem is with communication. Technical teams needs to be actively engaged with business teams to understand ever changing landscape of retail processes. New attributes may be flowing into dataload - which none or if lucky, only a portion of IT team may be aware of. New price rules, new contracts, new layouts.... While most of it is supposed to be transparent to development teams, it is often not! Teams may have implemented some customization that get affected or affects work done by business user. Worst case scenario, as I observed with one customer, is for the business team to start coming up with their own independent processes, working with non-e-commerce IT teams to create solutions in other parts of the organization such as ERP systems. And end up choosing solutions that deviate from WebSphere Commerce recommended patterns of solution.
How do we deal with this
There should be a monthly interlock with business analysts and technical architects - where requirements are continually re-evaluated and validated. While meetings can solve one side of the problem, often truth is in data! Hence there should be some process put in place to ensure QA and PREPROD environments are updated with any non-transactional change that is introduced in PROD - in that sense almost anything that is affected through Commerce management center is a good candidate for replication in lower environment.
One way to ensure this replication is automated is via the use of Data Extract utility. Folks that are familiar with Data load, knows how the framework is geared to read up comma separated values in a specific format being mapped through an XML and used to created BOD objects to be loaded into WCS DB. Extract is just a reverse of that process - from DB data is read into BOD and fields from BOD are mapped to field columns that are written into CSV. The framework is exactly the same and the XML file used are very similar. In FEP8 - there is ability to add custom SQLs to the extract process by just inserting them within relevant tags - use of that for extracting promotions is documented in the knowledgecenter and a similar approach should work well for other parts of the DB : http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.data.doc/tasks/tmlextractpromotiondata.htm?lang=en.
Things to Watch out
One caveat is that attachments require an associated process to collect and FTP images/htmls that they may refer to -- these may have been directly deployed to web server or CDN - hence a separate process is needed to collect them and bring them down to suitable location for lower environments. Another hurdle I faced was in handling custom catalog associations that were implemented through custom code than through data model or BOD definition - so, the extract would need to mimic the same customization to ensure right information is extracted. Another was related to store and catalog organization model - especially if sales catalog and categories were directly created in PROD - then extra care needs to be taken to replicate those first. Lastly, inspite of the utility giving great capability, some data massaging may still be required.
It is worth investing in such a solution for any ecommerce project. For one, it helps to keep lower environments refreshed. But same process can be used to provide outward feeds from WebSphere Commerce - for example, sending catalog data to your ratings and reviews provider or your analytics provider.