Managing your organisation's capabilities is the third of my critical success factors for big data in business. In this final post on this topic, I shall expand on what this means.
Ensure alignment of the organisation and project. Simply designing and building a big data application will not ensure its success. Consideration must be given to who is going to provide the infrastructure for the application and more importantly who is going to operate it and how. The application will become costly or even legacy if no thought is given to who is going to maintain it and how. Most importantly, a big data project is likely to disrupt how the business currently operates, and so the project needs to consider the business change required to make full use of the application and how it will transform. This spans process, structural and cultural change. All parts of the organisation involved in the project need to focus on a common goal to succeed. Sound governance must be put in place to deliver and sustain the project to realise the desired benefits.
Apply an ethical policy. Incorporation of new data sources into big data systems coupled with significant improvements in the capabilities of analytics technology provides organisations with opportunities to gain far greater and far deeper insight than ever before. For example, bringing together corporate records on customers with log files on customers' use of applications, social media data and statistical modelling techniques, allows a rounded, up-to-date view of individuals to be formed. However, this does not mean that any insight should be derived nor should insight necessarily be acted upon. Consideration should be given to the original purpose for which the individual gave information about themselves and whether an organisation's intended use of that data is reasonable, and indeed seen to be reasonable. Moreover, data quality becomes more important with big data because errors are amplified. Poor quality data may also detract from minimising false positives and false negatives. So if resulting actions are wrong, an organisation risks reputational damage or contravention of regulations.
Employ the right skills. Organisations should utilise their existing business intelligence staff in big data projects: big data is not something separate, but it augments what these people do already. However, skills development is needed to be successful with big data. Firstly, big data systems utilise large scale infrastructure which requires skills to design and operate it successfully. Secondly, skills in statistics and programming are needed to reflect the business opportunity in the resulting applications. Taking an approach which only utilises data warehousing skills will simply result in today's techniques being applied on big data technology, thereby not fully exploiting the opportunity. As an aside, organisations should recognise that Hadoop is not necessarily a replacement for a data warehouse: they have different design points. What is well suited to one may not be best suited to the other, and the skills required to build and operate each system differ. Ultimately, maximising the business return from a big data system is more than simply choice of technologies, and one of the factors that must be taken into account is acquisition of the right infrastructure and analytics skills to succeed.
In my first four posts on this blog I have described what I believe are the critical success factors for big data in business. In summary these are: