• Volume.  The volume of data to be considered can be very large.  This is what we discussed above
• Variety.   Data can be of diverse form, beyond the relational data usually handled by relational databases.  Data can be unstructured text, eg documents.  It can also be signal such as sound, images, video.  Handling such data requires tools beyond what traditional databases offer.
• Velocity.  Data can be in motion i.e. obtained from a continuous stream.  Think of video feeds. Think of social media such as twitter as a text stream. Think of the data collected by a sensor, for instance the electricity consumption as measured by a smart meter.  Managing data in motion requires quite different technologies than managing data stored in a data warehouse.

• Veracity.  Data can be sure, data we can trust.  But data can also be known only to some extent or to some confidence level.  Think for instance about sensor data.  As is the case for any physical measure, data is only known up to some precision.  Data can also be predicted or simulated in some way.  For instance if it has been computed using some analytics technique it may come with a confidence interval, or a likelihood attached to it.

As we can see, the aim of big data is to harness all sorts of data.  As Paul Taylor, a colleague of mine, puts it, big data is about all your data.  This is a significant extension to the corporate data that was mainly stored in relational databases until recently.  Handling all kind of data spurred the development of new IT technologies and tools, which is what big data refers to.

A quite simple idea led to the best known tool in this area, namely Hadoop.  Instead of bringing data to the computer system that will analyze it, Hadoop sends the computation to where the data is.  Said differently, Hadoop is a tool to distribute computation across a distributed system storing the data to be analyzed.  Hadoop also contains the distributed file system that you can use to store the data.  Hadoop is an open source software project based on the MapReduce method that was  first developed at Google.

From a database perspective, Big Data means moving away from relational databases.  This is captured by the NoSQL (Not Only SQL) acronym for databases. 

SQL refers to relational databases.  These databases where developed to ensure consistency of data through well defined transactions.  More formally, the set of properties that  guarantee that database transactions are processed reliably is know as ACID (Atomicity, Consistency, Isolation, Durability) in computer science.

The advent of the web introduced the need for distributed data bases at a large scale.  Think of the data stored for Google search, Amazon, Ebay, or Facebook.  Given the number of requests these sites get, one needs to replicate the data on many servers, then distribute queries to these servers.  Can we then guarantee that the data stays consistent between all these servers while at the same time ensuring 24x7 responses to queries?  The answer is that we cannot.

This was first conjectured by Eric Brewer in 2000, then proved formally by Seth Gilbert and Nancy Lynch of MIT.   Brewer stated the problem as follows. 

It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:

• Consistency (all nodes see the same data at the same time)
• Availability (a guarantee that every request receives a response about whether it was successful or failed)
• Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system)

This became known as the CAP theorem.   Since any distributed system can only have two of the three features, looking at which pair is supported by a given system is a nice way to analyze it.  For instance, a relational database, even a distributed one, will ensure consistency and availability, but isn't robust vis a vis partition fault.  This post by Nathan Hurst provides a classification based on CAP of current NoSQL systems.

The CAP theorem is kind of fascinating, because it captures the very reasons why the distributed systems that power massive webscale systems such as Google, Amazon, Ebay, or social networks work in practice. This is well documented in The CAP Theorem, The kool aid Amazon and Ebay have been drinking

Readers willing to know more about the CAP theorem and its consequences can read the wikipedia entry, this recent article by Eric Brewer, and this interesting discussion.

There are many more tools available in big data beyond Hadoop and NoSQL databases.  Readers willing to know more about these and other big data tools can go to the Big Data University site.