Make Your Data Confess Its Secrets

The process of data mining refers, in general, to the act of applying various mathematical and statistical techniques to massive amounts of often disparate data in the hope of teasing out significant details. These details can then be leveraged for future (usually business or scientific) endeavors.

To make data mining work, one requires access to historical (company) data. The process involves looking for patterns among various independent pieces of data. The results can be leveraged to make better decisions in the future.

As a field, data mining is still relatively new. While some of the statistical techniques employed in data mining have been in existence for a century or more, it has only been in the last few decades that we have collectively had access to the kind of computing power required to carry out analyses of massive amounts of data. And, some of the most important data mining techniques themselves have only arisen within the past 15-20 years, such as the training of artificial neural networks.

Thomas C. Redman, author of the informative book “Data Driven” (2008), quotes a past colleague at Bell Labs who said that, “Data do not give up their secrets easily. They have to be tortured to confess.” We agree with this statement wholeheartedly. To do data mining right, it requires the alignment of multiple skill sets, aptitudes, and opportunities including:

* knowing which types of business decisions can be aided by data mining
* having access to adequate amounts of historical data about your company, such as transaction data or customer data
* being able to judge which types of data are eligible for analysis and which should be ignored
* having the ability to clean up dirty data
* knowing which of the many available data mining techniques to apply to the problem at hand
* understanding how to interpret the results of the analysis properly so that they can be applied to making better business decisions

When carried out properly, data mining can surprise you by making your data spills its secrets about your business past in ways that better inform your business future. By trusting the data that data mining can provide, you may be led to new vistas for your business’s future that you hadn’t even noticed before.