Data Mining is Not Evil

“NSA, snooping, spying, violation of personal privacy, Big Brother.”

These are just some of the buzzwords that come to mind these days when most of us hear the term “data mining.”

At MindEcology, we are here to tell you that data mining is not evil. At least not inherently. Of course, like any effective tool, technique or technology, data mining can be used for good, evil, or purposes that don’t necessarily fit into any such Manichaean category at all.

What is Data Mining, Anyway?

Data mining involves the use of computers to assist humans in combing through enormous amounts of data in order to find patterns that are potentially useful. In the hands of governmental bodies, the “data” might be personal communications, for example, while “patterns” might be indicators of nefarious acts being planned or committed.

When it comes to business, on the other hand, data mining takes on a different meaning altogether. Depending upon the application, “data” might refer to any number of things, such as:
* customer addresses
* website visit logs
* user comments on blog websites
* social media chatter
* order history
* electricity usage patterns
* equipment utilization rates

and so much more. Meanwhile, in the business context, “pattern” might also refer to all sorts of things, including:

* correlations between two or more variables
* messages (intended or unintended) hidden within social media or blog posts
* buyer behaviors or characteristics that can serve as predictors of future purchases
* patterns that are as-of-yet un-hypothesized

What Data Mining can do for Your Business
Data mining differs from more traditional data analysis techniques in three key ways:

1. It can work not only with structured data (i.e., data set up in pre-defined fields in a database, in a columns-and-rows format), but with unstructured data, as well
2. It does not require that you (the data miner) already know what you are looking for when you embark on a new project; rather, if you set up the analysis properly then the patterns can just emerge from the data
3. When combined with “big data”-inspired query tools, databases, and distributed computing techniques, data mining can be utilized to analyze very, very large data sets in a way that more traditional techniques could not

The result? The ability to see meaningful patterns that were just not visible when searched for via traditional means. These patterns can be made actionable by turning them into business decisions that directly affect the bottom line

Data mining was created by technologists, statisticians and practitioners in the field who wanted to answer the call to make better use of the petabytes of data that surrounds us each day. Traditional statistical and analytical techniques no longer remain valid or workable in many of the situations that analysts face today.

Does that sound evil? Not really. Of course, all of us want our personal privacy protected at reasonable levels. Ultimately, data mining is not about snooping where one doesn’t belong. Rather, it’s just about using a more powerful “microscope” to see more deeply into the data that we already have access to.