This recent GigaOm blog on the increasing need for “data scientists” is just another signal that we have indeed entered a fundamentally new era of IT, what I call in The Learning Layer the Era of Adaptation—an era in which our systems routinely learn from their experiences with us, and beneficially adapt based on this learning. And for our systems to be able to adapt to our needs and preferences, they need to be effectively predictive, and to be effectively predictive they need to be able to make good inferences, and to make good inferences, they need a foundation of lots of data.
Until recently, the bottleneck in this chain was that we simpy didn’t have access to the data required. But now vastly more data is being generated and captured than just a few years ago. Online click streams, postings, and direct feedback (e.g., the ubiquitous “like” button) are just the tip of the data iceberg—use of mobile devices and positional tracking promises to generate another order or two more of additional raw data. In the not too distant future, we’ll likely add to this corpus of available data things like physiological information (yeah, there’ll be some privacy issues to work out . . .).
Once we have the data, that’s where the data science comes in—various data conditioning techniques need to be applied, and then importantly, the right machine learning algorithms need to be brought to bear that will wring the most predictive power from the available data, taking care to neither under-interpret or over-interpret. So “Data+Algorithms” is clearly at the core of the competition in the Era of Adaptation. And it’s not just germane to commercial IT companies–as I wrote a few years ago, Data+Algorithms is increasingly at the heart of competitive success in a wide variety of industries. Data, and making meaning from data, can be a truly disruptive force–excellent news for data scientists or those who are interested in becoming one!