Engineering Serendipity

Serendipity, the notion of unintentional but fortuitous discoveries, has become a trendy concept of late, and particularly with regard to system platforms that can help foster it for us. For example, John Hagel’s recent book, The Power of Pull, addresses this idea in some detail, and very recently Eric Schmidt, Google’s CEO mentioned electronically calculating serendipity in the context of an interview on Google’s technology directions.

Although just about everyone would agree that some of our systems already serve to bring to our attention unanticipated but useful nuggets of information and people of which we would otherwise remain unaware, the topic has also generated some controversy—perhaps partly because of a potential over-selling of the degree to which today’s systems can truly facilitate useful serendipity, and partly because of a nagging worry that serendipity that is “engineered” may be engineered to serve the hidden purposes of the engineer.   

I discuss some approaches in The Learning Layer that address both of these issues. Engineering serendipity really amounts to delivering automatically generated recommendations of items of content or people. To do this well, the inference engine that generates a recommendation for you has to strike the right balance that lies between the extremes of delivering stuff you already know about and stuff that is so far afield it is very unlikely to have any useful serendipitous effect. And the degree to which that balance can be effectively struck is a function of the behavioral information the inference engine has to work with and its ability to wring the best possible inferential insights from that information. 

Progress on more sophisticated inferential insights is really where all the AI action is these days, and it is inevitable that these capabilities will become ever more powerful, and the resulting recommendations ever more prescient and helpful. But that then amplifies the second concern—how do we know the basis for the recommendations we receive? And even more to the point, how do we know we aren’t being manipulated? 

A solution to this issue that I discuss at length in the book is that recommendations, in any of their various forms (e.g., an advertisement is best thought of as a recommendation), should always be coupled with an explanation of why the recommendation was delivered to you. A good, detailed explanation will inevitably provide you with even more insights that can spur a useful serendipity. And an informative explanation can do even more—it can create a greater degree of trust in that system-based engineer of serendipity. Call it engineering serendipity with transparency. We would want nothing less from a friend doing the recommending, so why not demand the same from our systems?