Skip to content

Engineering Serendipity

Serendipity, the notion of unintentional but fortuitous discoveries, has become a trendy concept of late, and particularly with regard to system platforms that can help foster it for us. For example, John Hagel’s recent book, The Power of Pull, addresses this idea in some detail, and very recently Eric Schmidt, Google’s CEO mentioned electronically calculating serendipity in the context of an interview on Google’s technology directions.

Although just about everyone would agree that some of our systems already serve to bring to our attention unanticipated but useful nuggets of information and people of which we would otherwise remain unaware, the topic has also generated some controversy—perhaps partly because of a potential over-selling of the degree to which today’s systems can truly facilitate useful serendipity, and partly because of a nagging worry that serendipity that is “engineered” may be engineered to serve the hidden purposes of the engineer.   

I discuss some approaches in The Learning Layer that address both of these issues. Engineering serendipity really amounts to delivering automatically generated recommendations of items of content or people. To do this well, the inference engine that generates a recommendation for you has to strike the right balance that lies between the extremes of delivering stuff you already know about and stuff that is so far afield it is very unlikely to have any useful serendipitous effect. And the degree to which that balance can be effectively struck is a function of the behavioral information the inference engine has to work with and its ability to wring the best possible inferential insights from that information. 

Progress on more sophisticated inferential insights is really where all the AI action is these days, and it is inevitable that these capabilities will become ever more powerful, and the resulting recommendations ever more prescient and helpful. But that then amplifies the second concern—how do we know the basis for the recommendations we receive? And even more to the point, how do we know we aren’t being manipulated? 

A solution to this issue that I discuss at length in the book is that recommendations, in any of their various forms (e.g., an advertisement is best thought of as a recommendation), should always be coupled with an explanation of why the recommendation was delivered to you. A good, detailed explanation will inevitably provide you with even more insights that can spur a useful serendipity. And an informative explanation can do even more—it can create a greater degree of trust in that system-based engineer of serendipity. Call it engineering serendipity with transparency. We would want nothing less from a friend doing the recommending, so why not demand the same from our systems?  

3 Comments Post a comment
  1. Engineering Serendipity +1 🙂

    I’m a long-time believer in a future where information consumption will predominately come from recommender systems and not search. Although as you point out this will require much subtlety in the algorithms, in order to avoid the filter bubble issue, and some randomness thrown in to shake things up.

    I would also raise concern about the potential manipulation that the non-ethical data scientist could facilitate. This is where I believe that we not only evidence but also total transparency. And by this I mean that the recommendation system should be very clear on its ulterior motive in making the recommendation; maybe its trying to scare me about X so it can get my to buy Y. None of us want sly recommendation systems :/

    November 20, 2013
    • Thanks for your comment Marie! Yes, ultimately trust is clearly as important with respect to interactions with intelligent systems as with people. As I mentioned in the blog, providing additional transparency by delivering explanations for recommendations can go a long ways in that regard. But, of course, even an explanatory system could be programmed to be manipulative, so the trustworthiness of the system provider remains key.

      November 20, 2013

Trackbacks & Pingbacks

  1. Creative Combinatorics « The Learning Layer

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: