Skip to content

Posts from the ‘Uncategorized’ Category

Learning from Netflix

Some more recent thoughts from Netflix: “the importance of Netflix’s recommendation engine is actually underestimated”

The Learning Layer

Netflix recently published a blog that lays out some of their experiences with their recommender system.  The blog is notable in that Netflix was one of the pioneers of e-commerce recommendation engines, has one of the most famous recommendation engines, and packs a lot of details and good insights into the blog.

Here are a few takeaways:

75% of what people watch via Netflix is due to recommendations.  And given how impressively recommendations drive sales for businesses such as Amazon, it is not surprising that sophisticated recommender systems are becoming the norm in e-commerce.

“Everything is a Recommendation.”  Netflix uses this phrase to underscore the point that most of its interface now personalizes to the user. This approach is an inevitable direction for user interfaces most generally since it is clearly technically feasible and delivers business results.

Optimize for accuracy and serendipity.  People are complex and have…

View original post 248 more words


Self-inception is by far my most popular Learning Layer blog post of all time. It is a meme that not only sticks, but frequently seemingly spontaneously and independently re-emerges in minds around the world. Almost every day at least a handful of people arrive at this blog via independent searches for the term “self-inception.”

The Learning Layer

I recently saw the cool new movie Inception—where the term “inception” means the implanting of an idea into the brain of a target by way of hacking into the target’s dreams. And as those of you who have already seen the movie know, the plot plays with this idea in a recursive way—the dream hacking is conducted in dreams within dreams, making for a mind bending movie experience, and, of course, sufficient ambiguity between dreaming and reality to allow for many Hollywood sequel directions . . .

Watching the movie was a particularly enthralling experience for me because two key themes flowing through The Learning Layer are dreams and recursion (and, ok, because I’m a bit of geek I suppose). Yeah, The Learning Layer is most fundamentally a book about next generation organizational learning, but it’s also a book of many layers, and the undercurrents of dreams and recursion…

View original post 571 more words

Portable Personalization at Your Service

As the social layer continues its inexorable expansion throughout organizations, a rich set of information about what the users of these social platforms find interesting and beneficial and not so interesting also inexorably expands. This is information that in the old days of computing (i.e., last year :)) would have simply been ignored and discarded.

But now there is an opportunity to put that information to work. From such social-based big data we can apply some fancy algorithms and make some very useful inferences. We can, for example, get a good sense about specific topics that are of interest to particular people. We can do that based on the realization that while each individual action of users tells us a little something about their interests, their collective actions can tell us a lot about their interests. For example, if you view a post or document associated with a given set of topics, it provides a hint that you might have a continuing interest in those topical areas. If you view a couple dozen documents associated with those same topics, it strongly suggests that you have a continuing interest in those topical areas. And, of course, viewing something is just one kind of behavior to consider–there are also actions such as contributing content, likes, comments, following, and so on. This rich variety of behaviors enables inferences to be even better. So applying this approach, systems can continuously learn from you, thereby anticipating your needs and not just reacting, and best of all, without you having to do any extra work!

Now, as a practical matter, even computers can’t be expected to actually always re-compute from scratch inferences of interests based on full behavioral histories. A method is needed to conveniently store a compressed, but still useful, summary of your interest profile. We can do just that with a little math construct that you may remember from your linear algebra class: a vector. A vector is just a fancy word for an ordered set of values. So (2, 33, 7, 12) is a vector. We can generate an interest vector by encoding what we have inferred about your degree of interest in various topics as a set of values (usually normalized in the range of 0-1) that correspond to specific topical areas. For example, (0.04, .0.56, 0.21, 0.78, . . .), where the 0.04 value corresponds to a first topic, 0.56 corresponds to a second topic, and so on. In this case the values imply that we have inferred that you have a much higher interest in the second topic (value of 0.56) than the first topic (value of 0.04). Now let’s say you begin viewing or liking or commenting on some new content associated with the first topic. It is then likely that 0.04 should increase. How much of an increase? That would depend on many factors that the inferencing algorithm would take into account.

So that’s how interest vectors work—they are continuously updated as new information becomes available and they can span hundreds or even thousands of topics, so the interest profile can be extraordinarily fine-grained. This allows them to be used to make very intelligent recommendations, and they can even serve auxiliary purposes, such as being compared among people to identify and match people with similar interests. And similar to interest vectors, expertise vectors can also be generated from behavioral information by using somewhat different algorithms and behavioral perspectives. For example, just viewing a document associated with a topic tells us perhaps a little bit about interests with respect to the topic, but probably nothing about expertise with respect to the topic. However, if you posted content associated with that topic and it received significant attention from those who have already been inferred to have higher-than-average expertise, that would tell us something about your expertise with respect to the topic. As in the case of interest vectors, an expertise vector is a personalization vector that can be applied to make intelligent recommendations, including recommendations of people whose inferred expertise may be quite relevant to a particular item of content you are viewing.

These personalization vectors that respectively summarize your interests and your relative expertise in various topical areas have another very useful property—they can be portable. That is, you could begin using a system you had never used before, but if you provided it your interest and/or expertise vector it could immediately begin being personally anticipatory with you, assuming that the system involved similar topical areas to those encoded in your personalization vectors. Suddenly it’s a very different world than what we have historically put up with—in this new world of portable personalization, all systems become much more intelligent by becoming immediately responsive to your particular interests and expertise! And with social networking becoming ubiquitous in our business as well as personal lives, this new world of portable personalization is just around the corner.

Does Personalization Pave the Cow Paths?

I notice the issue of poorly engineered recommendation systems causing a “filter bubble” continues to surface. This blog is still very relevant with respect to how to do it right. And since this blog was originally published there have been additional advances, such as techniques that identify contrasting subsets of people’s interest profiles that otherwise more generally correlate (for which I received a patent).

The Learning Layer

Michael Hammer, the father of business reengineering, famously used the phrase “paving the cow paths” to describe the ritualizing of inefficient business practices. Now the pervasiveness of personalization of our systems is being accused of paving our cow paths by continuously reinforcing our narrow interests at the expense of exposing us to other points of view. This latest apocalyptic image being painted is one of a world where we are all increasingly locked into our parochial, polarized perspectives as the machine feeds us only what we want to hear. Big Brother turns out to be an algorithm.

I commented on this over at Greg Linden’s blog, but wanted to expand on those thoughts a bit here. Of course, I could first point out the irony that I only became aware of the Eli Pariser’s book about the perils of personalization, The Filter Bubble, through a personalized RSS feed…

View original post 562 more words