The Explanation Engine Redux

In my 2010 book, The Learning Layer, I made the case that explanatory capabilities in AI systems are an imperative, but I also described why providing such a capability that is sufficiently informative is highly challenging. Over the next few months after The Learning Layer was published, I published a series of blogs that discussed this topic in more detail, describing how a separate function is actually required, an “explanation engine,” to provide intelligible explanations for the more inscrutable, mathematical-based capabilities that in reality make the decisions in advanced AI systems. The specific example I gave as follows was with respect to recommender systems, but it applies more broadly for any sufficiently complex AI-based system.

items to recommend are going to be the product of various high-powered mathematical evaluations and manipulations of vectors and matrices. How can they possibly be compactly conveyed in an explanation to the recommendation recipient? After a while we realized that they really cannot—explanations necessarily have to be an approximation of the actual thought processes of the recommendation engine. A very useful approximation, but an approximation nevertheless. We also realized that to do them right, it was an architectural necessity to have a dedicated engine for explanations complementary to, but separate from, the powerful but inarticulate recommendation engine.

I further made the point that this two-part architecture of, 1) a complex and inscrutable decision-making function coupled with, 2) a separate explanatory function to explain the decisions of the inscrutable decision-making function to others (or even to itself, which I described as self-inception) basically recapitulates the evolution of the analogous two-part structure of human cognitive processes.

Turns out we are no different—we humans have a language-based explanation engine that explains to others, as well as ourselves, why we do the things we do. And it has become increasingly apparent from psychological studies over the past decade or two that our explanations are really only approximations of our underlying, unconscious decision making. In fact, it has been confirmed by recent brain imaging studies that although we tend to believe that our conscious and logical explanation engine is making the decisions, in reality it is just providing an after-the-fact explanation for a decision already unconsciously made elsewhere in our brain. . . our brain most fundamentally is nothing more than a vast, weighted network of neurons. Decisions are assessed and made by inscrutable interactions among this network. We humans, fairly recently in our evolutionary history, have developed a language-based explanation engine that has essentially been grafted on to this underlying network that enables us to effectively communicate with other humans. To achieve a reasonable compaction, the explanations we give must necessarily typically be extreme simplifications, and quite often will also have some degree of fabrication woven in.

Later, I was delighted to read Daniel Kahneman’s book, Thinking, Fast and Slow, which was published a little over a year after The Learning Layer, as well as after my follow-on blogs. Kahneman explained that human decision-making processes can be categorized into two fundamental processes: a first process that is fast, intuitive, pattern matching, and heuristic in nature, which Kahneman termed, “System 1,” and a second slower, more logical, reasoning and explanatory capability, which Kahneman termed, “System 2.” It is readily apparent, of course, that System 1 maps directly to the inscrutable decision-making function and System 2 maps to the explanation engine that I described with respect to AI systems. And, in fact, Kahneman’s book was based on earlier insights from cognitive research such as the those that I mentioned in The Learning Layer and in my blogs. Kahneman’s book greatly popularized these insights from the field of cognitive science, and for me, provided further credence that there was a strong parallel between the way AI would necessarily need to evolve and the way the human cognitive processes had evolved.

I also emphasized that we tend to think of ourselves as our explanation engine rather than the underlying, inscrutable processes that studies had shown actually drive our decision making.

And our explanation engine continuously explains itself to us—so that we come to believe that we are our explanation engine, that there is a clear logic to what we do, that we have an explicit freedom to decide as we wish, and a variety of other explanatory conceits.   

We humans believe that we are our language-based explanation engine, and we therefore literally tell and convince ourselves that true meaning is solely a product of the conscious, language-based reasoning capacity of our explanation engines.

And because we think of ourselves as our explanation engine, it is not surprising that our initial focus in developing AI was almost solely directed toward embedding this type of symbolic, rule-based, logical capability within our AI, and more broadly, within our computing systems in general. But that’s not the way nature proceeded, and if we wanted to develop truly intelligent systems, I advocated that we needed to take lessons from nature in this regard!

Inarticulate inferencing and decision making capabilities evolved over the course of billions of years and work quite well, thank you. Only very recently did we humans, apparently uniquely, become endowed with a very powerful explanation engine that provides a rationale (and often a rationalization!) for the decisions already made by our unconscious intelligence—an endowment most probably for the primary purpose of delivering compact communications to others rather than for the purpose of improving our individual decision making. So to focus first on the explanation engine is getting things exactly backward in trying to develop machine intelligence. To recapitulate evolution, we first need to build intelligent systems that generate good inferences and decisions from large amounts of data, just like we humans continuously and unconsciously do. And like it or not, we can only do so by applying those inscrutable, inarticulate, complex, messy, math-based methods. With this essential foundation in place, we can then (mostly for the sake of our own conceit of consciousness!) build useful explanatory engines on top of the highly intelligent unconsciousness.

Well, since The Learning Layer and these subsequent blogs, my advice has indeed been heeded (although correlation should certainly not be confused with causation!), and the significant AI advances, particularly in the past decade, have come from advancing those “inscrutable, inarticulate, complex, messy, math-based methods” I mention in the above passage, primarily in the form of neural network-based technology, specifically deep learning. And in particular, with the advent of transformer-based deep learning technology in 2017, we have achieved remarkable facility with language in the form of large language models (LLMs) and their conversational interfaces that are profoundly changing our world.

And now that we’ve walked down memory lane, I want to get to the real point of this piece. We now have even more evidence that the path to artificial general intelligence (AGI) is by recapitulating the evolution of the explanation engine (or System 2) in humans. And that further evidence derives from the advances in the application of Chain of Thought (CoT)-based techniques in improving LLM-based performance. Chain of thought is a technique of prompting LLMs to explain their reasoning steps in deciding on a conclusion or an action, often performed in an iteratively interactive manner. It has been demonstrated to greatly improve the resulting output from LLMs. This prompting method is classically performed by a user, and in that case the explanation engine is partially external to the LLM-based system. But that need not be the case—the chain of thought prompting can be fully performed automatically by another system or sub-system of the LLM. And such an autonomous architecture should sound familiar—it’s basically the explanation engine, whether embodied in minds or machines!   

This automated, built-in chain of thought capability is now a key direction of all the major LLM providers that is designed to achieve a more System 2-like capability to complement the inherently System 1-like capabilities of LLMs. These explanation engine-based architectures therefore promise to be ubiquitous. That such an architecture seems required to flexibly achieve truly human-level capabilities adaptable to many different domains, and the fact that nature has evolved a similar architecture, leads me to the following generalized hypothesis.

Hypothesis:

1. Only large scale, connectionist-based correlative learning systems are flexible enough to provide the base support for effective real-world decision making and agency (e.g., human System 1, AI LLMs/foundation models)
   
2. Every such connectionist, correlative learning system requires a complementary chain of thought-type function to achieve arbitrarily good inferences and decisions (e.g., human System 2, AI chain of thought)

The first point we have learned the hard way over many decades of trying to build AI. The second point I outlined nearly a decade and a half ago and is further confirmed by the essential nature of CoT techniques in augmenting LLMs. This hypothesis suggests that even if the human language-based explanation engine/System 2 initially evolved primarily for social communicative purposes as I speculated in the blog passage above, it undoubtedly also is applied internally to facilitate logical reasoning that was unattainable by our System 1 alone, and thereby improving our decision making.

The most fundamental open technical question right now in AI is whether an explanation engine/System 2 functionality can be based solely on neural network-based systems or does it need to be augmented by symbolic-based systems. The human brain suggests it can be done with just connectionist models, so my bet is AI architected the same way will be able to achieve human level capabilities, but that we will necessarily want to augment these capabilities with specialized systems that may be more symbolic in nature, just as we currently employ to augment our own capabilities.

Learning from Netflix

Some more recent thoughts from Netflix: “the importance of Netflix’s recommendation engine is actually underestimated” https://gigaom.com/2014/11/22/why-technology-and-content-are-inseparable-at-netflix/

The Learning Layer

Netflix recently published a blog that lays out some of their experiences with their recommender system.  The blog is notable in that Netflix was one of the pioneers of e-commerce recommendation engines, has one of the most famous recommendation engines, and packs a lot of details and good insights into the blog.

Here are a few takeaways:

75% of what people watch via Netflix is due to recommendations.  And given how impressively recommendations drive sales for businesses such as Amazon, it is not surprising that sophisticated recommender systems are becoming the norm in e-commerce.

“Everything is a Recommendation.”  Netflix uses this phrase to underscore the point that most of its interface now personalizes to the user. This approach is an inevitable direction for user interfaces most generally since it is clearly technically feasible and delivers business results.

Optimize for accuracy and serendipity.  People are complex and have…

View original post 248 more words

Self-Inception

Self-inception is by far my most popular Learning Layer blog post of all time. It is a meme that not only sticks, but frequently seemingly spontaneously and independently re-emerges in minds around the world. Almost every day at least a handful of people arrive at this blog via independent searches for the term “self-inception.”

The Learning Layer

I recently saw the cool new movie Inception—where the term “inception” means the implanting of an idea into the brain of a target by way of hacking into the target’s dreams. And as those of you who have already seen the movie know, the plot plays with this idea in a recursive way—the dream hacking is conducted in dreams within dreams, making for a mind bending movie experience, and, of course, sufficient ambiguity between dreaming and reality to allow for many Hollywood sequel directions . . .

Watching the movie was a particularly enthralling experience for me because two key themes flowing through The Learning Layer are dreams and recursion (and, ok, because I’m a bit of geek I suppose). Yeah, The Learning Layer is most fundamentally a book about next generation organizational learning, but it’s also a book of many layers, and the undercurrents of dreams and recursion…

View original post 571 more words

Portable Personalization at Your Service

As the social layer continues its inexorable expansion throughout organizations, a rich set of information about what the users of these social platforms find interesting and beneficial and not so interesting also inexorably expands. This is information that in the old days of computing (i.e., last year :)) would have simply been ignored and discarded.

But now there is an opportunity to put that information to work. From such social-based big data we can apply some fancy algorithms and make some very useful inferences. We can, for example, get a good sense about specific topics that are of interest to particular people. We can do that based on the realization that while each individual action of users tells us a little something about their interests, their collective actions can tell us a lot about their interests. For example, if you view a post or document associated with a given set of topics, it provides a hint that you might have a continuing interest in those topical areas. If you view a couple dozen documents associated with those same topics, it strongly suggests that you have a continuing interest in those topical areas. And, of course, viewing something is just one kind of behavior to consider–there are also actions such as contributing content, likes, comments, following, and so on. This rich variety of behaviors enables inferences to be even better. So applying this approach, systems can continuously learn from you, thereby anticipating your needs and not just reacting, and best of all, without you having to do any extra work!

Now, as a practical matter, even computers can’t be expected to actually always re-compute from scratch inferences of interests based on full behavioral histories. A method is needed to conveniently store a compressed, but still useful, summary of your interest profile. We can do just that with a little math construct that you may remember from your linear algebra class: a vector. A vector is just a fancy word for an ordered set of values. So (2, 33, 7, 12) is a vector. We can generate an interest vector by encoding what we have inferred about your degree of interest in various topics as a set of values (usually normalized in the range of 0-1) that correspond to specific topical areas. For example, (0.04, .0.56, 0.21, 0.78, . . .), where the 0.04 value corresponds to a first topic, 0.56 corresponds to a second topic, and so on. In this case the values imply that we have inferred that you have a much higher interest in the second topic (value of 0.56) than the first topic (value of 0.04). Now let’s say you begin viewing or liking or commenting on some new content associated with the first topic. It is then likely that 0.04 should increase. How much of an increase? That would depend on many factors that the inferencing algorithm would take into account.

So that’s how interest vectors work—they are continuously updated as new information becomes available and they can span hundreds or even thousands of topics, so the interest profile can be extraordinarily fine-grained. This allows them to be used to make very intelligent recommendations, and they can even serve auxiliary purposes, such as being compared among people to identify and match people with similar interests. And similar to interest vectors, expertise vectors can also be generated from behavioral information by using somewhat different algorithms and behavioral perspectives. For example, just viewing a document associated with a topic tells us perhaps a little bit about interests with respect to the topic, but probably nothing about expertise with respect to the topic. However, if you posted content associated with that topic and it received significant attention from those who have already been inferred to have higher-than-average expertise, that would tell us something about your expertise with respect to the topic. As in the case of interest vectors, an expertise vector is a personalization vector that can be applied to make intelligent recommendations, including recommendations of people whose inferred expertise may be quite relevant to a particular item of content you are viewing.

These personalization vectors that respectively summarize your interests and your relative expertise in various topical areas have another very useful property—they can be portable. That is, you could begin using a system you had never used before, but if you provided it your interest and/or expertise vector it could immediately begin being personally anticipatory with you, assuming that the system involved similar topical areas to those encoded in your personalization vectors. Suddenly it’s a very different world than what we have historically put up with—in this new world of portable personalization, all systems become much more intelligent by becoming immediately responsive to your particular interests and expertise! And with social networking becoming ubiquitous in our business as well as personal lives, this new world of portable personalization is just around the corner.

Does Personalization Pave the Cow Paths?

I notice the issue of poorly engineered recommendation systems causing a “filter bubble” continues to surface. This blog is still very relevant with respect to how to do it right. And since this blog was originally published there have been additional advances, such as techniques that identify contrasting subsets of people’s interest profiles that otherwise more generally correlate (for which I received a patent).

The Learning Layer

Michael Hammer, the father of business reengineering, famously used the phrase “paving the cow paths” to describe the ritualizing of inefficient business practices. Now the pervasiveness of personalization of our systems is being accused of paving our cow paths by continuously reinforcing our narrow interests at the expense of exposing us to other points of view. This latest apocalyptic image being painted is one of a world where we are all increasingly locked into our parochial, polarized perspectives as the machine feeds us only what we want to hear. Big Brother turns out to be an algorithm.

I commented on this over at Greg Linden’s blog, but wanted to expand on those thoughts a bit here. Of course, I could first point out the irony that I only became aware of the Eli Pariser’s book about the perils of personalization, The Filter Bubble, through a personalized RSS feed…

View original post 562 more words

Our Fuzzy Social Graphs

One of the problems we often encounter with our social networks is the lack of “fuzziness” that they provide us with respect to our relationships—that is, with standard social networks you either have a relationship with another person, or you don’t. I discussed this issue in The Learning Layer:

People clearly comprise networks, and the relationships between people are not necessarily just digital in nature. We all have some relationships that are very strong, and others that are much weaker. Some people are our soul mates, some are friends, some are colleagues, and some are just acquaintances. There are shades of gray in our social relationships, just as in the case of relationships among items of content and topics. And there are different types of relationships among people, and among people and content that should be explicitly recognized. Some of these types of relationship may, in fact, be digital—for example, someone is your classmate or is not; someone is an author of an item of content or is not. But some types of relationships, such as the degree of similarity of preferences between two people, or the degree of interest a person or a group of people have with regard to a topical area, clearly will not be digital. They will be much more nuanced than that.

The inability to manage our online relationships in a more nuanced (i.e., fuzzy) fashion leads to ever bigger headaches as the scale of our social networking connections (i.e., our “social graphs”) increase. I had some comments on the way social networks have attempted to address this problem in the blog post, Social Networking and the Curse of Aristotle. At the end of the post, I mentioned that leveraging the power of machine learning provides a way for us to share activities and information in better accordance with the specific wishes we would have if we actually had the time to fully consider whether to share a particular item with each specific person to whom we are connected.

Along these lines, a recent study confirms just how well our interactions within a social network (e.g., Facebook) can be used to infer the strength of our real-world relationships. And, in fact, under the covers, Facebook’s algorithms already use this type of information to decide what to deliver in your feeds and what not to deliver. Likewise, Synxi learning layer apps do something quite similar in recommending other users or their content to users of enterprise social platforms. So machine learning is already on the job for you—your social graph is fuzzy, whether you know it or not!

Enterprise Personalized

The rise of machine learning-based personalized discovery features over the past few years is one of the biggest stories of IT. The statistics are truly staggering. For example, even as of several years ago, over 30% of Amazon’s sales were reportedly due to their personalized recommendations—the figure is no doubt even higher now. LinkedIn has reported that fully 50% of their users’ connections, group memberships, and job applications are driven by their personalized discovery features. And 75% of what people watch via Netflix is due to personalized recommendations. In addition, of course, targeted advertisements can be considered a form of personalized recommendations, as can personalized search, both of which have largely replaced their non-personalized precursors.

So in the consumer world automatic personalization has become an indispensable feature for users and a competitive imperative for providers. What about in the enterprise? Not so much—until now, that is. I wrote The Learning Layer to lay out the path toward making adaptive, personalized discovery a core feature of enterprise IT, and we at ManyWorlds are excited that our Synxi-brand technology is now making that vision a reality!

We are delivering adaptive discovery apps for the major social platforms that continuously learn from users’ experiences and apply this learning to provide users with real-time, personalized recommendations of knowledge and expertise, and which are sensitive to the context of their current activities. Even better, we also have connectors among these apps that extend a layer of learning across platforms. That means users can receive cross-contextualized and personalized recommendations of knowledge and expertise from one platform (e.g., SharePoint) based on what they are doing in another platform. And finally, we have booster products for enterprise search that enable search results to be personalized and/or additional personalized content to be recommended based on the context of a specific search result. That provides users, for the very first time, an enterprise search experience that tops the search experience internet search providers can deliver.

These learning layer technologies are collectively leading toward enterprises becoming truly personalized. And an enterprise personalized is an enterprise that is more productive, as well as being an enterprise that is more compelling to be a part of and to work with.

Just the Facts Ma’am?

Now that Siri has a bona fide competitor, Google Voice Search, a bit of a kerfuffle has emerged with regard to personalities or lack thereof of these assistants. While Siri strives to project some personality by being conversational and peppering her responses with a bit of whimsy, Google Voice Search is all about just giving us the facts. Each approach has advantages and its vocal adherents. And as the systems’ capabilities leap-frog one another with each new version, the latest incarnation of Google Voice Search seems to have gained some speed and effectiveness advantages versus the current incarnation of Siri. Of course, both of these incarnations promise to be fleeting given the pace of the respective development cycles.

Although Google labels their product “search,” the functionality has clearly already morphed more generally into a recommender—i.e., providing suggestions given a context of various of kinds. This trend is a reflection of a generalization noted in The Learning Layer—plain old search is really best considered just a recommendation in which the context is of a particular type, i.e., a search term provided by the user. The inevitable next step in general-purpose recommender technology is delivering “meta-recommendations”—that is, explanations as to why the recommendation was provided, particularly when an explanation is specifically asked for by the recommendation recipient. A capacity for a limited degree of explanatory capability has already been incorporated into the Apple and Google gals to some degree.

Then comes the really interesting advance—making the recommendations and even the explanations adaptive to the user.  That is, learning from her experiences with us to adapt her recommendations and explanations accordingly. Which is followed by one more short step in which aspects of her overall personality become adaptable to us and our particular circumstances as well. A little humor when called for, a bit of sympathy at other times; and all the while learning as to what works best and when, and tuning accordingly. I’ve got a feeling that at that point, which at the current pace of innovation, is not far away, always just providing the facts will be perceived to be somewhat stilted behavior—coming off like a cheesy movie version of AI of the 1960s.

So my guess is that there are times when, indeed, we are all Joe Friday’s, but more often than not we’ll welcome more than just the facts.

Learning from Netflix

Netflix recently published a blog that lays out some of their experiences with their recommender system.  The blog is notable in that Netflix was one of the pioneers of e-commerce recommendation engines, has one of the most famous recommendation engines, and packs a lot of details and good insights into the blog.

Here are a few takeaways:

75% of what people watch via Netflix is due to recommendations.  And given how impressively recommendations drive sales for businesses such as Amazon, it is not surprising that sophisticated recommender systems are becoming the norm in e-commerce.

“Everything is a Recommendation.”  Netflix uses this phrase to underscore the point that most of its interface now personalizes to the user. This approach is an inevitable direction for user interfaces most generally since it is clearly technically feasible and delivers business results.

Optimize for accuracy and serendipity.  People are complex and have diverse tastes, moods, etc.  A good recommender will try to help recommendation recipients keep from just re-paving their own cow paths.

Diversity of behavior types trump incremental algorithm advances.  Given already advanced algorithms, recommender improvement comes grudgingly if based on limited behavioral types (e.g., just course-grained ratings). Deriving inferences from a greater diversity of behaviors, as well as contextual cues, delivers greater advantages.

Explanations of recommendations are critically important.  Recommendations that are accompanied by explanations as to why the recommendation was delivered to the recipient are perceived to have greater levels of authority and credibility, and promote trust.

These are important takeaways for e-commerce, but they are just as applicable to enterprise adaptive discovery systems. In fact, as I discuss in The Learning Layer, because there are often more behavioral types, as well as topical structures, with which to work in the enterprise environment, adaptive systems in the enterprise have some inherent advantages versus those in purely consumer-facing environments.

I am often asked by executives about the value proposition of adaptive discovery or learning layer systems in the enterprise. A moment’s reflection on the ability to deliver the right knowledge and expertise to the right person at the right time generally suffices. But from a macro-economic standpoint, simply observing how recommendation systems have transformed e-commerce also should make the point.  On-line businesses without world-class personalized discovery capabilities simply cannot hope to compete going forward. The same lesson surely applies to systems within the enterprise.

The Era of Adaptive Education

No one seems quite happy with the current state of education. It’s too rote. Too much just teaching to the test. The system fails both the least and most capable students. Teachers are increasingly stressed. Etc. And we shouldn’t look to technology to solve all of this. I get that. However . . . two recent announcements surely signal the beginning of the end of education as we have known it.

First, there was the announcement of Knewton’s system for adapting digital textbooks and materials to the student based on continuous assessments of progress. Then there was the Apple iBook announcement. I know, I know, there are legitimate concerns about Apple’s “walled garden” approach and potential lock-in to their brand of educational process. Nevertheless, there is no going back. These two announcements, the first by a high profile start-up in conjunction with the top text book company, and the second by the world’s top tech company, coming within just a few months of one another, usher in a new era education—one in which all of the education process will adapt to the specific needs of the student. In other words, the end of Zombie education processes!

Exactly how this will play out among the various competitors and complementors in this space is hard to predict, but the train has certainly left the station. The most basic adaptive approaches will be based on personalizing the instruction in response to explicit behavioral information such as test results. However, even more nuanced approaches are inevitable, with the adaptation and personalization being based on more subtle cues from the student, and/or from peers whom are inferred to be in some way similar to the student. And it seems obvious that very shortly Siri-type natural language-based interactive capabilities will be integrated with these learning platforms.

This will have profound implications for learning, as well as for the teaching profession and its administration. It can be expected that the technology will lead, and the necessary adaptation of formal education policies and processes will tend to lag, but inexpensive, adaptive learning tools are destined to rapidly and dramatically reduce the barriers to high quality education (just as Khan Academy videos have already begun to do). Within five years a personal tutor will be available 24×7, and what we have known as education promises to blur into a global learning layer.