The Adaptive Stack

IT always evolves by building new system layers on top of preceding layers, while concurrently abstracting away from users of these new layers extraneous details of the underlying layers. In the resulting current IT “stack,” we predominantly interact with content and application layers, and when applicable and available, process layers. And we are well on our way toward abstracting away the networking and hardware infrastructure on which our applications run—bundling that big buzzing confusion into “the cloud.”

Much more recently we have begun to add a social layer on top of our other software layers—still a work in progress in most organizations. So far, these social-based systems can more often be considered architectural bolt-ons rather than a truly integral part of the enterprise IT stack. But that is clearly destined to change.

And coming right on the heels of the social layer is the learning layer—the intelligent and adaptive integrator of the social, content, and process layers. The distinguishing characteristic of this layer is its capacity for automatic learning from the collective experiences of users and delivering the learning back to users in a variety of ways.

So this is the new IT stack that is taking shape and that summarizes the enterprise systems architecture of 2011 and beyond. And since auto-learning features promise to be an integral part of every system and device with which we interact, it is the reason that the next major era of IT is most sensibly labeled “the era of adaptation.”

As I discuss in the book, there is something qualitatively different about the combination of these last two layers of the stack—the social and learning layers—in contrast to all the layers that came before. These new layers cause the boundary between systems and people to become much more blurred—it is no longer just a command and response relationship between man and machine, but rather, a mutual learning relationship. And exactly where the learning of the system and the learning of people begins and ends is a bit fuzzy.

Perhaps then, our new stack more accurately summarizes the next generation enterprise architecture, not just the IT architecture–an enterprise architecture of a different nature than that which has come before, one in which learning and adaptation is woven throughout.

Watson: Will Zombies Inherit the Earth?

Last week we witnessed a modern-day St. Valentine’s Day massacre when yet again a computer entered a hallowed arena of human intellectual combat and trounced the best that the human race had to offer. And Watson’s Jeopardy victory was a seriously impressive feat of engineering, with many more practical business applications than the last widely publicized machine-on-man intellectual violence perpetrated by IBM, Deep Blue beating chess champion Gary Kasparov.

So give Watson its due. Still, Watson is a zombie. In The Learning Layer I describe zombie systems as those that are unable to pay attention to, and learn from, human behaviors and to adapt accordingly. And furthermore, as we all know from late night movies, zombies just do things—they don’t have the ability to articulate why they do things. Just like probably every system you have ever interacted with.

Of course, now we have the technology that could rescue Watson from the realm of the zombies by adding a capacity for social awareness, and an ability to automatically learn from this social awareness. This Watson Jr. would combine Watson Sr.’s formidable textual searching, pattern matching, and natural language processing, with social learning skills. If you work in a call center, you should already be a bit worried about Watson Sr. But Watson Jr. would be quite a formidable force in a whole lot of different job markets!

We could even go a step further and put an explanation engine into Watson Jr.  He could then explain effectively, in human terms, why, for example, he proposed one Jeopardy response versus another response. Of course, since the actual rationale would be basis all of those highly inter-related and complex pattern matching algorithms he inherited from Watson Sr., he would often find it hard to explain to us his specific reasoning. Perhaps he would humor us by just making something up that he thought we would find plausible.

Come to think of it, if we asked Ken Jennings why he guessed one response versus another response, he might have trouble articulating exactly why. Just a hunch or a feeling, perhaps. Maybe there is a little zombie in all of us . . .

The Architecture of Learning

If we want our systems to automatically learn, how should they be architected? The obvious thing to do is to take a lesson from the one “machine” that we know automatically learns, the brain. And, of course, what we find is that the brain is a connection machine; a vast network of neurons that are inter-connected at synapses. And a closer look reveals that these connections are not just binary in nature, either existing or not, but can take on a range of strength of connection. In other words, the brain can be best represented as a weighted, or “fuzzy,” network. Furthermore, it’s a dynamic fuzzy network in that the behavior of one node (i.e., neuronal “firing”) can cascade throughout the network and interact and integrate with other cascades, forming countless different patterns throughout the network. Out of these patterns (somehow!) emerges our mind, with its wondrous plasticity and ability to so effortlessly learn.

Yes, taking this lesson may seem like, well, a no-brainer, but amazingly the lesson has generally been ignored when it comes to our systems. We’re well over half a century into the information age, but looking inside our organizations, we still find hierarchical system structures predominating (e.g., good old folders). The problem with hierarchy-based structures is that they are inherently brittle. They simply don’t embody enough relationship information to effectively encode learning. They don’t even scale (remember the original Yahoo! site?)—as we have all experienced to our great frustration. There is a reason nature didn’t choose this structure to manage information!

Fortunately there has been a revolution in systems structure over the past decade—the rise of the network paradigm. The internet was, of course, the driver for this revolution, and we now find network-based structures throughout our Web 2.0 world—particularly in the form of social networks. But even these networks are not fuzzy—we are limited to establishing our relationships only in binary terms, yes or no. Sure, we can categorize with lists and groups, but we are still at a loss in representing all of the relationship nuances that range from soul mates to good friends to distant acquaintances. And that makes it difficult to apply sophisticated machine learning techniques that truly add value. What’s the percentage of “recommended for you” suggestions that you receive in your favorite social network system that actually hit the mark, for example?

But the situation is even worse with regard to our organizations’ content. In this land-that-time-forgot, our knowledge remains entombed in the non-fuzzy world of hierarchies, or at best, relational structures. Not only are these systems incapable of learning and adapting, but it is often a struggle to even find what you are looking for.

This sad state of affairs can and must be rectified. All we have to do is take our lesson from the brain, integrate our representations of people (i.e., social networks) with our content, allow the relationships to be fuzzy, and we have something that is architected a whole lot like the brain, i.e., architected for learning. That’s not sufficient—we also need clever algorithms to operate against the structure, to create the necessary dynamics and patterns that deliver the benefits of the learning back to us. But the architecture of learning is the necessary prerequisite, quite doable, and therefore quite inevitable.

The Chinese Room Explanation

My favorite recommendation explanation, a real honest-to-goodness computer-generated recommendation, is the one pictured below. I was amazed when I received it even though I had a hand in designing and developing the system that generated the recommendation—the system had truly become complex and unpredictable enough to spark a delighted surprise in its creators!

The recommended item of content in the screen grab is about AI and the explanation engine whimsically mentions as an aside that it has been pondering the Chinese Room Argument—the explanation engine’s little joke being that, as AI aficionados know, the Chinese Room Argument is philosopher John Searle’s famous thought experiment that suggests computer programs can never really understand anything in the sense that we humans do. The argument is that all computers can ever do, even in theory, is just operate by rote and that computers will therefore always be relegated to the realm of zombies, albeit increasingly clever zombies.

The Chinese room thought experiment goes like this: you are placed in a room and questions in Chinese are transmitted to you. You don’t know Chinese at all—the symbols are just so many squiggles to you. But in the room is a very big book of instructions that enables you to select and arrange a set of response squiggles for every set of question squiggles you might receive. To the Chinese-speaking person outside the room who receives your answers, you appear to fully understand Chinese. Of course, you don’t understand Chinese at all—all you are doing is executing a mind-numbing series of rules and look-ups. So, even though in theory you could pass the test posed by that other famous AI thought experiment, the Turing test, you don’t really understand Chinese. And this seems to imply that since, most fundamentally, all computers really ever do is execute look-ups and shuffle around squiggles in prescribed ways, no matter how complex this shuffling is it can never really amount to truly understanding anything.

That’s all there is to it—the Chinese Room Argument is remarkably simple. But it is infuriatingly slippery with regard to what it actually tells us, if anything. It has been endlessly debated in computer science and philosophy circles for decades now. Some have asserted that it proves computers can really never truly understand anything in the sense that we do, and certainly could not ever be conscious—because, after all, it is intuitively obvious that manipulations of mere symbols, no matter how sophisticated, are not sufficient for true understanding. The counterarguments are many. Perhaps the most popular is the “systems reply”—which posits that yes, it is true that you don’t understand Chinese, but the room as a whole, including you and the book of response instructions, understands Chinese.

All of these counterarguments result from taking Searle’s bait that the Chinese Room Argument somehow proves that there is a limit on what computer programs can do. What seems most clear is that these armchair arguments and counterarguments can’t actually prove anything. In fact, the only thing the Chinese Room really seems to assure us of is that there is no apparent limit on how smart we can make our zombie systems. That is, we can build systems that are impressively clever using brute-force programming approaches—even more clever than us, at least in specific domains. Deep Blue beating Gary Kasparov is a good example of that.

As I have implied previously, understanding, as we humans appreciate it, is a product of our own, unique-in-the-natural-world explanation engine, not our powerful but inarticulate, unconscious, underlying neural network-based zombie system. The Chinese room doesn’t directly address explanation engines, or any capacity for learning, for that matter. There is no capacity in the thought experiment for recursion, self-inception, and for the system self-modification that occurs, for example, during our sleep. These are the capabilities core to our capacity for learning, understanding, creativity, and yes, even whimsy. We will have to search beyond our Chinese room to understand the machine-based art-of-the-possible for them.

And we will surely continue to search. We know we can build arbitrarily intelligent systems; but that is not enough for us to deign attributing true understanding to them. This conceit of our own explanation engines is summed up by the old adage that you only truly understand something if you can teach it to others—that is, only if you can thoroughly explain it. Plainly, for us, true understanding is in the explaining, not just the doing.

The Explanation Engine

Recommendation engines are now becoming quite familiar parts of our online life, although we are still in the early stages of the recommendation revolution. The ability to infer your preferences and deliver back to you valuable suggestions of content, products, and even other people is the wave of the future—another manifestation of the dominant IT theme of this decade, “adaptation.”

But what intrigues me even more than the intelligent recommendations themselves is a topic I discuss at length in The Learning Layer: the ability for the system to deliver to you an explanation of why you received a recommendation. You may have already experienced some simple examples of the “people that bought this item also bought these items” variety. But in more sophisticated systems that infer preferences and interests from a great many behavioral cues, there is an opportunity to provide much more detailed and nuanced explanations. Among other subtleties, these increasingly sophisticated explanations can provide you with not just the rationale for making the recommendation, but also any caveats or reservations the system might have, as well as a sense of the system’s degree of confidence in making the recommendation. Providing such a nuanced, human-like explanation is in many ways a more daunting technical achievement than providing the recommendation itself.

Among the many things that intrigue me about explanations for recommendations is how they can shed some light on the peculiarities of how our own brains work. This was brought home to me as we worked on designing explanatory capabilities for our recommender systems. At first we thought this would be pretty trivial—we would just have the system regurgitate the recommendation engine’s rules for recommending the particular item. But here’s the rub—the rules are not always or even typically “if-then” rules of the sort that we humans are going to easily understand. More typically, rating and ranking of potential items to recommend are going to be the product of various high-powered mathematical evaluations and manipulations of vectors and matrices. How can they possibly be compactly conveyed in an explanation to the recommendation recipient? After a while we realized that they really cannot—explanations necessarily have to be an approximation of the actual thought processes of the recommendation engine. A very useful approximation, but an approximation nevertheless. We also realized that to do them right, it was an architectural necessity to have a dedicated engine for explanations complementary to, but separate from, the powerful but inarticulate recommendation engine.

Turns out we are no different—we humans have a language-based explanation engine that explains to others, as well as ourselves, why we do the things we do. And it has become increasingly apparent from psychological studies over the past decade or two that our explanations are really only approximations of our underlying, unconscious decision making. In fact, it has been confirmed by recent brain imaging studies that although we tend to believe that our conscious and logical explanation engine is making the decisions, in reality it is just providing an after-the-fact explanation for a decision already unconsciously made elsewhere in our brains. Moreover, studies involving hypnosis have revealed that our explanation engine just makes up stories as required to provide a rationale for an action if the real motivation is not accessible to the conscious explanation engine!

Now this seems like a pretty weird state of affairs. But taking some lessons from building explanatory facilities for recommender systems suggests that this seemingly strange brain behavior is really the only way it can be. After all, our brain most fundamentally is nothing more than a vast, weighted network of neurons. Decisions are assessed and made by inscrutable interactions among this network. We humans, fairly recently in our evolutionary history, have developed a language-based explanation engine that has essentially been grafted on to this underlying network that enables us to effectively communicate with other humans. To achieve a reasonable compaction, the explanations we give must necessarily typically be extreme simplifications, and quite often will also have some degree of fabrication woven in. And our explanation engine continuously explains itself to us—so that we come to believe that we are our explanation engine, that there is a clear logic to what we do, that we have an explicit freedom to decide as wish, and a variety of other explanatory conceits.

There is really no other way we could be—it’s an inevitable architectural solution. Turns out the explanation engine has a good deal to say about us!

Data as a Disrupter

This recent GigaOm blog on the increasing need for “data scientists” is just another signal that we have indeed entered a fundamentally new era of IT, what I call in The Learning Layer the Era of Adaptation—an era in which our systems routinely learn from their experiences with us, and beneficially adapt based on this learning. And for our systems to be able to adapt to our needs and preferences, they need to be effectively predictive, and to be effectively predictive they need to be able to make good inferences, and to make good inferences, they need a foundation of lots of data. 

Until recently, the bottleneck in this chain was that we simpy didn’t have access to the data required. But now vastly more data is being generated and captured than just a few years ago. Online click streams, postings, and direct feedback (e.g., the ubiquitous “like” button) are just the tip of the data iceberg—use of mobile devices and positional tracking promises to generate another order or two more of additional raw data. In the not too distant future, we’ll likely add to this corpus of available data things like physiological information (yeah, there’ll be some privacy issues to work out . . .). 

Once we have the data, that’s where the data science comes in—various data conditioning techniques need to be applied, and then importantly, the right machine learning algorithms need to be brought to bear that will wring the most predictive power from the available data, taking care to neither under-interpret or over-interpret. So “Data+Algorithms” is clearly at the core of the competition in the Era of Adaptation. And it’s not just germane to commercial IT companies–as I wrote a few years ago, Data+Algorithms is increasingly at the heart of competitive success in a wide variety of industries. Data, and making meaning from data, can be a truly disruptive force–excellent news for data scientists or those who are interested in becoming one!

The Recommendation Revolution Accelerates

GigaOm recently published a blog by Mathew Ingram about Gifts.com’s application of Hunch’s recommendation function. Hunch uses your Facebook social graph as a basis for its recommendations. What was really eye-opening is that Gifts.com’s conversion rate of shoppers to buyers increased by up to 60% after implementing recommendations! That is huge, particularly given that Gifts.com was already a highly sophisticated e-commerce site. And it is consistent with the uplift that Amazon.com and other e-commerce sites that have implemented recommendations have experienced.

As I argue in The Learning Layer, as the positive results of recommendation engines applied in e-commerce keep rolling in, it is inevitable that recommendation capabilities will eventually become a ubiquitous e-commerce feature. And, of course, the same is true for applications within the enterprise—it’s just that the recommendations will be of people, content, and apps rather than that high-tech electric razor!

The Public Learning Layer Emerges

John Battelle recently posted a great blog essay in which he distinguished between the “Dependent Web” and the “Independent Web”—the Dependent Web being the part of web that basically has some degree of understanding of who you are and then tailors its responses to you, while the Independent Web is basically the dumb old legacy web that really is not aware of “you” and just operationally treats you like everyone else. 

John makes the point that even the Independent Web is being dragged, kicking and screaming, into what I call the new IT “era of adaptation” by virtue of the advertising delivered on these sites being personalized basis inferences of preferences and interests, even if the substantive, non-advertising functions of these sites are still mired in the legacy of the era of dumb old non-personalized systems. 

John’s take on this inexorable evolution toward a web that learns from its experiences with us is that, “it’s clear that we’re in the early phases of a major shift in the texture and the experience of the web.” Indeed, we are witnessing the plain old web transforming into various learning layers. And as I indicate in The Learning Layer, just how the web will eventually evolve into the one learning layer to rule them all is the real interesting question, and John goes into a good deal of detail on ways that this could/should occur. 

It clearly won’t play out like the original web in which a relatively simple standard was rapidly and extensively leveraged—the evolution of the learning layer will be a function of many different economic interests and objectives, as well as complex technical approaches, almost surely resulting in more of a learning layer mosaic, presenting users with a great variety of different learning textures. Which is likely ultimately a good thing. Of course, as I stress in The Learning Layer, the implementation of company or institution-specific learning layers is much more straightforward and will deliver numerous benefits all the while the long and fascinating process of the “sausage making” of the public web-based learning layer continues.

The Learning Layer: Adaptive Middleware for the Cloud

Some of us from ManyWorlds recently attended the Rackspace Software as a Service (SaaS) Summit. ManyWorlds is a long-time Rackspace customer—we are fortunate to have chosen Rackspace to power our Epiture^® learning layer platform back in Rackspace’s early days, well before they became the leader in computing infrastructure and cloud computing services that they are today.   

We were particularly interested in the SaaS applications part of the conference, and it was amazing to see how the number of highly valuable enterprise applications being served from the cloud has absolutely exploded over the past few years. It is clear that delivery of applications from the cloud is inevitably becoming the standard model, even for the largest enterprises—and Rackspace is orienting its entire business around this reality. 

As a still-recovering big company CIO who spent a lot of time worrying about such things, however, perhaps the most intriguing topic at the conference for me was with regard to integration—how applications served from the cloud can be effectively integrated with existing enterprise applications. Most cloud apps have a relatively narrow scope—they do one thing, and they do it very well. But they need to fit within the context of an existing application environment, which is often the province of broad-scope applications such as enterprise resource planning (ERP) systems (e.g., SAP) and customer relationship management (CRM) systems. As the number of cloud-based applications grows, this integration issue becomes front and center for the CIO. Legacy applications generally cannot simply be ripped out and replaced—so integration approaches, whether ad hoc, or better, through standardized APIs, promises to increasingly dominate the CIO’s agenda. 

The Summit served to reinforce for me that the good news is that the learning layer can play a strong role in ameliorating this integration problem. The “ethereal network of learning” that I discuss in the book can be applied to integrate together myriad cloud applications with complex legacy application environments. Think of it as an “adaptive middleware” layer sitting between the cloud-based applications and the legacy applications, providing a single, coherent environment that includes a capability for integrative workflow, and that automatically adapts and personalizes based on usage.

When we first conceived of the learning layer, we primarily thought of it as providing an integrated layer of learning over existing applications and content—it is now clear that, perhaps even more importantly, it will serve as an adaptive bridge between the new and the old of the application world.

Self-Inception

I recently saw the cool new movie Inception—where the term “inception” means the implanting of an idea into the brain of a target by way of hacking into the target’s dreams. And as those of you who have already seen the movie know, the plot plays with this idea in a recursive way—the dream hacking is conducted in dreams within dreams, making for a mind bending movie experience, and, of course, sufficient ambiguity between dreaming and reality to allow for many Hollywood sequel directions . . .

Watching the movie was a particularly enthralling experience for me because two key themes flowing through The Learning Layer are dreams and recursion (and, ok, because I’m a bit of geek I suppose). Yeah, The Learning Layer is most fundamentally a book about next generation organizational learning, but it’s also book of many layers, and the undercurrents of dreams and recursion are never far away.

Why the undercurrent of dreams? Well, it has become increasingly clear that the purpose of dreaming is that it is a process for rewiring the network of the brain. The strengthening and weakening of connections in a network is the essence of learning, and dreaming, particularly during the rapid eye movement (REM) stage of sleep, appears to be when meaning is made of the raw information we have taken in during the day through a process of appropriately editing the wiring in our brains. Which may be why, not just humans, nor even just mammals, but every organism with a brain that has been closely examined seems to need sleep. So in The Learning Layer I make the case that if we want our systems to learn effectively they better have the capacity for the equivalent of dreaming.

Why the undercurrent of recursion? Because it’s what invariably lies behind new, unpredictable, emergent phenomena. Recursion is the idea of a feedback loop operating on basic units (e.g., network subsets), and it can give rise to something of a different nature than the individual units on which it operates. Not surprisingly, it’s a fundamental property of the brain, a property that leads to that peculiar little phenomenon emerging from our network of neurons, our mind. And so if we want truly emergent qualities to spring forth from our systems, recursion better be at their core.

In other words, just like our brains, we need our systems to be architected to embrace recursion, and also to dream! And what makes our mind even more powerful and yet so wonderfully unpredictable is the capacity for self-inception. That is, myriad feedback loops can be brought to bear in the vast network that is our brain whereby one part of the brain can modify another part of the brain. This can happen unconsciously (e.g., during our sleep) and/or consciously (i.e., self-inception). In fact, as in the movie, the boundaries between dreaming and inception can become fuzzy, and I relate in the book a simple experiment you can perform on yourself to illustrate this:

You can easily see that feedback dynamic at work whenever you awaken from a dream. If you don’t try to remember the dream, you will almost always forget it. But if you will yourself to recall more details of it, and then remember it, you can make the memory of the dream consolidate, potentially forever. When you do so, one part of your brain literally physically affects another part of your brain, which in turn may later influence other parts of your brain, and so on, for the rest of your life. Your mind will never by quite the same because of that little whim to remember that particular dream!

It’s really quite amazing if you think about it (yep, another opportunity for self-inception!). But maybe even more amazing is that we can actually architect our systems to do the same thing. That’s the essence of the learning layer concept—we provide our systems with the ability to recursively modify themselves based on their experiences with us, which amounts to modifying the connections among the representations of us and our content. And that, of course, is the analog of dreaming. Or maybe it’s not just dreaming, maybe it could be considered self-inception. I suppose that’s just a question of whether that system self-modification is performed consciously or unconsciously . . .