I've recently finished reading "How Emotions are Made" by Lisa Feldman Barrett. It's a terrific book, one that presents a revolutionary theory behind how emotions work. While the author does her best in explaining just complex scientific discoveries, I still struggled a bit to put all the parts together. I thought it might help if I try to describe it using my own words.
Being a programmer, I found that I could explain the processes of the brain by trying to compare them to computer programs. I try my best to avoid computer jargon and to write in a way that non-programmers can understand, too. I don't delve into artificial intelligence or machine learning, although it is obvious that the domain has taken a lot of inspiration from how the brain works.
My explanation is probably wrong in many ways. I don't know the underlying neurological mechanisms. It's just a way for me to try and understand the book a little better, so take this with a grain of salt, and read the book if you want to learn more. It's worth it!
The article is quite long (a bit over 10 000 words), so I'll start with a brief description of the entire process and then go into the details.
How emotions are made - the condensed version
The brain needs to analyse its surroundings. Different receptors transform exterior input into signals the brain can process. This input consists of the five senses, but also the sensations coming from inside your body. These are just signals which go through an incredibly complex hierarchy of neurons. Patterns are detected by groups of neurons that activate together. Each level detects patterns in the lower levels. In other words, higher levels summarize information of the input into a concept. More and more complex concepts are thus assigned to the input. The lower levels detect things like shapes and objects, or syllables and words, while higher levels deal with more elaborate concepts, like categorizing animals into species, or purely mental concepts such as recognizing a person as being your relative, and, of course, emotion concepts.
This process is run in reverse at the same time. Complex concepts which are statistically probable to match the input are unpacked into lower-level ones. These are predictions that the brain makes regarding the input. There are billions of such predictions done at the same time, which are stitched together to match the whole picture. Each prediction is compared to the actual input and in some cases it doesn't match. The brain can either adjust its prediction to match the input or disregard the input and keep its prediction.
After this adjustment, the predictions that match the input the most win, thus becoming your perception of reality. This includes what you see, hear, feel, taste, smell, but also what energy withdrawals your body makes, what sensations these cause you to feel.
The brain's goal is to manage your body's energy budget. Based on how the energy budget is affected after its decision, it learns and adjusts its internal connections so that it makes more suitable predictions.
Emotions are high-level concepts, the patterns of which have been shaped by what your brain has learned regarding the emotion: all the societal norms attributed with it that have been established across generations in the society that you lived in, and what the emotion means to you personally, as experienced throughout your life. If the winning predictions contain such patterns, then you experience the emotion. This is more than what you feel in the body and how the body responds, it's also about what it means to you, what you should do about it.
In other words, emotions are learned responses the brain activates in order to keep your body's energy budget in balance, as it constantly models the exterior world and internal sensations, by recognizing patterns in the signals received from the body's receptors.
Let's dive deeper by starting with the last part of that sentence.
Sensing
The brain requires to understand its surroundings in order to ensure survival and procreation. It does this using the five senses. Of these, vision is the easiest to reason about, so I'm going to focus on it, but the explanations should apply to the other senses as well.
How does vision work? Your eyes contain receptors that respond to light, which is actually made up of photons of different energy levels or wave lengths. These receptors, upon stimulation from photons, emit signals which travel throughout your brain. That is what your brain needs to interpret to make sense of what you actually see.
The process is similar to a program that interprets an image or a video, like for example the program that recognizes your face in order to unlock your electronic device. Receptors in a camera are just like the receptors in your eyes. They transform light into electronic signals, which are interpreted as ones or zeros. That is what the program understands at the lowest level. At a bit higher level, a program can deal with numeric values, but still, the input from the camera is just a multitude of rows of pixels, each having a certain value for red, green, and blue.
The brain executes a similar process, but instead of pixels, which are digital, it deals with analog values, such as the wave lengths of photons. It's the same for hearing, where your eardrums transform vibrations of the air into signals to your brain, and something similar for the other senses.
There is no additional information other than the raw input available. It is the brain's job, just like the program's, to analyse this input and detect shapes, eyes, faces etc. To detect a face, it has to detect the eyes, nose, mouth first. To detect an eye, it has to detect its oval shape which contains a circle for the iris, which contains another circle for the pupil.
Before talking about how the brain does this, we need to talk about storage first.
Storage
Think about the program that recognizes your face. To do that, it must first know what your face looks like, so before using the program you must provide it with an image of your face, which it will then use to compare with what it sees whenever you want to unlock the device and you look at the camera.
Let's consider another program, one which, when presented a picture, it will say whether it is a picture of a cat. How would it work? Well, first it would need to be "trained". It will be given a list of pictures of cats. Then, when someone uses the program, it will compare the picture it receives with the pictures it knows about. If any of them is very similar to it, then the conclusion is that it is a picture of a cat. The more pictures it was given while "training", the better the results. Also, the list needs to contain a wide variety of cats. For example, you need to have an orange cat in there, otherwise the program will simply not consider an orange cat as a cat.
In addition, it's best to have a list of pictures of animals that are not cats. For example, if you only have a list of cats, then a tiger might be very similar to one of those pictures, and the program will report it as being a cat. But if you also have a picture of a tiger and it is in the non-cat list, then that picture will match more than one from the list of cats, and the program will answer correctly. In fact, if you don't just label the pictures as cat and non-cats, but instead label each animal as such, then the program can very easily recognize all sorts of animals, not just cats.
To sum up, the process is quite simple: The program has several pictures of animals, each labelled with the name of the animal. When it receives a new image as input, it compares it with the pictures it knows about, and chooses the one which is most similar. Then the answer is the name of the animal that picture was labelled with.
In fact, that list doesn't have to be entirely predefined. The program can start with an initial list of pictures, and it will try its best to answer based on that. If it gets something it doesn't know about, like a tiger, it will say it's a cat, but the user can then correct the program, and label the picture as that of a tiger. The program will now store this picture and the next time it sees a picture of a tiger it has a better chance of recognizing it. If it fails again, it will add the second picture to the list of tigers as well, and it will get better at recognizing them, simply because it will have more examples to compare with.
The brain works in an analogous way. When it sees or hears or senses something, it will compare it with instances of that thing it has seen before. That's how it recognizes shapes, animals, faces, words, smells. Of course, when you meet someone new, you don't recognize them, but the brain labels the image of their faces with the person's name (once they introduce themselves). The next time you see them you might still not recognize them at first, but the more you see them, the easier it will be.
The essence here is that brain cannot just make sense of a bunch of sensory data using that data alone. To understand, it must compare to things it has seen in the past. In order to understand that it sees a cat, it must have seen a cat before. And, as we will see, in order to feel happiness, it must have learned what happiness is, beforehand. In order to feel afraid when you see a twig in the forest that looks similar to a snake, you must have had some previous experiences that lead you to feel this way in that particular moment.
And for this, the brain must store these experiences. Not necessarily as memories, because you don't have to be conscious about them. You can think of these as multi-sensory images, which encompass all of the senses, not just vision.
And of course, your brain's storage is limited, just as your computer's storage is limited. It cannot remember everything, so it has to make some decisions. One such decision is to not store a piece of information that doesn't add any value. For example, if our program receives a new image that matches very well with one it knows about, it makes no sense to keep it as it will not improve its future performance. Pictures that it labels incorrectly are very probably good to keep onto. Learn from your failures, right? But also when it labels correctly and the match was not so good, it might be useful to keep the new image. For example, if the program has no images of orange cats and receives one, it will be a good idea to remember it, even if it did label it correctly.
These are just simple examples, but you can imagine it's much more complex. Even by using these simple rules, the lists could just grow and grow and consume all of the storage space. The program could then perhaps clean-up the list. For example, it could remove pictures that haven't matched in quite a while - maybe those cats went extinct, or they simply aren't as common. It's better to have better accuracy of things you are more likely to encounter than to keep a trail of data around which is only useful once in a blue moon.
The list of images would be constantly updating and shifting. And you can probably make analogy with yourself. You forget people you haven't seen in a while, and your thoughts and behaviours change ever so slightly, but over a long time they accumulate, and you feel like a different man than you were several years ago.
This all sounds easy, but how does one compare two images and decide how similar they are?
Finding patterns
The most important skill of the brain is pattern matching, meaning it can notice patterns that appear at various times and in different contexts. Let's say you've never ever seen an animal before, and you are given a series of pictures of cats. Pretty soon, you brain notices the similarities: fur, four legs, tail, whiskers, etc. without being given any other information of what it sees. That is pattern matching: noticing things in the current input that are similar to what it has seen in the past.
A program that looks at a face and tries to decide whether it is your face does something similar. When you initially trained it with a picture of yourself, it calculated some features, such as the length between your eyes, the position of your nose and so on. It then calculates the same features on the image it is seeing now and if they are very close to the initial photo, then it is a match.
But how does the program and the brain know what features to look at? A program can, of course, be programmed that way. And it makes sense for a face recognition program, where there is a clear, short list of features.
But the brain can't work like that because the process doesn't scale. Faces aren't the only things the brain needs to recognize. There are cats, dogs, and loads of other animals, there are trees, plants, cars, furniture, and even more abstract things like words, sounds and so on. Each of these categories have different features that are important when recognizing them and also when distinguishing them from other things. If the brain were somehow programmed to recognize them, then it would need a different "program" for each thing. And the more you learn, the more programs it would need. But, at least after a certain age, the brain is what it is. It doesn't grow. It can't just add more and more information.
Even our simple program that recognizes animals would have difficulties. We said that by simply labelling each picture with the name of the animal we can make the program recognize all of the animals. But if we programmed it to calculate some features of cats, then it won't be able to match pictures of horses, because those would have different features. Instead, it needs a mechanism that is the same for all animals. And the brain needs a mechanism that can recognize things in general. Recognizing cats shouldn't be different from recognizing dogs, or from recognizing words.
Neuroscientist Carla Shatz distilled how this process works into a simple idiom: "Neurons that fire together, wire together". When the brain receives some input, signals travel throughout the brain from neuron to neuron. Each time a neuron sends a signal to one of its neighbours (they fire together), the connection between them strengthens (they wire together).
Let's say that you look at a rectangle for the first time in your life. It is a tall red rectangle. Through the magic of how the neurons communicate together, some of them will activate upon receiving input from the receptors in your eyes. Some will activate together because of the colour red, others will activate together because you see lines, others will activate because some lines are parallel, others will activate because some lines are perpendicular, others will activate because some lines are short, and others are long, and so on.
The brain itself has no idea why each specific neuron or why a group of neurons activated. But as it sees more rectangles something interesting happens. Let's say the next one is blue and shorter. Most of the neurons that activated previously will activate this time as well, except for the ones that activate for the colour red and those that concern the length of the lines.
As you see more and more rectangles, some neurons will activate always, while others only sometimes. The connections between the ones that activate always will become stronger amongst themselves than the connections to the other neurons. In essence, those neurons recognize rectangles.
The important thing to note is that nobody instructed the brain what a rectangle is, it just noticed that the same pattern forms again and again. That is why this mechanism is a generic mechanism for recognizing anything. It's as if the program for recognizing a certain thing is built into the connections of the neurons. Not only that, but the programs change over time as these pathways change with new experiences. The programs are being created as you learn.
Note that this does not mean that these sets of neurons deal only with recognizing rectangles. Each individual neuron is free to be part of other patterns. For example, the ones that detect lines can be part of a group of neurons that detect triangles when activated together.
The process doesn't stop here. It happens on a vast scale among a hierarchy of billions of neurons. The lowest levels detect patterns in the raw input, like lines and shapes. Each level in the hierarchy detects patterns in the lower level. For example, think of a simple hand-drawn house, which is mostly made up of rectangles and triangles. The lower level deals with detecting these shapes, while the higher level detects the pattern in the arrangement of these shapes and recognizes a house, no matter where the windows are placed, how many they are, or what size the door is.
And, of course, let's remember that it is not the visual input that is analysed, but all of the senses together.
There are many ways you can think of this process. One way is that the brain assigns meaning to the input it receives. In fact, that's what we said its job was in the first place, wasn't it? Let's dig into this a bit deeper.
Concepts
Whenever something is recognized in the input, it's like the brain has just assigned a label to what it recognized. These four things are labelled as lines. And these four lines together are labelled as a rectangle. And then a house, and so on.
These labels are concepts. Concepts are how we categorize the world. They can be concrete, like categorizing animals into species, a house into the parts it is made of. They can be more abstract, like the names of various colours. Or they could be purely mental concepts, like nationalities, or money, or emotions.
What the brain does is assign concepts to things it senses, just like our animal recognition program assigns the concept of a specific animal species to a picture it receives as input. The pattern matching network is responsible for detecting these concepts. And it can detect concepts based on other concepts, like the concept of a house which is made from other concepts such as rectangles and triangles, which themselves are made up of concepts like lines and angles.
While some concepts can be defined quite precisely (e.g. defining what a rectangle is), most cannot. Even trying to define species of animals is quite hard. But more importantly, there can't be universal definitions of concepts, because we can make up whatever concepts we want. Just think of any topic you want to categorize: music genres, hair styles, types of food. You can invent a category for anything you want, like sounds that are pleasant, quotes you find interesting, things that are good on pizza, and on and on.
Categorizing something depends on the purpose. For example, if I look at my desk, I can distinguish a book from the other objects on the desk, like the keyboard, mouse, monitor etc. All these are different concepts. I can also distinguish this particular book from the rest of the books on my bookshelf, based on its size, cover, but also by its title. All of them I would put under the label "book", but each is its own separate concept, too. If a robber came into my house, I would be looking for things to throw at them. In other words, I would categorize the objects around me and label them as "unsuitable for throwing", "moderately suitable for throwing" and "most suitable for throwing". The book would fall in one of the latter two. If it's a large and heavy book and there is nothing heavier around it would be "most suitable", at least in that situation. If other objects are around, it might fall into the "moderately suitable" category.
So, the brain doesn't just rely on its neural connection, but also on its intention. Part of it, called the control network, involves a sort of implicit decision making. It favours certain connections among neurons to others. For example, if you are in the robber situation above, it will favour the connections among the neurons that detect things to throw when they see a book. But in another situation, they won't activate when seeing a book.
All of this is very abstract, and it is just our own way of trying to make sense of the brain's process. In reality, the brain has no idea what a concept is, it is not labelling anything. All it does is strengthen the connections of some neurons guided by the control network. The difference between our program and the brain is that the program knows about the labels beforehand, while the brain, in an abstract way, creates the labels itself, on the fly, without even knowing what the label is. To put it another way, the concept is the maximal set of neurons that always activate together across various inputs. A concept does not have a concrete definition, instead it is defined by all the instances in which it appears, as the thing that is always the same in all of them.
Another way we can look at the mechanism of pattern recognition is through the prism of storage again.
Storage revisited
We had a pretty naïve approach to how an animal-recognition program would work. Based on what I said about pattern matching, we can see that brain doesn't actually store previous images. The brain didn't have to store any images of rectangles. Instead, the information from those images is captured in the neural connections. They are not the actual image, but patterns that are enough to detect when a similar one occurs.
When you see something new, it's not that the image is stored, it's that new neuron connections are formed. Information that is very similar to what you already know has no impact, as the connections are already strong. The most it does is to keep that connection from fading and thus forgetting that particular past experience.
Programs like our animal-recognition program actually work in a similar way. In fact, the way these machine learning programs work has been inspired by how the brain works. Our program doesn't have to keep those images. Instead, with each answer, it modifies its internals so that the next time it receives a similar image, the computation will be correct.
We said that higher-level neurons detect patterns in the lower levels. Storage-wise this means that it is summarizing the information of the lower levels. If you think about a concept like a house, we said that the details of that house, like its height, number of windows, colour, are effectively thrown away, because they are not essential to what defines a house. A concept does not have a definition, but rather it is a summary of all its instances, just like we derived what a rectangle is by looking at multiple examples of one. This summary is encoded in this higher level of neurons, which leads to a very efficient storage, because the brain doesn't have to store every instance of previous houses. Instead, it stores the summary of what a house is.
Again, this is an abstract explanation of the mechanism, in actuality these multi-sensory summaries are embedded in how the neurons activate.
Efficiency
One of the important qualities of the brain's processes is efficiency. Until now we've only talked about pictures - a snapshot in time. But your brain never has to deal with just a static image. It constantly receives input, and it has to interpret it. Think of a program that interprets not an image, but a video, for example the software that highlights the puck in a hockey game.
The program itself is quite simple conceptually. In the digital world, a video is basically a series of static images called frames - on the order of tens per seconds. It takes the first image of the video, and it has to find where in the image there is a small black blot representing the puck and it just draws a colourful halo around it. Then it should do the same for the next image, and the next one, tens of times each second. That should be no problem for a computer, but it's terribly inefficient when you have to analyse what you see while walking, looking out for obstacles, unexpected events and all the things a brain does.
The brain simply cannot afford to analyse its surroundings completely every single moment. Instead, what it does, is to just look at the differences. If you think about our program again, after the first image, it knows where the puck is. In the next image, it can't have moved much, it should be in the vicinity of where it was before. So, the program doesn't have to analyse all the pixels of the new image, it can look just in the small area surrounding where the puck was a tenth of a second ago - that's like 1% of the image - a hundred-fold increase in efficiency!
In fact, in can do even better. Instead of just analysing a new image as if it were a totally new one, unrelated to the previous one, it can use the recent history of images to know which direction the puck is heading. So, instead of looking all around the puck in the previous image, it can safely just look along the direction of movement. Even better, it knows the speed and can calculate pretty accurately where the puck will be in the next frame.
That is sort of how the brain works. Instead of analysing its surroundings completely each time, it just looks at the differences. In fact, it holds a model of what it knows about the world and predicts how it should change. This model is represented by the multitude of multi-sensory summaries embedded into your brain's wiring. The process for creating them is run in reverse for those that are deemed to be the most similar to the current context, effectively unpacking them. These are the predictions your brain makes, and one of them will be the most similar to the input, and it will win, thus becoming the way your brain interprets the input.
This is very important to stress out: what you see, what you sense, is not the world as it is, but as your brain expects it to be. That might sound shocking, as if you were living in a world of your own imagination, a fake reality, a dream world. But it's not quite that dramatic. It is tied to the real world, and it's not entirely made up by your brain. It's, just as in all things, very much tied to the reference point, the observer, it's in the eye of the beholder.
Another way to view this process is as if the brain selects a few concepts which are most likely to match the current situation, its intent. Such a concept is recognized when a group of neurons activates. But those neurons activate when other groups also activate, or in other words, the concept is recognized when the smaller concepts that form it are also recognized. And there are multiple possibilities. For example, when the concept of a rectangle is unpacked, there are multiple variants that can be recognized as a rectangle, just as we discussed, involving various sizes, colours. The ones that are most suitable to the situation are selected and are unpacked further. And in the end the instance that matches the input the most, is selected.
Let's take an example. You are in the case where the robber is in your house, and you are looking for things to throw. Your brain already has the concept of throwable things from your past experience. As you scan around the room, this concept is unpacked. It being very abstract, it is unpacked into various sub-concepts like weight, size, aerodynamic shape. In the meantime, input is processed, and pattern matching is applied. You see the book, and some neurons activate, thus applying some labels to the book, what colour it is, who the author is, but also whether it is heavy or light, whether it has aerodynamic capabilities or not. The two processes meet in the middle and if the expected neurons activate, that means that the book matches the required features necessary for being labelled as being suitable for throwing at the robber.
Prediction error
You can deduce that quite often these predictions will not match reality. What happens when the sensory input the brain receives is different than what the brain predicted? The brain has two choices: it can either change its prediction to match the input or ignore the input and keep its prediction. It surely happened to you that you misrecognized an object as something else, like a twig as a snake. But when you look again, you just see the twig and wonder why you even saw a snake in the first place, since the twig looks nothing like one. That's when the brain made a prediction correction. It updated the model it had of the world. Basically, it realigned with reality.
Sometimes it doesn't take much for a prediction to be corrected, other times your brain might need serious coercion to not filter the input. Just think of any optical illusion where you had to be told very specifically what to look for in order to actually be able to see it differently. And then you couldn't unsee it!
You might think that correcting the prediction is always the right thing to do, and ignoring the error is just a defect in the brain. Yes, constantly disregarding inputs is a cause of some brain disfunctions, but it is not always the bad choice.
Think again about the hockey puck program. As we said, it always predicts where the puck should be and then looks for it there. But what if it doesn't see it there? You might say the correct or reasonable thing to do is to correct the prediction, by simply analysing the whole image again and looking for the puck. Or perhaps something not as inefficient, like just scanning the area around the predicted spot, growing the area until the puck is found.
But let's say the puck went behind a player. The program would search the whole image only to conclude there is no puck. Then it will do the same for the next frames, until finally the puck comes from behind the player. Kind of inefficient.
What if instead the program just ignored this fluke and continued with its prediction for the next frames? Pretty soon, after a few frames, the puck will finally be exactly where the program predicted it should be. I would say this is the reasonable action in this case since the effort of searching the whole picture would have been in vain.
It's important to realize that the program has no idea why the puck was not found. It has no concept of players or anything else, it just knows how to identify the puck. It had no idea why the puck disappeared and whether it will ever appear again. Of course, if the puck doesn't reappear after several frames (e.g. it was hit by another player while it was obstructed from view and it changed direction), the program should admit defeat and do a whole scan to find it. Otherwise, the program would be quite terrible at its job, since the puck will never be where it predicted, except by chance.
As you can see, it's quite a balance between efficiency and accuracy. Leaning too much to either is a sign of a dysfunctional brain, but otherwise it is a vital characteristic of how a brain works: it constantly predicts how the world should be and updates its predictions more or less based on the differences that the sensory mechanisms detect.
Predictions are not made against the whole "picture". The sensory "image" your brain receives as input is broken up into uncountable little pieces, each interpreted in one way or another. There are billions of predictions being made for each tiny part. Thus, even if there are a lot of errors, they most likely won't affect the bigger image significantly. For example, when you are focused on what's in front of you, you might not even notice if something changes slightly in your peripheral vision. Or when there is some background noise, but you tune it out, because you are focused on talking to the person in front of you. And that's fine, because the brain doesn't just have to follow a single object like a hockey puck, it has to pay attention to everything.
Inner sensations
Up until this point we've only considered external inputs, the five senses. But everything also applies to input coming from within the body itself. You could say that, to the brain, which is locked up in the skull, everything is external, even if it is inside your body. It needs to monitor all organs and keep them functioning. The part of the brain that deals with this, also has a body-budgeting function, which predicts the energy expenditure of your body and releases chemicals and hormones when it predicts that the body needs it. For example, when you need to run from a dangerous animal, but also when you are playing sports or just moving.
You aren't aware of all these internal inputs that the brain keeps track of, like the levels of glucose in your blood or how much bile your liver produces. But some you are conscious of, albeit indirectly, in the form of affect. Affect is how you feel at a particular moment. It can be pleasant, unpleasant, or neutral, and it can range from calmness to agitation. This is similar to how you interpret light or sound. Light itself is not objectively bright, and a sound is not inherently loud. Brightness and loudness are manifestations of your consciousness, how your brain interprets the light coming into your eyes, or the sound waves received by your ears. Affect is something similar, applied to the sensations coming from the inner body. Seeing a powerful light feels bright to you, just as you experience hunger if your body needs an energy supplement.
These sensations complete the picture of the world that the brain is constantly modelling. They, too, are part of the multi-sensory summaries. This means that when a prediction is made, it predicts also whether your body requires an energy withdrawal and the sensations that will cause you. Your exterior senses are intertwined with your interior sensations. In the brain's representation of reality, these are not separate, they are a single model, just like in physics spatial dimensions are bound with the time dimension and represented as space-time. In simpler words, what you see or hear is affected by how your body feels. This is pointed out by a famous case study that discovered that judges were more inclined to judge unfavourably around noon when they became hungry.
We haven't really talked about emotions yet, but more about how the brain deals with input. Affect might seem as being emotion, but it's not. When you feel hungry, you can experience different emotions: you could be sad, frustrated, angry, or even afraid. You could even not feel any discernible emotion, just the sensation.
The traditional view of emotions, and the most intuitive one, is that your brain receives input, analyses it, and then triggers an emotion based on certain rules. But that's not how it works. Just like there can't be different "programs" for recognizing different things, there can't be different "programs" for triggering emotions. Instead, emotions are created by the brain based on the mechanisms we've discussed.
Emotions
Before putting it all together, let's just define emotions. An emotion is a concept of the highest level. It represents an extraordinarily complex pattern recognized by your brain among various constituent concepts, which are activated together. When you talk about "happiness" or "sadness" or "anger" it implies not just how you are feeling, but the context. There are different contexts in which you feel sad or happy, but somehow they all describe the same thing.
That somehow is not ingrained to your brain, but rather in culture. It is of a human creation. Humans have the super power of sharing concepts through the means of language - be it spoken, written, drawn, sign language, Braille etc. As we saw, concepts are something very abstract that describe how the brain detects patterns at a high level. They don't have names, best one can do is look at brain scans and see what neurons activate. Humans, however, can label those very complex patterns through words.
Babies can quite quickly learn to distinguish animals. They know instinctively that dogs are dogs, which are different from cats, before they can even speak. They don't even need to know the labels; their brains just notice the visible similarities among the species. You can describe a dog to a blind person, and they would understand. Sure, they can't visualize one, but eliminating the visual aspect, they could recognize dogs just as somebody else would. And they've only learned the concept through communication.
Even more fascinating is that humans can label and communicate abstract concepts. Nationalities, for example, are just mental creations. You cannot point to different people in order to teach the concept of a nationality to someone. You can't do it through exemplification, only through words.
Emotions are mental concepts. And as we've seen, when the brain recognizes a concept, it doesn't have a definition of it. Instead, it has multiple instances of it. Thus, I will not try to give a definition. For as many people there are, there are probably just as many definitions of happiness. Instead, the "definition" of happiness is the collective instances of happiness that a society shares.
Just as our program could recognize a cat by first having a list of many instances of cats, so does our brain recognize an emotion.
From the moment we are born we start learning what different emotions mean, from our parents making faces to us as babies, to children's books "helpfully" showing frowning and smiling faces with the emotion labelled underneath, from movies, literature, church, school, and wherever else we interact with society.
And they are ever so slightly different for each society. Americans feel happy a bit different than Brits. The first prefer to deal with extremes, finding things either exciting or not, while the latter prefer more nuance. There are many examples, but one would be that there are different words for "anger" in different languages. English has just one, while Russian has two, German has three and Mandarin five. Each one comes with its own set of feelings and social norms. If you are Russian, you feel one way when you are angry at some other person, but you feel another way when you are angry at something more abstract, such as the political situation.
These are concepts passed down through generations, so if they feel like they are set in stone, that's why, because they've been stripped down to the essences that are resilient with the passage of time, quite the same as gene evolution.
Of course, emotions are not set in stone, and they can change with every generation's own influence. And actually, with every individual's own experience. We take these concepts that we've learned through countless different experiences and apply them to the different contexts we face ourselves: a high-school break-up, being promoted at work, having an annoying neighbour, and all the variations possible.
We haven't necessarily encountered examples of how we should feel in exactly these situations, but the brain can piece together smaller patterns that it identifies in the current situation from many instances of other past experiences and creates a big picture that it applies. The culture we've been raised in, the experiences we've been through, the education we've been exposed to, the habits we've developed, the energy balance of our bodies, all have a say in how we feel in that moment.
All throughout your life, your brain, mostly without your awareness, has been working hard, looking for high level patterns in all your experiences. Each one either created new neuron connections or strengthened existing ones. These connections light up based on your current situation and some of them are linked with affective responses, making you feel pleasant or unpleasant. These neural pathways have been strengthened over time. When a prediction made by such a pathway is matched with the current input (accounting for prediction error), that's when the brain, in essence, applies a label to what you experience, and that label is an emotion: sadness, happiness, anger, or even more subtle ones such as frustration, schadenfreude, etc.
If we're not careful, our brains end up applying emotional concepts where they are not useful. We panic when taking an exam for which we have thoroughly prepared for, or feel a sense of loss after accomplishing something we've worked very hard for.
This is all due to our brains being able to find similarities on a vast scale, from the tiniest level to making correlations between vastly different concepts and combining these past experiences together to achieve its goal for the moment.
But what is its goal? Why should we feel happy or sad or angry? In a limited way you guide your brain towards a goal. Just like the animal recognition program can learn if the user corrects its answer, so does the brain. However, most of the learning is not conscious, it is not directly influenced by us. Rather, the body budget plays a very important part in moulding the brain's predictions. The brain needs to decide whether to withdraw some energy which will be used by the body.
Let's take anger as an example. You have been very stressed lately and you are trying to solve something which requires you to deal with a civil servant, but he informs you that he cannot solve your issue right now because of some bureaucratic issue. It was very important for you to solve this thing urgently because you are about to leave on your holiday the next day. Suddenly, you feel angry. Why is that? What is the use of this emotion?
Well, let's first think what would happen if you couldn't feel any emotion. You would just feel some internal disconfort because your body is tired from the stress you've had to deal with lately, and your energy levels are low, your body's ability to self-regulate are a bit out of whack. Now comes this information that you have something to solve, but you can't, or you have to give up your holiday to solve it, which means your energy will go even lower.
Essentially, you are very much trapped. Sure, it's not a physical entrapment like what an animal would be put up with, it's a little more subtle. It's a pattern that only your evolved brain would be able to detect, the similarity of stress-induced entrapment when compared to physical entrapment. It's actually astonishing to think how different the two situations are, but the essential pattern is recognized by your brain.
And what does the body need to do when it feels entrapped? It needs to run, of course. So your brain releases hormones preparing your body to run. This is part of unpacking the concept of "entrapment", which is not just about the exterior situation, but it also involves this bodily response, and also the affect that you feel.
But now, at an even higher level, your brain notices an even more intricate pattern. You are trapped, you are energized by the hormones that were just released in your body. And through your past experiences you've learned about the concept of anger. The multi-sensory summary held by your brain regarding anger contains information that it eliminates internal pressure and brings the body's energy levels back to a more suitable equilibrium. (Why does it have that information? Well, because in most of the past experiences of anger, that's how your body responded, that's how your experiences have shaped the connections between neurons.)
Thus, your brain chooses to match the concept of anger with your current situation, which implies the reactions associated with it. It could have matched other emotions as well, such as anxiety, hate, or maybe something totally different. It all depends on how your brain got to be wired up.
Why didn't you just run? It never even occurred to you. Because there is no concept of running while confronting this situation that your brain knows about. It's just like when you're looking for things to throw, you wouldn't even notice the photograph right next to a heavy book. The society you've lived in has taught you that running away is not a suitable response in this situation. Being angry, however, is, although it does have some consequences involved.
Just like brightness is the brain's way of telling you the light that is coming through your eyes is potentially dangerous, anger (or any other emotion) is its way of very efficiently describing to you the situation you are in. It's a shorthand. Rather than telling you: "Hey, your energy levels are low, and this guy just informed you that they're going to become even lower, so here's a hormone boost, figure out how to use it. You could run, but that will put you in an embarrassing situation which will further affect your body. Perhaps you might feel better if you shout or maybe even punch this guy." It will just tell you: "You're angry".
To sum up, an emotion is like a shortcut for describing your current situation, giving a meaning to your internal sensations, regulating your body's energy, and prescribing suitable reactions, all of which has been learned through past experiences and shared knowledge with the rest of society.
Putting it all together
We can now connect all these pieces and go through an example from start to finish. Let's say you have been walking for the whole day and are now in a forest and you see a twig, which has a similar shape and colour to a snake.
The brain constantly sifts through the input it receives building up a model of the world, which includes information coming from the organs inside your body. This is done through its vast neural network, as neurons activate together and identify different patterns: the shapes and colours of the trees, of the leaves on the ground, of the twig, sounds around you, like the wind rustling the leaves or the brushing of grass, the smell of the leaves and plants, of dirt, increased heart rate, muscle fatigue, deep, but not necessarily fast, breathing, slight hunger.
It constantly applies labels to your reality. As you walk, it already has a model of your surroundings, so it doesn't need to identify the mentioned patterns all the time. Instead, it makes predictions for what changes, its goal being to keep you safe, to avoid obstacles, other dangers, and trying to prevent you from draining all of your body's resources.
As the light waves reflected of the twig enter your eyes and are transformed to neural signals, your brain already tries to predict what it sees. Let's keep it simple and say that either a twig or a snake are the most reasonable options. Those are the two predictions your brain makes, and it has to choose the one it considers the most appropriate for the situation.
The decision will be made based on the model it already has, based on the neural pathways that are already active. This model is built up of the tiny pieces that I've mentioned. These bits have been a part of multiple experiences from your past, in different combinations and in different contexts.
The smell of dirt and leaves might have been present during an uncomfortable experience while hiking in the dark, which made you feel a bit of anxiety. You've felt tired plenty of times before, but often while playing football, or jogging, or just walking a lot.
You've seen snakes at the zoo, and you know their shape and colour. You watched them quite relaxed, perhaps amazed by these strange creatures without any legs or arms. You've also seen a documentary where a hiker was bitten by a snake, and you felt quite uneasy.
You've seen plenty of twigs that were similar in shape and colour to snakes. But you didn't mistake them for snakes.
However, in this situation, your energy budget is quite low. Your brain's goal is to keep you safe, but doesn't have many resources to draw on. The smell of dirt and leaves correlated with tension in your muscles is reminiscent of the anxiety you felt in the dark. The shape of the twig is very similar to the snake you saw at the zoo, and snakes in the forest are dangerous according to that very distinguishable memory of the documentary you saw. And many other tiny details, like the forest surroundings which probably also correlate with an amount of anxiety, from past experiences such as the many horror movies you've seen.
Of course, the brain doesn't make this complicated rational deduction. Instead, the neurons just light up together because of all these details, at the same time the brain unpacks the two predictions it made, and matches them up: what it expected, with what the neural pattern matching identified. In this case, the snake fits better than the twig.
And it's not just the image of a snake, but the associated bodily reactions. The brain releases chemicals that prepare you for running, which come together with a sensation of a heavy chest, increased heart beating and so on.
At an even higher level, the brain has also unpacked various emotion concepts: fear, disgust, courage, and so on. Depending on how you've guided your brain to respond through your past experiences and reactions, and on what these emotions mean exactly in the social context you grew up in, you will experience one emotion or another, or even a mixture of them in various degrees.
And they all have quite different effects on your body. If you are easily frightened, your brain would have been constantly predicting fear, but not materializing it, until this opportunity came along. It would ignore the prediction error of seeing a snake instead of a twig with aplomb. Instead, if you were raised in a culture where braveness is applauded, and you are with someone else, you would feel courageous, even though the bodily sensations are the same. Courage would be unpacked with a release of different chemicals, which would make you feel energized, instead of exhausted. Or, instead of courageousness, your brain could choose shame, encouraging you to hide your feelings, because if the others found out that your heart started to beat faster, they would make fun of you. And making fun of you has, in the past, affected your body budget negatively, in the form of being shunned from your entourage, or perhaps being denied some benefits by the rest of the group.
If we change some of the small pieces, the brain might create another picture. If you were just at the start of your walk and you had plenty of energy, or you weren't in the forest, but in a park, then the twig prediction would have been more suitable than the snake prediction. You would have just walked by without even realising what misprediction your brain just avoided.
If we change some of the past experiences, like perhaps you've never seen that documentary with people bitten by snakes, you could still see a snake, but you wouldn't feel the need to run, because the brain considers that your energy levels weren't affected in past experiences, so it doesn't need to make any such predictions now. And of course, in a few seconds you realize that the snake is in fact a twig and smile thinking how funny the brain is to make you see things that aren't actually there.
I think that understanding this mechanism helps us make better use of our brain's complicated mechanisms. Rather than just reacting to whatever emotions or labels it decided to apply, we now understand that this mechanism is not perfect. It is not reality; it is just our brain's best effort to make sense of reality in an instant and in every moment of our existence. Rather than accepting it as reality, we should use our own conscience to help the brain in making better choices in the future. Of course, most of the times we should not give it much thought and accept what the brain gives us: to feel happy when we see our birthday cake, to feel excited when learning new things, to feel loved when your partner offers you support, even to be afraid when in a dark unknown place. But there are a few cases where it's better if we intervene. Does it help us to panic during an exam? Is it really useful to feel angry in a particular situation? Anxiety is sometimes useful, as it tells us we need to be wary of something, but giving in to it, and interpreting as the nature of our experience, can lead to anxiety disorder. Is happiness such a good thing if all it does is give us a boost of dopamine that we will have to account for later, without any other benefits such as knowledge gained or a stronger relationship with someone else?
The brain itself keeps constantly reprogramming itself, slightly modifying the neural connections. Let's see how we can intervene in that programming.
Reprogramming
To modify a program, we need to be able to modify its code somehow. The pattern matching function of the brain is embedded in the neurons, which we can't change without surgery or something. Luckily, that is not the important part. The pattern matching network, as we've said before, effectively creates its own programs, which have the function of detecting certain things. Those are the programs that we want to change. And not just any of those. We don't care much about how the brain detects shapes or sounds. We want to change the programs that deal with emotions. While we cannot change them directly, it's clear now that we can influence them.
First of all, we can be more aware of what concepts we learn. There are many more emotions than just the simple anger, happiness, sadness, and fear. By expanding our vocabulary, we increase our emotional granularity, and the brain will have more possibilities to choose from. Instead of feeling angry, you could feel irritation, or frustration. This might just be more helpful in the civil servant scenario. They might feel compassion for your frustration and will try to find a workaround for your issue, which they surely wouldn't do if they notice you are angry and just want to punch them.
It's especially important to learn an emotion concept. As we've said, the brain can create new concepts by combining ones it already knows, but it is not the same thing. For example, the Danes have the concept of hygge, which simply put is a feeling of cosiness in the presence of others. Anybody can feel that, but for them the feeling is much more complex, and distinguishably different from mere cosiness, simply because they have ingrained cultural experiences regarding it.
It is not enough to just read the definition of a concept. You need to have repeat experiences of it. You need to learn how to recognize it in different contexts and how to differentiate it from other similar emotions. It's like learning a word in another language. At first it will sound strange, empty. But with practice you soon apropriate its meaning. When you hear it, you don't have to think about it, the meaning comes together with the sound. You would still have to think about how to place it in a sentence correctly, conforming to all the grammatical rules. You need to know if it's the most proper word in a context. At first, this will require some thought, it won't come automatically. But with even more practice, you will learn this as well, and when you hear someone use the word incorrectly it will make you cringe, because the brain has already unpacked this emotion, an almost instantaneous process which, along the way, involved detecting that the sentence you heard was incorrect. And still, there will be phrases that use the word, and confer a new meaning, which will be new to you. But you will learn that too and expand your vocabulary. It's the same with emotions.
Although not mentioned in the book, I was wondering if it's not possible for you to create your own emotion concepts. Sure, you won't have the benefit of generations of cultural entrenchment (unless you can somehow teach your concept to your community and have them accept it), but you could at least nudge your brain in the direction you want. Try to define the concept: what you should feel, in what contexts, how you should react, why you should feel it, what it means to you. Base it on existing emotions and specify what is different. Perhaps imagine an alien species and try to describe how this emotion would be used among them. And then try to apply it to your own life. See if you can detect contexts where you should feel it. At first it would be hard, it won't feel much different, just like a new word feels empty the first time you hear it. But perhaps with time you will start to feel this emotion instead of the more general one you were feeling.
And it's important to associate a word to it. This miracle of humankind helps a lot with learning new concepts. Just think of how I described what you would feel in the civil servant scenario. It's much simpler to just say "anger". Using words is our own way of creating summaries. And the brain will pick that up, and it will be easier to summarize the concept. A word can be the alias of a concept, a shortcut to everything that concept represents. You can just hear a word like "war" and that can spark emotions and bodily reactions, automatically and almost instantaneously.
Another way of affecting what emotions you feel in certain contexts is to correct your brain. When our program receives the image of a tiger and says it is a cat, you correct it. It will adjust its internal computation and next time it will have a higher chance of answering correctly. You can do the same with emotions. When you feel anxiety before an exam, try to transform it into determination. Remember that the emotion concept that is activated does not represent some absolute truth. Just like a book can sometimes be suitable for throwing and other times it is not. The conditions that lead to anxiety can be the same that lead to determination. The brain just activates the instance that it has learned would be the most suitable.
This is usually easier said than done and it does require practice. Awareness is particularly important. You should try to decompose the emotion you are feeling into the smaller concepts that form it: what are the inner sensations you are feeling, what is the context you are in, what social constraints have led you to feel that certain emotion. Afterwards, you can recompose these smaller pieces into an emotion that is more useful to you at that moment. This is as if you were consciously trying to help your brain learn a desired pattern, rather than letting it do the work all by itself.
Remember that bodily sensations play a big part in emotions. Be aware of what your inner sensations are. Notice that you are hungry and that might lead you to take different decisions than if you weren't. Butterflies in your stomach might suggest that you are feeling a connection towards the person you are talking to, or it might just be the first symptom of a flu. An important aspect of bodily sensations is that they are very slow to change, and this can be important for what your brain decides. For example, in the twig or snake scenario, once you experienced fear, and your body got flooded with hormones, it will take a while for them to be flushed out, and you will still feel tense for some time even though you realized the snake was actually just a twig within a few seconds. It's like when you are hungry and you eat more than you should. This is because it takes a while for the body to digest the food and inform the brain that the energy crisis has been resolved. Personally, I noticed that when something upsets me, that feeling lingers on for a while and affects my emotions in unrelated situations, even though I might not even remember what upset me in the first place. Being aware of this, I can give my brain extra information, that that tension that's left in my body should not be used to judge the current situation, because they are not connected. Sometimes it works.
There are more tips in the book. I just wanted to touch the ones that deal with the programming aspect of our brains' mechanisms. I wholeheartedly recommend reading the book. I can't do it enough justice with my own take on it, but maybe it helps in understanding it better. For me, personally, writing this article helped a lot in solidifying my understanding, and rereading passages from the book, I felt I gained new insight.
