Lab mice go wild: making experiments more natural in order to decode the brain

In a dimly lit laboratory in London, a brown mouse explores a circular tabletop, sniffing as it ambles about. Suddenly, silently, a shadow appears.

In a split second, the mouse’s brain whirs with activity. Neurons in its midbrain start to fire, sensing the threat of a potential predator, and a cascade of activity in an adjacent region orders its body to choose a response — freeze to the spot in the hope of going undetected, or run for shelter, in this case a red acetate box stationed nearby.

From the mouse’s perspective, this is life or death. But the shadow wasn’t cast by a predator. Instead, it is the work of neuroscientists in Tiago Branco’s lab, who have rigged up a plastic disc on a lever to provoke, and thereby study, the mouse’s escape behaviour. This is a rapid decision-making process that draws on sensory information, previous experience and instinct.

Branco, a neuroscientist at the Sainsbury Wellcome Centre at University College London, has wondered about installing a taxidermied owl on a zip wire to create a more realistic experience. And his colleagues have more ideas — cutting the disc into a wingspan shape, for instance. “Having drones — that would also be very nice,” says Dario Campagner, a researcher in Branco’s lab.

The set-up is part of a growing movement to step away from some of the lab experiments that neuroscientists have used for decades to understand the brain and behaviour. Such exercises — training an animal to use a lever or joystick to get a reward, for example, or watching it swim through a water maze — have established important principles of brain activity and organization. But they take days to months of training an animal to complete specific, idiosyncratic tasks. The end result, Branco says, is like studying a “professional athlete”; the brain might work differently in the messy, unpredictable real world.

Mice didn’t evolve to operate a joystick. Meanwhile, many behaviours that come naturally — such as escaping a predator, or finding scarce food or a receptive mate — are extremely important for the animal, says Ann Kennedy, a theoretical neuroscientist at Northwestern University in Chicago, Illinois. They are “critical to survival, and under selective pressure”, she says. By studying these natural actions, scientists are hoping to glean lessons about the brain and behaviour that are more holistic and more relevant to everyday activity than ever before.

Habit-linked brain circuits light up in people with eating disorders

As neuroscientists continue to hone their naturalistic set-ups using the latest technologies for brain imaging and behaviour tracking, they are finding better, more nuanced ways to use animals to study pain responses and conditions such as Down’s syndrome and autism. Others are rethinking popular theories about aggression and fear. And some are looking for ways in which these methods could enable richer studies of human behaviour, says Sarah Lisanby, a psychiatrist who directs the Division of Translational Research at the US National Institute for Mental Health (NIMH) in Bethesda, Maryland. That could be a game-changer for research into some psychiatric conditions.

“Until we understand the brain basis of what goes on for individuals when they are experiencing symptoms, we’ll continue to not effectively serve them,” says Lisanby, who in the past year helped to launch more than US$25 million in funding for research to quantify natural behaviours in humans and other animals.

Researchers in the field acknowledge that there is a lot to learn about their new set-ups, and the utility of the approach is an open question. “Do we really learn more by letting animals do what they want?” asks Sandeep Robert Datta, a neuroscientist at Harvard Medical School in Boston, Massachusetts, who studies naturalistic mouse behaviour. “Collectively we’re just getting started.”

Back to the wild

The approach takes cues from early ethologists, who studied natural behaviour through detailed diary entries and long-running activity logs of their favourite birds or insects. Dutch biologist Niko Tinbergen, considered one of the founders of ethology, would spend hours sitting on sand dunes on the North Sea coast jotting down descriptions of gull behaviour. (He won the 1973 Nobel Prize in Physiology or Medicine for his contributions to ethology, alongside two other scientists.)

Sex, food or water? How mice decide

These biologists studied natural behaviour in some depth, but “they never got to the brain, because they couldn’t”, says Branco. By contrast, scientists who have been able to explore the brain have a limited palette of behaviours that are accessible for study.

Some teams, such as Branco’s, are studying specific behaviours while recording brain activity using electrodes mounted on animals’ heads. Downstairs from his team’s fake-predator experiments, for instance, researchers at the Sainsbury Wellcome Centre are tracking animals as they seek food. The team has built an arena with little wheels embedded in the floor. When a mouse digs at the wheels, cereal pellets pop out at a rate chosen by the scientists. The goal is to reproduce the variability of food sources in the wild.

Other labs let the mice mind their own business, and catalogue what they do from moment to moment. Even seemingly simple behaviour has a complexity that is interesting to neuroscientists, says Datta. “Running in an empty bucket in the dark represents a significant cognitive challenge to the animal,” he says. Using 3D imaging, Datta and his colleagues have catalogued a ‘grammar’ of mouse body language, broken into simple modular actions or ‘syllables’ such as rearing up on the hind legs or bobbing the head¹. They have used it to look at how different stimuli or genetic manipulations might change the patterns of actions (see ‘Cataloguing behaviours’).

The computing power and analytical techniques needed to record, automatically track and catalogue hours of behaviour have become available only in the past five years — aided by machine learning. This has set the field in motion. For example, neuroscientist Mackenzie Mathis at the Swiss Federal Institute of Technology in Lausanne debuted an open-source, motion-tracking software package called DeepLabCut² in 2018, which has been installed 500,000 times. And other, similar packages have emerged.

“We have the tools to answer these big questions about what gives rise to behaviour, which is so fundamental to who we are,” says Mathis. “I don’t see a limit.”

Already, the young field is producing a slew of insights into the diversity of animal behaviour and how the brain creates it.

Act natural

Some discoveries question long-held ideas about why animals behave the way they do. For instance, researchers have often suggested that the behaviour of female animals is made more variable by hormones that govern the menstrual cycle, such as oestrogen. But when Datta’s team left mice to explore an environment, the team found that, overall, female behaviour was less variable than that of males, and that oestrogen levels had little effect on behaviour³.

Studying mice fighting each other led Kennedy and her colleagues — who at the time were members of neurobiologist David Anderson’s lab at the California Institute of Technology in Pasadena — to rehabilitate a classic ethological theory that had fallen out of favour. The idea, from Austrian ethologist Konrad Lorenz, who shared the 1973 Nobel prize with Tinbergen, was that an internal signal of aggression builds up until it tips the animal into action. Some critics thought the model was overly simplistic, lacked a neurobiological basis and didn’t account for learning.

‘Mirror neurons’ fire up during mouse battles

Kennedy and her colleagues, however, found evidence for the process in mice⁴. When a mouse interacted with another, a group of neurons in the hypothalamus gradually ramped up activity to a level that correlated with what the mouse did. At low levels, the animal might freeze or ignore the other mouse. But as levels built up over tens of seconds, it might show signs of aggression, such as trying to mount the other mouse. At high levels, mice began to attack others outright. The neurons seem to function like a volume dial for aggression — and what’s more, the levels vary between mice, with some creatures almost never attacking and some quick to do so⁴.

Other experiments refine previous theories of how the brain controls the body. Decades of studies had suggested that a region called the amygdala governs fear, “and that everything that deals with defensive response has to involve the amygdala”, says Branco. When his team looked at mice escaping from a predator, they found a circuit that acts as a shortcut from a mouse’s eyes to the back of its brain, where it can initiate the movement for escape⁵. Later on, the mouse’s amygdala might help it to learn from the experience — but the mouse doesn’t need this region in the moment.

Pause over pain

Some research into natural behaviours might already have clinical implications. Ishmail Abdus-Saboor at Columbia University in New York City and his colleagues use naturally behaving mice to study pain, hoping to build a better picture of its causes and potential treatments than standard models allow. “If you see the doctor for pain, it’s not because you sat at home and someone poked you in a constrained area,” he says. It’s because walking hurts or lying in bed gives you back pain. “We haven’t been measuring that in animal models.”

In 2021, a team led by Abdus-Saboor and Victoria Abraira, a neuroscientist at Rutgers University in New Brunswick, New Jersey, published a preprint showing that a commonly used anti-inflammatory painkiller called meloxicam seemed to work fine when tested in a standard pain assay, but not when the mice were behaving spontaneously. This suggested that the animals were still in pain but that conventional tests were missing something⁶.

Chronic pain can be treated — so why are millions still suffering?

In both set-ups, mice were given an injection in their paw that causes inflammation, and then received the painkiller. In the standard test for sensitivity to pain, the scientists subjected the paw to heat and saw that the mice had a minimal response, suggesting the drug was working.

But when the team observed mice behaving spontaneously, they noticed that certain actions, such as rearing onto the hind legs, persisted — the mice were still showing symptoms of pain. “This was quite provocative and a little shocking and alarming,” says Abdus-Saboor. Spontaneous behaviours revealed a more complex side of the pain response, such as the way an injured foot can change the way a person walks. Perhaps animal models are not capturing the full experience of pain, says Abdus-Saboor — which could explain why painkillers that seem to work in rodents have often failed in human trials.

Abdus-Saboor consults for the US pharmaceutical companies Eli Lilly and Doloromics, which are both considering adopting naturalistic assays in drug discovery. Other groups are also working on naturalistic models of pain.

Better science

Studying natural behaviour has produced some interesting findings — but it could also improve neuroscience more broadly.

At the NIMH, neuroscientist Yogita Chudasama directs the Rodent Behavioural Core, a centralized facility that helps researchers across the National Institutes of Health (NIH) to characterize rat and mouse behaviour. Her unit is setting up equipment that will allow researchers to collect data on spontaneous behaviours over long time periods to decrease variability in experiments and make the findings more reliable. In a typical experiment, an animal might be lifted from its home cage and taken somewhere else to be observed. But this new environment could affect how it behaves. Longer-term observation flattens out variables that might be affecting the animal. Furthermore, an environment with less human interference will allow the rodent to behave more naturally than it would in a constrained setting. The next step would be to integrate brain-activity recording.

Some researchers using the core facility are monitoring animals for the long term, to compare those in which a particular gene has either been left intact or mutated in a way that disrupts its function. Some of the changes that result can be subtle. “We believe that by looking over long timescales, we will see nuances of behavioural impairments,” says Chudasama.

Vivek Kumar, a neurogeneticist at The Jackson Laboratory in Bar Harbor, Maine, is also looking for ways to improve animal models. He has been studying behaviour in animal models of Down’s syndrome and autism — conditions that come with cognitive changes that are hard to reproduce in animals. But by looking instead at movement, which is much simpler to do, Kumar has found that animals with gene variants related to these conditions display differences in gait⁷. If these differences are caused by the same genes or circuits as the cognitive changes, says Kumar, then an intervention for one could influence the other. The team hopes to use tests of motor behaviours to screen hundreds of compounds for their effects on gait.

Cash injection

The interest in natural behaviour is spreading from animals to humans, and with it lots of cash. The NIH will dish out $20 million in 2024 and 2025, as part of its BRAIN Initiative, for researchers who wish to develop systems that track behaviour and brain activity in humans. “There is a lack of access to what the brain is doing when people engage in complex behaviour,” says Lisanby, who helped to develop this and other funding opportunities to support naturalistic neuroscience.

Human psychiatric conditions such as obsessive compulsive disorder, can manifest in the lab and be studied in a scanner. But most episodes occur at home, where it is a huge challenge to monitor brain activity while people are mobile. Lisanby hopes that the NIH fund will help researchers to develop tools for measuring the brain and behaviour outside the lab. This might include sensors that people can wear at home, and mobile brain-recording devices that are better than those currently available.

In such a new field, there are plenty of teething problems. Abdus-Saboor says that it can be hard to find researchers who have the mix of skills required. “Most experimentalists are not trained in mathematics, computation, computer science and coding. So we do find a disconnect,” he says. Last year, he and others started a course for graduate students and postdoctoral researchers at The Jackson Lab on quantifying behaviour.

Many scientists have ambitious plans for the young field. A lot of teams, including Branco’s, have dreams of tracking more than one animal, and over timescales of weeks to months. They hope to understand how the brain chooses between behaviours, to monitor the social dynamics of groups and even to study how brains lay down memories or plan for the future. To track multiple animals, neural recording would need to be wireless to prevent cables from getting tangled and use advanced algorithms to track movements. Existing systems can have trouble distinguishing between animals when they interact and overlap, especially if they are similar sizes and colours. Mathis wants to catalogue mouse behaviour over an animal’s lifetime, and use the information to create ‘digital twin’ mouse models to be used as a reference.

Researchers acknowledge that conventional approaches are not going away. Although they are excited by the new wave of technologies, they are realistic about how much the latest methods can achieve. “These tools aren’t magic. They’re just a pair of glasses,” says Datta. “They help us look.”