A study of Neanderthal skulls suggests that they became extinct because they had larger eyes than our species.
As a result, more of their brains were devoted to seeing in the long, dark nights in Europe, at the expense of high-level processing.
By contrast, the larger frontal brain regions of Homo sapiens led to the fashioning of warmer clothes and the development of larger social networks.
The study is published in Proceedings of the Royal Society B.
Neanderthals are a closely related species of human that lived in Europe from around 250,000 years ago. They coexisted and interacted briefly with our species until they went extinct about 28,000 years ago, in part due to an ice age.
The research team explored the idea that the ancestor of Neanderthals left Africa and had to adapt to the longer, darker nights and murkier days of Europe. The result was that Neanderthals evolved larger eyes and a much larger visual processing area at the backs of their brains.
The humans that stayed in Africa, on the other hand, continued to enjoy bright and beautiful days and so had no need for such an adaption. Instead, these people, our ancestors, evolved their frontal lobes, associated with higher-level thinking, before they spread across the globe.
Eiluned Pearce of Oxford University decided to check this theory. She compared the skulls of 32 Homo sapiens and 13 Neanderthals.
Pearce found that Neanderthals had significantly larger eye sockets - by an average of 6mm from top to bottom.
Although this seems like a small amount, she said that it was enough for Neanderthals to use significantly more of their brains to process visual information.
"Since Neanderthals evolved at higher latitudes, more of the Neanderthal brain would have been dedicated to vision and body control, leaving less brain to deal with other functions like social networking," she told BBC News.
This is a view backed by Prof Chris Stringer, who was also involved in the research and is an expert in human origins at the Natural History Museum in London.
"We infer that Neanderthals had a smaller cognitive part of the brain and this would have limited them, including their ability to form larger groups. If you live in a larger group, you need a larger brain in order to process all those extra relationships," he explained.
The Neanderthals' more visually-focused brain structure might also have affected their ability to innovate and to adapt to the ice age that was thought to have contributed to their demise.
There is archaeological evidence, for example, that the Homo sapiens that coexisted with Neanderthals had needles that they used to make tailored clothing. This would have kept them much warmer than the wraps thought to have been worn by Neanderthals.
Prof Stringer said that all these factors together might have given our species a crucial advantage that enabled us to survive.
"Even if you had a small percent better ability to react quickly, to rely on your neighbours to help you survive and to pass on information - all these things together gave the edge to Homo sapiens over Neanderthals, and that may have made a difference to survival."
The finding runs counter to the idea that Neanderthals were not the stupid, brutish creatures portrayed in Hollywood films; they may well have been as intelligent as our species.
Oxford University's Prof Robin Dunbar, who supervised the study, said that the team wanted to avoid restoring the stereotypical image of Neanderthals.
"They were very, very smart, but not quite in the same league as Homo sapiens," he told BBC News.
"That difference might have been enough to tip the balance when things were beginning to get tough at the end of the last ice age," he said.
Up until now, researchers' knowledge of Neanderthals' brains has been based on casts of skulls. This has given an indication of brain size and structure, but has not given any real indication of how the Neanderthal brain functioned differently from ours. The latest study is an imaginative approach in trying to address this issue.
Previous research by Pearce has shown that modern humans living at higher latitudes evolved bigger vision areas in the brain to cope with lower light levels. There is no suggestion though that their higher cognitive abilities suffered as a consequence.
Studies on primates have shown that eye size is proportional to the amount of brain space devoted to visual processing. So the researchers made the assumption that this would be true of Neanderthals.