Even if the odds of successful interbreeding were just 5 percent, Neanderthal genes would make up the majority of the human genome today. As it is, a lack of viable sex explains why none of the Neanderthals’ mitochondrial DNA made its way into modern humans, and why so little of their main genome did.
Currat and Excoffier suggest that either modern humans and Neanderthals didn’t have sex very often, or their hybrids weren’t very fit. They favour the first idea. According to their model, it would only have taken between 197 and 430 liaisons between ancient humans and Neanderthals to fill 1-3 percent of modern Eurasian genomes with Neanderthal DNA. Considering that they two groups probably interacted for 10,000 years or so, it would have been enough for one human to sleep with one Neanderthal every 23 to 50 years.
From what I gather in the comments this is due to the fact that if there was a wave of advance very small levels of admixture per unit of advance can build up rather rapidly. I think this is easy to express in temporal rather than spatial terms.
For example, let’s imagine a population of modern humans expanding into a population of Neandertals. The original source population doesn’t receive any more contributions after the initial push, so you have a series of admixture events over time. Assuming 5% admixture per generation, this is the dilution of the “original ancestry” which would occur over 30 generations, or 750 years:
The model outlined in Ed Yong’s post needs to be examined with care though. No doubt there are all sorts of assumptions which can be disputed. Though I think I accept the final result as entirely plausible.