The poll could be taken to suggest that it is the future expectation of immigration that drove the support for leave (hence the power of the Turkey "threat" and the "breaking point" poster), and is thus a matter of flow rather than stock. I'd personally take that with a pinch of salt, given popular ignorance of the ethnic and foreign-born share of the population. Clearly immigration is an immediate concern, not the calculation of a future discounted utility. We know that anti-immigrant sentiment is highest in areas with low immigration, which means that "pressure on public services" isn't a credible explanation, and we also know the issue is immigration across the board, not just the free movement of EU citizens. The suggestion during the campaign that fewer Polish plumbers would mean more Bangladeshi curry chefs was not what most leavers were hoping to hear.
It would be easy to assume that abstract xenophobia is the driver, or that immigration (like the EU) is a proxy for modernity more generally, standing in for developments as diverse as gay marriage and kale juice. There certainly seems to be evidence that reactionary views, rather than age or education, are the best indicator of whether someone voted leave. Some even espy a cultural divide: "liberal cosmopolitanism versus anti-liberal populism", or a cognitive division between "those who dislike difference – signifying a disordered identity and environment – and those who embrace it", but these look like magazine articles masquerading as academic studies. There are plenty of reactionaries in London and 38% of those who voted in Sunderland opted for remain (I doubt many of them would self-identify as liberal cosmopolitans). I'm going to suggest an alternative thesis: that the concern over immigration is driven in part by internal migration, as experienced by the "left behinds", and related to this, that the antipathy towards the EU incorporates a large dose of resentment towards London.
Much has been written about the impact of commonwealth immigration on old textile towns in Lancashire and Yorkshire during the 1960s and after, but less attention has been paid to the way that internal migration impacted on mono-cultural areas such as the North East and South Wales from the 1980s onwards. In Britain, there has historically been a fundamental difference between the larger industrial conurbations and both the smaller industrial towns (that specialised in sectors such as coal, shipbuilding and steel) and rural towns. The conurbations were populated in the 19th and 20th centuries by immigrants from both the surrounding countryside and further afield (e.g. Irish and Welsh factory-hands in Manchester), while the small towns were often relatively insular with only a small number of skilled immigrants (e.g. Scottish engineers) or itinerant workers who were not encouraged to stick around (e.g. seasonal Irish agricultural workers). What distinguished these in turn from small towns in other countries, like Ireland or Italy, was the absence of regular emigration in the twentieth century.
Though we think of the pivotal shift from manufacturing to services as occurring in the 1980s, it's worth remembering that the UK has been a predominantly service economy for the best part of a century. What changed in the 80s is that service jobs, as well as manufacturing jobs, were lost from small towns to cities as industries like financial services consolidated and as globalised business services expanded. This trend was exacerbated by the development of the digital economy in the 90s. Far from allowing people to work anywhere, the Internet made remote service delivery easier and thus amplified agglomeration in city-based "hubs". This process simultaneously transferred jobs to the larger service centres (i.e. provincial cities as well as London) and accentuated the differential in wages between the metropolis and small towns. To give an example, the Isle of Wight, which voted heavily leave, is poorly-served (many online retailers won't deliver there), wages are low, the economy is overly-dependent on pensioners and the state (prisons), and talented youth head to Southampton or London.
We're familiar with the fact that many old industrial towns in the North have lost both skilled jobs and many of their young, but the same outcome - a reliance on low-wage work and a growing proportion of the elderly - has affected small towns across the South and Midlands as well. Before the 1980s, the worry that the cities would lure away the small-town youth of Britain was largely limited to those families whose children benefited from the expansion of further education in the 1960s - i.e. classic "social mobility" that often entailed geographic mobility. Thatcherism extended this deracination to skilled workers ("get on your bike"), and not just in the North and Wales but across the rest of the UK too. This was then exacerbated by the further expansion of tertiary education in the 1990s, which funnelled teenagers who might otherwise have looked to apprenticeships with local employers into often-distant urban colleges as a stepping-stone to work in the service sector.
In other words, the sense of disturbance captured in the fear of immigration may be more to do with contemporary internal migration, and the negative impact this has on family ties, than the dismantling of industries a quarter of a century ago, let alone the loss of empire. The young quitting Sunderland for London, or York for Leeds, may actually be more significant than migration from Poland (blaming migrants for your town's decline is one way of dealing with guilt over "desertion" by your adult children). In these smaller cities and towns, the average age and the proportion of OAPs has gradually increased, not because retirees are moving in but because the young are moving out. This has led to suggestions that a policy of managed decline should be adopted for some areas, further encouraging the emigration of the young or skilled. This suggestion is typically directed at old industrial towns in the North, but the problem of poor wages and too many pensioners is just as relevant in the South outside London.
If this thesis is correct - that the concern over the arrival of EU nationals during the last decade is actually resentment over the departure of native youth that started in the 80s and accelerated in the 90s - it helps to explain why concern over immigration mounts in the late-90s, 5 years before the accession of East European states to the EU in 2004. The political focus on immigration and asylum that started in the mid-90s certainly validated these as "legitimate public concerns", but its hard to believe that their resulting salience in small towns with minimal exposure to actual immigrants (let alone asylum-seekers) can be fully explained by either the power of the press or an increase in racial prejudice. There appears to be something else at work, and something more tangible and immediate than a cultural divide. "We want our country back" may have been an anguished cry directed at children who rarely phone.