I think the issue is that most people live in cities, where populations tend to me more diverse. Then most polls probably also end up disproportionately asking cityfolk. So the polls ask people who live in areas with disproportionate numbers of immigrants (relative to non-urban parts of the country), and they forget how many non-immigrants are outside the cities.