The Government has been warned it ignores voters' concerns about the cost of living and immigration at its peril.