After spending time with asylum seekers, we highlight the importance of not grouping them with the radical few who may have travelled among them