Tag: ” cautions van Aswegen. She says a caffeine allergy is difficult to detect and can take several hours for symptoms to become apparent. Caffeine is also the last thing you associate the response with.