question on time documentation

Doctor states he spent more than 30 minutes with a patient. He coded it a 99204 but I think I would have to down code to a 99203 because he is not specific enough with the amount of time? To me, more than 30 minutes does not constitute 45 minutes for a 99204. Thoughts?