I believe there are downsides only if you are drinking too much caffeine or if it is in conflict with one of your medications or existing health conditions, afaik.

Healthline weighs the pros and very few cons of caffeine.