For most part-time and hobbyist drummers out there – and many full-timers, too – you probably will never be asked to read a chart or define how a 5/4 time signature breaks down.
I stress with my own students that learning how to sight read has a lot more to do with opening up new avenues of musical expression across the entire drum set; discovering cool new rhythms and patterns you would never have played without sight reading them first; and providing yourself with an instant framework to focus yourself during practice.