This all started with a text from a client, that their network was down (that's how these always start).  The first set of checks showed that the network had connectivity and there were both PCs and phones communicating.  The client noted that their switches were all flashing "way more rapidly than usual" - like a fool I discounted that observation, because of course your switch LEDS flash rapidly right?  (more on this later)