Like a majority of Magento ecommerce users, I call realtime API quotes from FedEx/UPS/USPS to get the shipping rates. The majority of the time, everything works okay. However, on occasions the API goes down (scheduled maintenance or randomly).
Scheduled maintenance, the administrator can go ahead and set a flat rate or table rate ahead of time. However when it randomly goes down, this can cause a massive headache. An example of this, is when Fedex went down for 57 minutes on Black Friday (Source). This affected approximately 200 customers/orders.
I am thinking of either finding a extension or writing my own that calls one API (lets say FedEx), if Fedex doesn't respond within 3 seconds, it will attempt to call the API of UPS. Of course there will be mapping involved (Fedex Ground = UPS Ground and so on) and I know the quotes price won't be exact (FedEx might be $6.75 and UPS might be $7.35), however I am attempting to give the customer a shipping quote without them noticing a problem or a problem that prevents them to checkout.
My question is, how do other Magento sites prepare for this? Do you have failsafes? Are the failsafes done manually or programmatically?