accessible account admin ago allow apparently appears clients common dev down download during few free hardware history importance investigate ipn isn link log maintain manually minutes move notice orders paid paypal problems project reported sales seller send servers service shared similar somewhat supplies supply top update verify

We suffered an unplanned outage this afternoon, lasting about 30 minutes from ~4:15-4:45pm MST. Our service appears to be restored as of a few minutes ago. More details will be reported here as Dev/Ops staff investigate and provide them.

If you have made any download sales which went unfulfilled during this outage, for PayPal sales, log into PayPal and go to My Account > History > IPN History. PayPal may allow you to resend the IPN from there. If you cannot resend the IPN, or for download sales not paid for via PayPal, you can fulfill orders manually from your E-junkie Seller Admin > "Send free download link".

We do apologize for the inconvenience.

UPDATE:

Apparently a router suffered a critical power supply failure at the datacenter where we colocate our own servers and other hardware. Their router's failure also caused an outage for other clients also colocated at the same facility, so I'm sure they've got quite a few irate people demanding explanations and reassurances right now.

This actually bodes well for avoiding any similar problems in the future, as they're certain to perform a thorough audit of all their facility's common/shared equipment to verify everything else is in top shape, and also figure out how their pro-grade Cisco router with dual redundant power supplies could have been taken down by a power supply failure (i.e., did both supplies somehow fry at the same time, or was one already dead but somehow escaped notice, or what exactly?). This is already the best datacenter in town, so there really isn't any better facility to "move up to" that would also be physically accessible to us for upgrades or emergencies involving our own equipment.

This is also somewhat eerie timing, as our Dev/Ops staff recently had already been discussing ways we could somehow maintain service even in the case of a broader datacenter outage such as this, so this incident just underscores the importance of pursuing that project.