We already use Amazon S3 for our long-term download storage/caching and delivery, whether you upload files to us or use our Remotely Hosted Downloads feature, so that's about as robust, reliable, and high-performance a download service as can reasonably be expected at a remotely affordable cost. The Remotely Hosted Downloads section of this help page explains more about how this works for remotely-hosted files:
When you add new files to sell or replace your remote file with a newer version, you might consider using the "Send free link" function in your Seller Admin to send yourself a free link and test that download to completion, which would also get a jump start on caching that file in our Amazon S3 provisions. You might also try this if you get a report of an unusually slow download, as infrequently-downloaded files may get flushed from our S3 cache; then, after you test their link and download the file to completion, wait at least 15 minutes for the file to be synced to S3, use the "Re-activate expired links" function in your Seller Admin and ask the buyer to try their download again.
This FAQ page discusses some other details pertaining to reports of slow download performance: