NO IPN notification for the past three hours

[mikesay98]

It's a PayPal issue. Here is their UK Twitter (their US Twitter is silent on the matter oddly enough).



https://twitter.com/PayPalUK



What I did was take all the transaction IDs and block them on eJunkie and then send out manual codes. Then when PayPal fixes the issue and sends the IPNs, eJunkie should send nothing out.

[jerrycards]

It has been 12 hours....Somebody need to be fired in paypal.

[instantdisplay]

From what I've read - Paypal's estimation of time until fixed - 48hrs was the only time scale told to people who have spoken to them on the phone...



That better be absolute maximum time needed.



Not happy...

[jeffz]

How you blocked transaction id's on e-junkie?

mikesay98It's a PayPal issue. Here is their UK Twitter (their US Twitter is silent on the matter oddly enough).



https://twitter.com/PayPalUK



What I did was take all the transaction IDs and block them on eJunkie and then send out manual codes. Then when PayPal fixes the issue and sends the IPNs, eJunkie should send nothing out.

[jerrycards]

48 Hours... That is tooooooo long......

[bugwt]

This is bad. Any news?

[E-junkie_Monster]

We're posting updates in our service status thread here, any time a serious issue such as this comes up we'll always have the latest information on it in that forum:



http://www.e-junkie.com/bb/topic/6331



The problem is that PayPal is not sending out Instant Payment Notifications this morning, and our system (and others) rely on those notices, without them our system cannot fill any orders.



This is affecting regular PayPal checkout, which is the most common form of checkout in our cart, so its holding up a lot of transactions. When PayPal fixes the issue all of those notices should come through which will get everything that's been held up this morning processed.

[E-junkie_Guru]

PayPal has updated their system status post to announce they are now implementing a fix for this issue. Within the hour, IPNs for new payments should be getting sent to us almost immediately (as normal), and they'll be gradually catching up with their backlog of delayed IPNs over the next 24 hours:

https://www.x.com/content/update-5-issue-receiving-instant-payment-notifications-ipns-jan-27 UPDATE 5: We have implemented a fix and IPNs for transactions processed after Jan 28, 10:47 AM PST (Jan 28, 6:47 PM GMT) are being sent now. We are expecting IPNs for new transactions to start being delivered in near real-time within the next hour. We are also estimating that missing IPNs for transactions processed between Jan 27, 11:04 PM PST (Jan 28, 7:04 AM GMT) and Jan 28, 10:47 AM PST (Jan 28, 6:47 PM GMT) will be delivered within the next 24 hours.



We will provide the next update by Jan 28, 3:30 PM PST (Jan 28, 11:30 PM GMT).

Sent Jan 28, 1:11 PM PST by SC

If you are selling downloads, you can use "Send free download link" in your Seller Admin to fulfill delayed orders manually, at least for impatient buyers who complain they don't want to wait for PayPal to catch up with sending us IPN for their payment. If you do this, you may also want to enter the PayPal Transaction ID for those orders in your "Block buyers/transactions" screen, so our system won't process the original order automatically once PayPal finally sends us the delayed IPN confirming payment for that order.



Thanks to everyone for your patience and understanding as we've all been waiting on PayPal to sort out the issue at their end. Suffice to say, this has been a most harrowing day for everyone involved on all sides, but it appears an end to the trouble is now clearly in sight.

[paulusmaximus]

All seems fine now , but I am missing an entry in the EJ log.

[crazygametime]

It looks to me that the IPN service is back.

[gorilla]

Not here, still not working and i just got a sale notification from 9AM this morning. (12 hours ago)

[crazygametime]

gorillaNot here, still not working and i just got a sale notification from 9AM this morning. (12 hours ago)

I checked my record and everything seems to be working properly. It seems that the IPN service is back since 18:32:56 MST。

[E-junkie_Guru]

PayPal has posted a couple late-breaking updates to confirm they have the issue fixed, are now sending out IPNs for new payments almost immediately, and are catching up with their backlog of IPNs that were delayed during the outage.



NOTE: our system is processing orders for these delayed IPNs as we receive them from PayPal, so that accounts for any late sale notifications you may be receiving now.

https://www.x.com/content/update-7-issue-receiving-instant-payment-notifications-ipns-jan-27

UPDATE 7: We are continuing to deliver IPNs for transactions between Jan 27, 11:04 PM PST (Jan 28, 7:04 AM GMT) and Jan 28, 10:47 AM PST (Jan 28, 6:47 PM GMT).



We will provide the next update by Jan 28, 10:00 PM PST (Jan 29, 6:00 AM GMT).

Sent Jan 28, 6:51 PM PST by SC



UPDATE 6: IPNs for new transactions are currently being delivered in near real-time. We are currently deliverying the IPNs for transactions between Jan 27, 11:04 PM PST (Jan 28, 7:04 AM GMT) and Jan 28, 10:47 AM PST (Jan 28, 6:47 PM GMT). We expect to complete this within the next 24 hours.



We will provide the next update by Jan 28, 7:00 PM PST (Jan 29, 3:00 AM GMT).

Sent Jan 28, 3:28 PM PST by SC

[jeffz]

My system came in order and processed all transactions this nght.



I would like to thank e-junkie for their cooperation.



But next time I wish that e-junkie be more aware of these issues and as soon as something arise they submit a statement to all customer via email.

[jerrycards]

Paypal should give more details on what happened and make sure it will not happen again.

[jeffz]

Most of us where lost, if this was Paypal or e-junkie.



In my opinion an issue as this e-junkie should issue an immediate statement.

JerryCardsPaypal should give more details on what happened and make sure it will not happen again.

[E-junkie_Guru]

For some bigger-picture context, you may be interested in this help page, which explains our order-handling process in layman's terms using familiar real-world analogies:

http://www.e-junkie.com/ej/faq.how.laymans-terms.htm



The issue yesterday was entirely at PayPal's end. E-junkie's system was working normally the entire time, during which we continued to process orders paid through other payment services such as Google Checkout or Authorize.Net, or even card-based direct payments using PayPal Advanced/Pro that didn't go through PayPal's checkout site. We simply weren't receiving any payment confirmations from PayPal for orders paid through their checkout site, so those were the only orders affected.



We depend on receiving IPN (Instant Payment Notification) from PayPal to confirm when their checkout site has completed payment for a given order, so we can then process that order for you (issue links, send emails, log the order details); this normally happens almost immediately after the buyer finishes checkout. However, if we don't receive IPN for an order, our system has no way of knowing that order was actually placed and paid-for.



Early yesterday morning (shortly after midnight MST), PayPal's system started having some issues, or possibly one major issue with diverse side-effects, one of which was that their system stopped sending IPNs to anyone, including E-junkie. PayPal resolved their problem yesterday afternoon, at which point we started receiving IPNs for new payments almost immediately (as usual), and they started catching up with sending the backlog of IPNs that were delayed as a result of the problem, so our system resumed processing orders for both new and delayed IPNs as they rolled in from PayPal.



This service status posting on PayPal's developer site gives a timetime of updates (from newest to oldest) regarding this issue as it developed and was resolved; these are probably all the details they're going to provide on this matter:

https://www.x.com/content/resolved-issue-receiving-instant-payment-notifications-ipns-jan-27



PayPal posts notices of known issues at their end on their developer site's Live Site Status page here:

https://www.x.com/developers/paypal/documentation-tools/site-status/pp-live



If you happen to use PayPal Advanced or PayPal Payflow Pro, issues specific to card-based payments with those services are posted in their Gateway Live Status page here:

https://www.x.com/developers/paypal/documentation-tools/site-status/pf-live



Whenever we are aware of an issue affecting our system, we post notice to our Service Status & Updates subforum here:

http://www.e-junkie.com/bb/forum/6



If you're into RSS, you can subscribe to a live feed of our System Status & Updates subforum via this URL:

http://www.e-junkie.com/bb/xml/rss-forum-6.xml



I've asked Development to consider the mass-email notification suggestion. Our Twitter feed and Facebook page are managed by our Marketing team which doesn't have their pulse on the technical side of things, so we've obtained access to those social media accounts and will be able to post service status notices there if necessary in the future.