Disruption to create payment requests, payment methods, pay invoices, and recurring payment

Incident Date: 2023-11-29

Affected Products: Virtual Account, Over-The-Counter, QR, E-Wallet, Cards, Direct Debit, Checkout, Recurring Payments

Report URL: https://status.xendit.co/incidents/5g9n6t6fks89 

What happened? 

At 2023-11-29 13:51 GMT+7, we detected errors from our monitoring, affecting around 2% of all incoming requests and causing disruption to create payment requests, payment methods, payment to invoices, and recurring payments. This triggered an alert to the team, and activated an incident response.

At 14:13, we completed the recovery process and resumed processing all new incoming requests.

Our incident investigation revealed a bug in our software deployment tool that resulted in the inadvertent re-execution of previous configuration changes, leading to disruptions.

What measures are we taking to prevent this issue in the future?

We are committed to building reliable services for you. To prevent similar issues in the future, we are:

• Working closely with our vendor to improve the quality of their software deployment tool.
• Enhancing the safeguard mechanism of our software deployment to limit the impact of similar bugs.
• Updating our network configuration to allow faster recovery for similar disruptions.
• Continuously reviewing the performance of our vendor and tooling to uphold the highest quality standard.

We understand that you are counting on our reliability for the smooth operation of your business. We sincerely regret any inconvenience this may have caused you and your customers. We are committed to do better by applying our learnings from this event to continuously improve our services to serve you better.

If you require any assistance or have further questions, please contact us at help@xendit.co or through live chat at https://xendit.co.

Thank you for your trust in using Xendit to power your business.