Disruption to Xendit APIs and Dashboard
Incident Report for Xendit
Postmortem

Incident Report - Disruption to Xendit APIs and Dashboard

Incident Date: 2021-05-06

Affected Products:

  • Payouts
  • XenPlatform Accounts
  • Direct Debit
  • E-wallet payments v2
  • Retail Outlet Payments
  • PHP Disbursements and PHP Batch Disbursements
  • Activations of new services via Dashboard
  • Go-live requests via Dashboard

Report URL: https://status.xendit.co/incidents/7qkrt15w7wwy

What happened?

At 20:45 WIB we detected an increase of 5xx server errors from Xendit APIs and Dashboard. In the process of investigation, we found that the errors were caused by connectivity issues on some web services. We immediately focused on system recovery, Xendit APIs and Dashboard were gradually resumed to normal operations from 21:19 WIB and fully recovered at 21:41 WIB.

Our post-incident investigation revealed that there was a system maintenance to renew our infrastructure service mesh certificate at 20:30 WIB. Prior to this maintenance, we have successfully tested the certificate renewal in the testing environment. We identified the root cause was due to a process error during this maintenance. It caused the web services which were still using the old certificate to be unable to communicate with the web services which are using the new certificate.

What are we doing to improve?

Measures we will be taking to prevent similar issues from happening again in the future:

  1. Refine testing and release procedure to make sure thorough verifications are done and safeguards are in place when performing each steps of system maintenance
  2. Improve incident response plan to allow faster recovery from similar incident

We know that you are counting on Xendit’s reliability for the operation of your business. We are truly sorry for the negative impact of this incident which affected you and your customers alike. We are committed to improving our reliability and to be able to serve our customers better in the future.

Please contact us at help@xendit.co if you have any questions or require any assistance regarding this issue.

We will strive to continue improving our services everyday and do our best to prevent a repeat of this incident. Thank you for your trust in using Xendit to power your business.

Posted May 10, 2021 - 16:34 WIB

Resolved
All services are working normally again. We apologise for the inconvenience caused.
Posted May 06, 2021 - 21:36 WIB
Update
Most of our services have recovered.

The following services are being monitored for further issues:
- Indomaret payments
- Xenplatform Accounts
- Account activation via Dashboard
Posted May 06, 2021 - 21:29 WIB
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 06, 2021 - 21:19 WIB
Update
We have identified additional disruptions during this time:
- Activations of new services via dashboard
- Go-live requests via dashboard
Posted May 06, 2021 - 21:10 WIB
Update
The following services are also impacted:
- Retail Outlet Payments
- PHP Disbursements and PHP Batch Disbursements
Posted May 06, 2021 - 21:05 WIB
Investigating
Dear customers,

We are observing disruptions to several Xendit services.

Impact: Requests to the following APIs will return 5XX errors
- Payouts
- Xenplatform Accounts
- Direct Debit
- E-wallet payments v2

Our team is investigating.
Posted May 06, 2021 - 20:55 WIB
This incident affected: API (eWallets, Retail Outlet, Payouts, xenPlatform Accounts, Direct Debit) and Dashboard (Onboarding, Batch Disbursements).