Incident Date: 2023-11-11 Affected Products: Dashboard, API (E-wallet, Virtual Accounts, Retail Outlet, Payouts, xenDisburse, Direct Debit, Invoice, PayLater, QR) Payout, Virtual Accounts, E-wallet, QR, OTC, Third Party Integration, Reports, Payment API, Direct debit
What happened?
At 2023-11-11 21:48 GMT+7, we detected an increased number of errors from our monitoring, affecting about 4% of all traffic. This triggered an alert to the team and an incident response was activated.
The first fix was applied at 2023-11-11 22:31 GMT+7 and we managed to recover most of the impacted traffic. The second fix was applied at 2023-11-11 23:10 GMT+7 and we were fully recovered. We immediately focused on reconciliation and all affected transactions were reconciled by 2023-11-12 1:12 GMT+7 for the following payment channels: - Over The Counter - Payout - Payment API - Direct Debit
For the following payment channels, the reconciliation is in progress to be fully completed. They need to follow the T+1 timeline and are expected to be completed on 2023-11-13 due to the partner account statement report being generated - Virtual Account - E-wallet
Our initial incident investigation revealed that the previous update in Aug 2023 on our servers' certificate management had a bug leading to a breaking connection between our internal services at the time of the incident.
What measures are we taking to prevent this issue in future?
We are committed to building reliable services for you. To prevent similar issues in the future, we are: - Expediting the re-architecture of our infrastructure to programmatically manage our internal server certificate authorization to prevent configuration drift. - Having an additional second-factor monitoring process to detect certificate issues when the primary monitor is faulty.
We understand that you are counting on our reliability for the smooth operation of your business. We sincerely regret any inconvenience this may have caused you and your customers. We are committed to do better by applying our learnings from this event to continuously improve our services to serve you better.
If you require any assistance or have further questions, please contact us at help@xendit.co or through live chat at https://xendit.co.
Thank you for your trust in using Xendit to power your business.