Incident Report - API and Dashboard Disruption

Incident Date: 2023-01-17

Affected Products: Dashboard, API (Cards, eWallets, Cardless Credit, Virtual Accounts, Retail Outlet, Recurring Payments, Payouts, xenDisburse, xenPlatform Accounts, Direct Debit, Invoice, PayLater, QR Codes, xenPlatform Transfers, Customers endpoint), Third Party Integrations, and xenInvoice

Report URL: https://status.xendit.co/incidents/dmn9l9pb178z

What happened?

[2023-01-17 01:45 GMT+7] We noticed an unusual spike in memory usage which caused one of our core infrastructures to be restarted, as a result APIs and dashboard were unavailable.

[2023-01-17 01:49 GMT+7] The core service recovered and payments processing was able to resume. We continued to monitor the issue and started reconciling affected payments.

[2023-01-17 02:03 GMT+7] Fix identified and released. The incident was resolved.

[2023-01-19 22:00 GMT+7] Affected payments during the incident time period were fully reconciled and processed.

What measures have we taken to prevent this issue in future?

Extend our monitoring capabilities to identify similar issues earlier so customers are not impacted.
Enhance one of our core infrastructures to be more reliable.
Improve automated recovery to minimize disruptions to payments processing in the event of failure in this core system.

‌

We understand that you are counting on our reliability for the smooth operation of your business. We sincerely regret any inconvenience this may have caused you and your customers. We are committed to do better by applying our learnings from this event to continuously improve our services to serve you better.

‌

If you require any assistance or have further questions, please contact us at help@xendit.co or through live chat at https://www.xendit.co/.

‌

Thank you for your trust in using Xendit to power your business.

Posted Jan 20, 2023 - 19:44 WIB

Resolved

This incident has been resolved.

Reconciliation of affected transactions are being processed and are expected to complete over the next 24hours.

Posted Jan 17, 2023 - 14:03 WIB

Monitoring

We have rectified the issue and are monitoring the situation, and looking into reconciliation.

Posted Jan 17, 2023 - 13:49 WIB

Identified

Dear customers

We have detected issues in the following services:
- All API
- Dashboard access

We're looking into the issue now and will update with more information periodically.

Posted Jan 17, 2023 - 13:45 WIB

This incident affected: Dashboard (Dashboard), API (Cards, eWallets, Cardless Credit, Virtual Accounts, Retail Outlet, Subscriptions, Payouts, xenDisburse, xenPlatform Accounts, Direct Debit, Invoice, PayLater, QR Codes, xenPlatform Transfers, Customers endpoint), and xenInvoice.