Disruption On Verifying IP Address on API Requests
Incident Report for Xendit
Postmortem

What happened? 

At 13:43 WIB we detected a spike of unauthorized IP errors and several customers have raised the same issue happening on their side. Upon investigation, we found out that Xendit APIs were temporarily disrupted for customers who have enabled IP Whitelist feature and confirmed that there was an issue. We immediately get our hands together to focus on system recovery, and all Xendit APIs resumed to normal operations at 13:54 WIB. 

Our post-incident investigation revealed that the root cause happened during an effort to upgrade our reliability and traceability in one of our core services. During this process, the service’s NGINX web server configuration was changed during the deployment, causing IP information not being correctly passed to IP verification service. The configuration changes were made because the default port clashed with other services’ port and it needs to be adjusted to proceed the deployment

What measures have we taken to prevent this issue in future?

Action items we are taking to prevent issues from happening again in the future:

  • Improve monitoring to detect anomalous API Gateway requests including IP verification scenarios
  • Improve operational readiness testing procedures for critical services updates and infrastructure config changes

We know that you are counting on our reliability for the smooth operation of your business. We are truly sorry for the negative impact of this incident on your customers and your business. We are committed to learn from this event and to improve our services even further.

Please contact us at help@xendit.co or through live chat at https://www.xendit.co/ if you have any questions or require any assistance regarding this issue.

We will strive to continue improving our services everyday and do our best to prevent this incident to reoccur. Thank you for your trust in using Xendit to power your business.

Posted Jan 11, 2021 - 22:58 WIB

Resolved
Dear Customer,

We would like to inform you that all issues with API product has been resolved and you are now are safe to retry.

Our team will be running a post-mortem to ensure this issue does not occur in the future.

We apologize for any inconveniences caused. And we thank you for your patience.

Let us know if you are still experiencing issues.

Regards,
Xendit Customer Support
Posted Jan 08, 2021 - 14:04 WIB
Monitoring
Dear valued customers,
The fix has been implemented and we're verifying if the issue has been fully resolved.
We will keep you updated.

Regards,
Xendit Customer Support
Posted Jan 08, 2021 - 13:59 WIB
Investigating
Dear Valued Customer,

We would like to inform you that all of our API product is not working if you use IP whitelist.
For now, you can disable the IP whitelist from our dashboard until this issue is resolved.
Our team is currently fixing the issues and we apologize for any inconveniences caused.

Thank you,
Xendit Customer Service
Posted Jan 08, 2021 - 13:53 WIB
This incident affected: API (Cards, eWallets, Cardless Credit, Virtual Accounts, Retail Outlet, Recurring Payments, Payouts, xenDisburse, xenPlatform, Direct Debit).