At 09:42 UTC, Tray encountered a minor performance issue which caused around 50% workflow step attempts to fail and start to retry, during this time request/response workflows may have also timed out waiting for a response, CAPI request may have also timed out and some auth refreshes may have also been affected. The cause of this was due to a network configuration which succeeded in our pre-production environment however, when deployed onto production caused unintended consequences, we rolled back the change at 09:52 UTC. All retried steps either completed during the incident window or shortly afterwards. We apologise for any inconvenience this incident may have caused. If you have any questions or concerns, please feel free to reach out to support@tray.io.
Posted Oct 13, 2023 - 11:48 BST
This incident affected: Workflow Execution and Public APIs (Connectivity API).