API Errors - Degraded Performance of the Application
Incident Report for Tray.io
Postmortem

The recent degradation in execution speed was caused by the network capacity of some infrastructure services becoming saturated by some very-high-rate executions. We increased capacity and improved our internal control mechanisms, and we continue hardening the platform against this possibility. Due to the nature of the services impacted there were briefly elevated error rates from the UI and API while our platform adjusted to demand. Executions were slowed but not disrupted, so all should have completed correctly.

Posted Aug 26, 2020 - 18:00 BST

Resolved
A performance bottleneck around extremely high-rate workflows has been resolved, workflow execution speed continues to be fast and stable.
Posted Aug 26, 2020 - 11:46 BST
Update
Execution speeds on the platform are currently stable and we continue to monitor.
Posted Aug 25, 2020 - 13:40 BST
Update
We’re still in the process of investigating reduced execution speeds related to the recovery of executions.
Posted Aug 25, 2020 - 11:01 BST
Update
We’re continuing to investigate reduced execution speeds related to the recovery of executions.
Posted Aug 24, 2020 - 23:47 BST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 24, 2020 - 18:56 BST
Update
Our engineers have identified and implemented a fix and are now monitoring the results. The application is operational again.
Posted Aug 24, 2020 - 18:55 BST
Identified
We have identified and are resolving a networking issue that is impacting performance.
Posted Aug 24, 2020 - 18:34 BST
Update
We are continuing to investigate this issue.
Posted Aug 24, 2020 - 17:52 BST
Update
We are continuing to investigate this issue.
Posted Aug 24, 2020 - 17:51 BST
Update
We are continuing to investigate this issue.
Posted Aug 24, 2020 - 17:50 BST
Investigating
We are seeing intermittent elevated error rates from our APIs that we are actively mitigating. This may result in a degraded performance of our application. Users might experience delayed triggering and longer execution times or their UI failing to perform actions.
Posted Aug 24, 2020 - 17:48 BST
This incident affected: Workflow Execution, Embedded (Embedded API, Embedded App), and Workflow Builder (Workflow Builder API, Workflow Builder App, Workflow Builder Logs).