We experienced a minor outage on Thursday Sep 29 2022, between 11:01 AM and 12:12 PM (BST) which affected a small set of customers’ workflow executions. Specifically, 1-2% of steps running between those times may have executed twice or more times.
The incident was caused by change to the way execution state data is stored in order to make it more efficient. During the deployment, both the old storage format and the new storage format were in use at once, and, under very limited and specific circumstances, it’s possible that our workflow engine would have resumed execution of a workflow from a previous state.
We have held a post-mortem during which we learnt more about this incident. We now know how and why it happened, and we also know how to avoid making the same mistake in the future again.
If you have any questions or concerns, or you think that you may have been affected by this incident, then please feel free to reach out to support@tray.io. We will work with you to remedy the situation. Thank you for your patience and understanding.