For more information about our Incident Response and Communications please read this support article.
We also maintain a list of Known Product Issues separate from this site here.
We recently addressed issues affecting the execution of Relay workflows. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 12:00 am PDT on September 13, 2023 and 4:46 pm PDT on September 14, 2023, some users may have experienced difficulties while working in Box. During this time, scheduled workflows created in Relay were not executing and non-scheduled workflows were executing with delays. The issue occurred as a result of code change introduced earlier this year related to migration of scheduled workflows to new servers. We were able to resolve the issue by fixing the faulty code, which resolved the issue with our internal message queueing system that was responsible for this impact. In addition, we improved logging and observability to prevent similar issues from occurring in the future.
Analysis
The investigation has shown that the root cause was faulty code, which in specific circumstances, was creating multiple duplicate workflows, rapidly filling internal message queues. This resulted in delays in the execution of non-scheduled workflows and additionally temporarily prevented scheduled workflows from executing correctly. The faulty code was introduced as a performance improvement while doing the migration of scheduled workflows to new servers earlier this year.
Corrective Actions
The following corrective actions have been completed or are planned:
- Corrected the faulty code responsible for this issue.
- Introduce more testing of this code to reduce the likelihood of similar issues occurring again in the future.
- Add alerts and metrics measuring number of scheduled workflow executions and introducing additional logging to facilitate investigation of similar cases in future
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team