[Major] Delays in Box Relay

We recently addressed issues affecting Relay workflows. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between July 21, 2024, 3:00 PM PT and July 22, 2024, 2:00 AM PT, some users may have observed that some Relay workflows were delayed.

Analysis

Relay workflows are executed on a shared pool of compute resources. This issue occurred because the system was under unusually high load, which delayed the execution of some workflows. We were able to resolve the issue by rebalancing resources across outstanding workflows.

We are currently enhancing our monitoring and notification systems to ensure that engineers receive timely alerts about resource allocation issues. Additionally, we are focusing on improving our tools for rebalancing resources in order to proactively prevent similar issues from happening again in the future.

‌

Corrective Actions

Box has initiated the following corrective actions:

• Automatic recognition of workflow causing peak load and optimizing resource utilization.
• Reducing time to action by improving engineering troubleshooting steps to diagnose and detect resource allocation issues that cause delays.
• Improve internal tooling to allow quicker intervention for on-support specialists.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team