We recently addressed an issue affecting the public API, Downloads, Uploads, Login, and Sign. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 12:40 am PDT, on September 28, 2023, and 03:20 am PDT, on September 28, 2023, some users may have experienced difficulties while working in Box. During this time, users may have had a degraded experience or difficulty accessing Public API, Downloads, Uploads, Login, and Sign. The issue occurred due to an observed latency in one of our internal services responsible for providing security recommendations to the application. We were able to resolve the issue by deploying more instances of the impacted service. In addition, we are implementing improvements for enhanced control over the flow of traffic within our backend systems. We are also working to establish new alerts on the impacted processes that will decrease time to detect and mitigate if a similar issue take place in the future. 

‌

Analysis

Box services are underpinned by a common service communication layer called “Service Mesh” providing Box’s services with the abilities to securely communicate and automatically adapt the scale of the services to traffic bursts. During this incident we observed that the Service Mesh layer did not behave as expected. Specifically, the service did not auto-scale as planned and as a result, the service became overwhelmed leading to the issue. A deeper analysis of the issue demonstrated that the “Service Mesh” configuration, while optimum for the on-premise deployments, was no longer adapted to the traffic profile experienced in the public cloud.

‌

We have been working with the technology vendor for our Service Mesh implementation to tune the configuration according to the new environment and have been rolling out the changes incrementally. We have also increased the capacity on the critical services while rolling out the changes to not rely on auto-scaling during that period.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Configure limits for the number of retries within internal systems to improve performance during peak traffic times.
• Implement additional back pressure “circuit breakers” within our backend systems to protect the system from overloading in case of similar issues.
• Improve error handling to better differentiate Service Mesh errors from application errors.
• Improve monitoring measures to automatically detect, diagnose, and remediate such issues before customers are actually impacted.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team