We recently addressed issues affecting Box KeySafe. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

On Thursday, August 29, 2024, between 1:48am and 3:38am PDT, some users may have experienced difficulties while working in Box. During this time, some Box KeySafe customers experienced difficulty uploading new content to Box, as well as downloading content. Additionally, some customers encountered issues with Microsoft office co-authoring feature. The incident began due to a network connectivity issue within our cloud vendor that impacted Identity and Access Management (IAM) and access to Key Management Service (KMS). This caused a backup in requests to some of the Box services involved with Microsoft Co-Authoring. We were able to resolve the issue once normal operation with our cloud vendor was restored. In addition to implementing the recommended customer actions from the vendor-provided RCA, we're also working to enhance the resiliency of our system in related areas and improve visibility through health checks to detect and prevent similar issues from occurring in the future. 

‌

Analysis

A cloud vendor network incident disrupted Box's communication with Key Management Service (KMS) instances, affecting upload and download operations for KeySafe customers. This triggered a cascading effect, as requests accumulated on a backend service involved in Microsoft Office co-authoring. The increased workload on this service's Istio-proxy container led to memory and CPU spikes, causing DNS resolution errors and latency issues. These problems further hampered Microsoft Office co-authoring requests, creating a cycle of retries and failures.

‌

Corrective Actions

Box has initiated the following corrective actions:

• Using regional STS endpoints as recommended by our cloud vendor’s RCA of this issue.
• Implementing additional health checks for the impacted backend service.
• Enhancing connection handling of the impacted backed service to prevent excessive queueing.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team