[Major] Issues with Uploads, Downloads and Box Notes

We recently addressed issues affecting Preview and Search. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 6:43am PT and 9:02am PT on August 16, 2023, some users may have experienced difficulties while working in Box. During this time, users may have experienced issues when loading document previews or searching for documents. The issue occurred as a result of database latency and load which had downstream impacts on other services. We were able to resolve the issue by by increasing duration limits for database calls from downstream services. In addition, we are working on ensuring these downstream services are more resilient in the case of database latency in order to prevent similar issues from occurring in the future. 

‌

Analysis

Starting around 2:30am PT on August 16th, a DB pod started to run hot which caused latency to increase for calling services. DB latency stayed high through the morning which, when combined with increased traffic, eventually lead to calling services being rate limited starting at 6:43am PT. The rate limit errors increased until around 8:30am PT when DB latencies came down to normal levels. Rate limits were draining slowly and it was decided to increase the rate limit for calling services to enable full recovery. With rate limits increased, full recovery happened at 9:02am PT.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Improve resiliency of calling services in cases of DB latency.
• Improve visibility into calling services in cases of increased traffic.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team