[Major] Issue Accessing Box All Files, Notes and API

We recently addressed issues affecting downloads, uploads, and Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 9:38am and 9:50am on June 25th, 2024, some users may have experienced difficulties while working in Box. During this time, users may have encountered issues uploading and downloading files as well as retrieving results from our events API. The issue occurred as a result of an unexpected upstream failure in our database infrastructure that did not recover automatically. We were able to resolve the issue by manually triggering a failover operation of the impacted database and verifying successful completion of transactions after failover. In addition, we identified and fixed a bug in our database remediation tools that prevented our automated failover process from completing quickly for this type of failure. We are constantly testing our automated failover processes and working to improve any shortcomings as they are identified to prevent similar issues from occurring in the future. 

‌

Analysis

The issue occurred due to a failure in our database remediation service using a suboptimal upstream API to recreate failed database nodes under certain conditions. In order to ensure data integrity, our database remediation service paused further actions and escalated to our team to review the issue, at which point our engineers validated that a manual failover operation was the correct operation to restore service. Our team consulted with the upstream vendor to identify a methodology that would ensure the database remediation service would succeed for all classes of unexpected database node failures.

‌

Corrective Actions

Box has initiated the following corrective actions:

• Implement fixes to database remediation service to always succeed at recreating failed database nodes during unexpected failures.
• Increase retries in database remediation service when unexpected failures occur to ensure automated failover succeeds.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team