For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Major] Errors Accessing Box
Incident Report for Box
Postmortem

We recently addressed issues affecting certain parts of the Box Web App and public API, including Logins, Uploads and Downloads. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 11:35 AM PT and 12:42 PM PT on June 1, 2023, some users may have experienced difficulties while working in Box. During this time, some users experienced slowness and intermittent failures interacting with certain parts of the Box webapp and public API, including Logins, Uploads and Downloads. The issue occurred as a result of a recent code change in our ongoing effort to improve performance and stability. We were able to resolve the issue by rolling back the code change. We are working to improve our deployment and monitoring processes to prevent similar issues from occurring in the future. 

Analysis

Just prior to the start of the issue, we deployed a change that introduced a new database query in an attempt to simplify complex legacy queries and reduce database load. This new query was more expensive than anticipated, and caused the database to overload on larger tables in the production environment. Customers may have experienced failed logins, uploads, and downloads as a result. This change was tested in our pre-production environments for a week prior to release. However, our pre-production environments have different load characteristics, making it difficult to identify the problem the query would cause under production load. We are putting additional processes in place to thoroughly validate any database query changes prior to production deployment in order to prevent this from happening again.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Rework the solution to improve expensive database queries.
  • Improve the deployment process to include more incremental stages, streamline the verification process, and expedite rollback speed.
  • Formalize a process that tests DB effectiveness to better assess query impact when we adjust/create them

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team

Posted Jun 12, 2023 - 12:46 PDT

Resolved
After further monitoring, this incident is now considered resolved. All services have been restored to full functionality. If you continue to experience any issues, please contact Box Support at https://support.box.com.
Posted Jun 01, 2023 - 14:07 PDT
Update
We are continuing to monitor for any additional impact.
Posted Jun 01, 2023 - 13:19 PDT
Monitoring
Our team has taken steps to remediate this issue and all services should be returned to full functionality. We are continuing to monitor for any additional impact.
Posted Jun 01, 2023 - 12:50 PDT
Identified
Our team has identified the underlying cause of this issue and taken remediating steps. We are seeing initial recovery to metrics and services. We will provide additional updates as they become available.
Posted Jun 01, 2023 - 12:46 PDT
Update
We are continuing to investigate this issue. The affected categories have been updated.
Posted Jun 01, 2023 - 12:39 PDT
Update
We are continuing to investigate this issue. The affected categories have been updated.
Posted Jun 01, 2023 - 12:29 PDT
Investigating
Our team is investigating an issue with the Box Web Application, Box Platform API, Box Drive, and Box Sync. Users may see errors or delayed response times when logging in, accessing items, or taking general actions in Box or related integrations. We will provide additional information as it becomes available.
Posted Jun 01, 2023 - 12:06 PDT
This incident affected: Box Platform / API (Content API, Search, Box Sign), Box Web Application (Login/SSO, Search, Preview, Sharing (Shared Links), Admin Console & Functionality, Box Sign), Desktop Applications (Login/SSO, Box Sync, Box Drive), Mobile Applications (Login/SSO, Preview, Search), Box Notes (Web Application), and Partners / Integrations (Box for Salesforce, NetSuite).