For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Critical] Issues Accessing box.com
Incident Report for Box
Postmortem

We recently addressed issues affecting the Box marketing site (www.box.com). We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 10:07am PDT and 10:33am PDT on September 22nd, 2023, some users may have experienced difficulties while working in Box. During this time, the Box marketing site (www.box.com) was inaccessible and users attempting to access the site encountered a 404 error page. The issue occurred as a result of instability in Box’s marketing site vendor’s network. We were able to resolve the issue by working with the vendor to deploy a temporary workaround to remediate this issue. In addition, we are working with the vendor to improve their rollout verification process and also to identify Box-owned processes to remediate similar issues should they arise in the future. 

Analysis

The cause of this issue was a change in the leader election process of a key component in the vendor’s configuration management pipeline. This change triggered a race condition in a downstream component in the pipeline, which resulted in overwriting some customer configurations with an empty configuration. While the change was rolled out gradually with a canary process, the vendor’s automated rollout verification mechanism was not able to identify the issue. The issue was eventually detected by the vendor’s data plane monitoring systems, which alerted due to the rise in errors being served.

Corrective Actions

The following corrective actions have been completed or are planned by the vendor and Box:

  • Vendor: Identify and resolve the underlying race condition in configuration lifecycle that resulted in the unintended modifications.
  • Vendor: Improve tooling and processes to more quickly and safely rollback components in the configuration management system.
  • Vendor: Improve internal monitoring and visualization to better assess and understand source of error response alerting to expedite detection and remediation.
  • Box: Identify and implement Box-owned solution to remediate similar issues should they arise in the future.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team

Posted Nov 15, 2023 - 09:59 PST

Resolved
After further monitoring, this incident is now considered resolved. box.com has been restored to full functionality. If you continue to experience any issues, please contact Box Support at https://support.box.com.
Posted Sep 22, 2023 - 10:40 PDT
Monitoring
Our team has taken steps to remediate this issue and is seeing improvement when accessing box.com. We are continuing to monitor for any additional impact.
Posted Sep 22, 2023 - 10:34 PDT
Identified
Our team has identified the underlying cause of this issue and is working to take remediating steps. We will provide additional updates as they become available.
Posted Sep 22, 2023 - 10:33 PDT
Investigating
Our team is investigating an issue with accessing box.com. Users may see errors when attempting to access the box.com URL. Users can avoid this by accessing app.box.com. We will provide additional information as it becomes available.
Posted Sep 22, 2023 - 10:30 PDT
This incident affected: Box Website.