We recently addressed issues affecting the Box marketing site (www.box.com). We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 10:07am PDT and 10:33am PDT on September 22nd, 2023, some users may have experienced difficulties while working in Box. During this time, the Box marketing site (www.box.com) was inaccessible and users attempting to access the site encountered a 404 error page. The issue occurred as a result of instability in Box’s marketing site vendor’s network. We were able to resolve the issue by working with the vendor to deploy a temporary workaround to remediate this issue. In addition, we are working with the vendor to improve their rollout verification process and also to identify Box-owned processes to remediate similar issues should they arise in the future. 

‌

Analysis

The cause of this issue was a change in the leader election process of a key component in the vendor’s configuration management pipeline. This change triggered a race condition in a downstream component in the pipeline, which resulted in overwriting some customer configurations with an empty configuration. While the change was rolled out gradually with a canary process, the vendor’s automated rollout verification mechanism was not able to identify the issue. The issue was eventually detected by the vendor’s data plane monitoring systems, which alerted due to the rise in errors being served.

‌

Corrective Actions

The following corrective actions have been completed or are planned by the vendor and Box:

• Vendor: Identify and resolve the underlying race condition in configuration lifecycle that resulted in the unintended modifications.
• Vendor: Improve tooling and processes to more quickly and safely rollback components in the configuration management system.
• Vendor: Improve internal monitoring and visualization to better assess and understand source of error response alerting to expedite detection and remediation.
• Box: Identify and implement Box-owned solution to remediate similar issues should they arise in the future.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team