We recently addressed issues affecting the Box marketing site (www.box.com). We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 10:07am PDT and 10:33am PDT on September 22nd, 2023, some users may have experienced difficulties while working in Box. During this time, the Box marketing site (www.box.com) was inaccessible and users attempting to access the site encountered a 404 error page. The issue occurred as a result of instability in Box’s marketing site vendor’s network. We were able to resolve the issue by working with the vendor to deploy a temporary workaround to remediate this issue. In addition, we are working with the vendor to improve their rollout verification process and also to identify Box-owned processes to remediate similar issues should they arise in the future.
The cause of this issue was a change in the leader election process of a key component in the vendor’s configuration management pipeline. This change triggered a race condition in a downstream component in the pipeline, which resulted in overwriting some customer configurations with an empty configuration. While the change was rolled out gradually with a canary process, the vendor’s automated rollout verification mechanism was not able to identify the issue. The issue was eventually detected by the vendor’s data plane monitoring systems, which alerted due to the rise in errors being served.
The following corrective actions have been completed or are planned by the vendor and Box:
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
The Box Team