For more information about our Incident Response and Communications please read this support article.
We also maintain a list of Known Product Issues separate from this site here.
[Major] Delay in email notifications
Postmortem
We recently addressed issues affecting customer email notifications including password reset or 2-factor authentication code emails. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
On September 12, 2023 between 5:50 PM PDT and 9:40 PM PDT, some users may have experienced difficulties while working in Box. During this time, customer email notifications including password reset or 2-factor authentication code emails may have been delayed. The issue occurred due to the loss of internet access in one Availability Zone during a scheduled firewall redeployment. We were able to resolve the issue by manually updating routing configuration on the affected instances. In addition, we are working on improving our per-instance health-checks so that they do not take any traffic until they are ready.
Analysis
During our investigation, the root cause of the incident was identified as a network routing misconfiguration that occurred during the scheduled deployment. This led to loss of internet access in one Availability Zone, thereby causing latencies in customer email notifications including password reset or 2-factor authentication code emails. We have enhanced our post deployment verification by adding a check to ensure that instances are serving traffic effectively.
Corrective Actions
The following corrective actions have been completed or are planned:
- Integrate network configuration setup in the image creation workflow instead of post-creation to establish consistency.
- Enhance the existing monitoring setup with improved alerting features to expedite issue identification and resolution.
- Optimize our per-instance health check to prevent instances from receiving traffic until they are fully prepared.
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team