[Minor] Issues with Email Notification Delivery

For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

Incident Report for Box

Postmortem

NOTICE AND DISCLAIMER: BOX IS PROVIDING THIS PRELIMINARY INFORMATION SUBJECT TO FURTHER REVIEW AND ANALYSIS. TO THE BEST OF OUR KNOWLEDGE, THIS IS THE CURRENT STATE AND WE WILL UPDATE AS MORE INFORMATION IS CONFIRMED.

LAST UPDATED: OCTOBER 15, 2020

We recently addressed issues affecting Notifications, Uploads and Downloads. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 1:07 AM and 5:10 PM PDT on 9/12/2020, some users may have experienced difficulties while working in Box. During this time, the async-job processing pipeline was degraded. The issue occurred due to a combination of a latent bug causing a growing payload size, non-optimized configuration and a network glitch. We were able to resolve the issue starting on 9/12 by fixing the bug, optimizing the configuration and isolating the affected network link. In addition, we are working on enhancing our capability for handling thundering herd problems and taking steps to improve runbook and other processes in order to prevent similar issues from occurring in the future.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Posted Oct 15, 2020 - 17:16 PDT

Resolved

We're considering the issue fully resolved at this time. If you receive any new issues, please let us know at https://support.box.com.

Posted Sep 12, 2020 - 23:13 PDT

Monitoring

Notifications continue to be delivered as expected and previously queued emails are being processed. We’re monitoring the email notification delivery service to ensure full stability. We’ll provide more updates and information as soon as possible.

Posted Sep 12, 2020 - 18:39 PDT

Identified

We have taken steps to remediate impact and observed email notification delivery rates for the affected notification types (confirmation, upload, download, preview, delete, and comment notifications) have returned to normal. Previously queued emails are also being processed. We appreciate your patience and will provide more updates and information as soon as possible.

Posted Sep 12, 2020 - 10:31 PDT

Investigating

We are currently investigating an issue with sending notification emails for confirmation, uploads, downloads, previews, deletes, and comments. The affected email types are not currently being sent, but are still being queued for delivery. We will provide more updates as soon as possible.

Posted Sep 12, 2020 - 08:56 PDT

This incident affected: Box Web Application (Email Notifications).