[Medium] Degradation of Enterprise Events API

We recently addressed issues affecting Enterprise Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

‌

Between 2:31am and 4:45pm PDT on May 14, 2026, some users may have experienced difficulties while working in Box. During this time, customers using the streaming Enterprise Events API experienced delivery delays of up to 6.5 hours. The issue occurred when an internal indexing service unexpectedly generated events at an unconstrained rate, which caused a significant concentrated spike in the total events volume. We were able to resolve the issue by redirecting those events from our streaming enterprise events pipeline for later processing. In addition, we will exclude events originating from internal services from our enterprise events processing pipelines and proactively deprioritize unexpected surges in events to prevent similar issues from occurring in the future.

‌

Analysis

‌

There were a number of contributing factors leading to this incident and a reoccurrence on May 18, 2026 between 11:52am and 2:03pm PDT with delivery delays of up to 1.1 hours:

‌

Our internal search indexing service produced events, which should not have been created, at a rate significantly exceeding enterprise usage. These events were published to a queue to be written to the persistent datastore. Also, these events were concentrated in certain access patterns causing hot spotting in some database partitions. The significantly reduced throughput of these partitions caused the subscriber to fall behind. Once these access patterns were identified and the corresponding events removed, service was resolved promptly. Moving forward, these events will not be published.

‌

Corrective Actions

‌

The following corrective actions have been completed or are planned:

‌

• Disable Event Publishing for Search Services - We are disabling enterprise events from being produced and processed from Box internal services.
• Deprioritize Surge Workloads - We will monitor for workloads generating significant volumes of event data and deprioritize those events from the primary pipeline.
• Update Incident Response Procedures - We are updating our runbooks to account for these conditions and related actions.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team