Degraded Alerts Queue Processing
Resolved
Oct 07 at 02:00am UTC
On October 6, 2025 around 1pm UTC we identified an issue affecting the processing of triggered alerts with delayed execution. Delayed execution applies to alerts with a minimum duration greater than 0 seconds. Such alerts remained in our internal queue and did not execute as expected. Additionally, automated video requests with delayed execution were also not getting processed.
The root cause was a configuration issue in our AWS Lambda queue, introduced after last production release on October 2, 2025. The AWS Lambda configuration issue was resolved by 6pm UTC the same day. Rebalancing script was also developed and deployed with the fix at that time. Delayed triggered alerts were gradually released within 3 hours and delayed video requests were gradually released within 7.5 hours from the time of fix deployment. The issue was fully resolved by October 7, 2025 2am UTC.
We sincerely apologize for the inconvenience this may have caused. We’re implementing additional proactive monitoring to prevent similar issues in the future.
Affected services
Created
Oct 02 at 11:00am UTC
Degraded processing of triggered alerts with delayed execution.
Affected services