Timestamp Normalization Change Causing Clip Regeneration Cascade

Incident Report for Virgo

Resolved

The rollback has relieved the issue. We are observing autopx and normal procedure display behavior.
Posted Mar 20, 2026 - 16:00 UTC

Update

We are continuing to monitor for any further issues.
Posted Mar 19, 2026 - 20:22 UTC

Monitoring

A recent Virgo Cloud update modified clip request timestamps in a way that unintentionally triggered large-scale clip regeneration. This resulted in a cascading backlog, delaying processing for approximately two days’ worth of clips. The change has been rolled back, and a full review of the impact and underlying cause is currently in progress.
Posted Mar 19, 2026 - 20:21 UTC
This incident affected: Web Services (Virgo Cloud).