Turbot Pipes outage
Incident Report for Turbot
This incident has been resolved.

Unfortunately, through an operational error, we've lost metadata changes (users, workspaces, snapshots, etc) made between Sep 13, 2023, 14:57:51 and 22:31:47 US ET (~7.5hrs).

All data and metadata created or updated outside that specific time window is safe, available and not affected.

All affected users have been identified from logs and contacted individually with more details of actions they performed during that time.

Specifically, at 10:20:55 PM we mistakenly initiated a database restore in our production environment which completed at 22:31:47. This restore replaced our production metadata with a backup taken at 14:57:51, effectively erasing all metadata changes in between. We use both snapshots and point in time recovery (PITR) on the main database, but unfortunately PITR was not available for this timespan because we'd restored back over the same database instance. No interim backup snapshots were available because we'd restored from the most recent backup. As a result, approximately 7.5 hours of metadata was irrevocably lost.

We're very sorry. We know you rely on Pipes and that loss of data is frustrating and worrying. We'll be reviewing our operational procedures immediately to reduce the risk of taking future action in the wrong environment.
Posted Sep 14, 2023 - 04:37 UTC
We are continuing to monitor for any further issues.
Posted Sep 14, 2023 - 04:31 UTC
The issue has been identified and a fix is being implemented.
Posted Sep 14, 2023 - 03:33 UTC
We are currently investigating this issue.
Posted Sep 14, 2023 - 03:04 UTC
This incident affected: Turbot Pipes (Core Services, Workspaces).