Unfortunately, through an operational error, we've lost metadata changes (users, workspaces, snapshots, etc) made between Sep 13, 2023, 14:57:51 and 22:31:47 US ET (~7.5hrs).
All data and metadata created or updated outside that specific time window is safe, available and not affected.
All affected users have been identified from logs and contacted individually with more details of actions they performed during that time.
Specifically, at 10:20:55 PM we mistakenly initiated a database restore in our production environment which completed at 22:31:47. This restore replaced our production metadata with a backup taken at 14:57:51, effectively erasing all metadata changes in between. We use both snapshots and point in time recovery (PITR) on the main database, but unfortunately PITR was not available for this timespan because we'd restored back over the same database instance. No interim backup snapshots were available because we'd restored from the most recent backup. As a result, approximately 7.5 hours of metadata was irrevocably lost.
We're very sorry. We know you rely on Pipes and that loss of data is frustrating and worrying. We'll be reviewing our operational procedures immediately to reduce the risk of taking future action in the wrong environment.
Posted Sep 14, 2023 - 04:37 UTC
Monitoring
We are continuing to monitor for any further issues.
Posted Sep 14, 2023 - 04:31 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 14, 2023 - 03:33 UTC
Investigating
We are currently investigating this issue.
Posted Sep 14, 2023 - 03:04 UTC
This incident affected: Pipes (Core Services, Workspaces).