Usage-driven QuickSight cleanup

The problem

QuickSight had accumulated years of dashboards, analyses, and datasets. Business users didn’t know what existed or what most of it was for, and neither did anyone else with confidence. SPICE storage peaked at 2.4 TB. The bill has two parts, per-user fees and storage, and storage is the one that moves.

Cleaning that up by hand doesn’t work. You can’t tell whether a dashboard is dead by looking at it, and nobody wants to be the person who deleted the one an executive opens once a quarter. The decision needed evidence, and the evidence was in the logs.

How it works

Two inputs, both landing in Athena. The diagram above follows them through.

Usage. QuickSight activity logs are queried in Athena and rolled up into daily views: views per dashboard and per analysis by user, plus each dataset’s state, size, and import mode.

Inventory. Snapshots from the QuickSight API fill tables for the current dashboards, analyses, datasets, schedules, and refresh history.

The archiver joins inventory to usage and removes the assets nobody is using. Each archived asset is written to an archived_objects table with its name, type, the location of its definition, and when it was archived. A failure on one asset goes to an archive_error_log table with the function and the error, and the run keeps going.

Decisions

Usage over opinion. Logs answer the question the same way for every asset, without depending on anyone’s memory of what they still open.

Record before removing. The archive table is the receipt: what went, what type it was, where its definition went, and when. If someone does come looking for something, the answer is a query.

One failure doesn’t stop the run. Errors are logged with the function that hit them and the run continues, so a problem with one asset costs a row in an error table instead of the whole cleanup.

What it enabled

About 1,100 assets were archived, and SPICE storage fell from its 2.4 TB peak to about 700 GB. As far as I could tell, users didn’t notice, which is the point of archiving by usage.

It also cleared the way for a second track on what remained: generating datasets from configuration, deploying them identically to dev, test, and production, and sharing them through folders automatically instead of one by one.

Tradeoffs and lessons

Logs only show what was logged. A rarely used asset can look dead in a short window, so the usage window has to be long enough to catch quarterly readers.

Cleanup is a prerequisite, not a project. The storage drop was the visible result. The bigger one was that the BI layer became small enough to manage deliberately.