Skip to main content

Building a content inventory

A content inventory is a row per page with enough attributes attached that you can decide, in bulk, what happens to it. It is the single highest-leverage artifact in a migration, because it converts "the docs are a mess" into a count. As a hypothetical worked example, "412 pages, 118 of which had fewer than ten views last year" is a sentence you can plan against. Counts like that make scoping, sequencing and redirect mapping possible instead of guesswork.

Do it before you choose a navigation model, not after.

What is a content inventory and what goes in it?​

A content inventory is a machine-readable list of every published page with its metadata, usage and disposition. One row per URL, one column per attribute you will filter on later. Start with these columns and add only what you will actually sort by:

ColumnSourceWhy it earns its place
urlCrawl or sitemapThe join key for everything else, and the input to redirect mapping
titleCrawlReveals duplicates and naming drift
word_countCrawlThe best cheap proxy for conversion effort
last_modifiedPlatform export or APIAge is the strongest single predictor of wrongness
author or ownerPlatform exportYou need someone to ask
pageviews_12moAnalyticsSeparates the corpus from the content
entrancesAnalyticsDistinguishes destinations from waypoints
inbound_internal_linksCrawlA page nothing links to is either an orphan or an entry point
has_images / has_attachmentsCrawlFeeds media handling
content_typeHumanConcept, task, reference, troubleshooting, release note
dispositionHumankeep / merge / rewrite / archive
new_pathHumanThe target URL, which becomes the redirect map

Keep it in a CSV in the repository, not in a shared spreadsheet that becomes the only copy. It is project data with a lifetime; treat it like code.

Getting the raw list​

Start from the sitemap if there is one, because it is the platform's own claim about what is published.

curl -s https://docs.example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+</loc>' \
  | sed -E 's#</?loc>##g' \
  | sort -u > urls.txt
wc -l urls.txt

Then crawl those URLs for the attributes the sitemap does not carry. Any crawler works; the point is that the output is a CSV keyed on URL. A minimal pass with wget gives you a local copy of the rendered HTML to extract from and to diff against later:

wget --input-file=urls.txt \
     --adjust-extension --page-requisites --convert-links \
     --wait=0.5 --random-wait \
     --directory-prefix=./crawl

Two checks before you trust the list. First, compare the sitemap count against the platform's own page count in its admin UI; a gap means unlisted or permission-gated pages that will surprise you later. Second, pull the top 500 documentation URLs from analytics and confirm each one is in urls.txt. Live traffic to a URL that is not in the sitemap is exactly the page a customer will notice missing.

Joining analytics without lying to yourself​

Export twelve months, not three — documentation traffic is seasonal around releases, and a quarterly window will condemn pages that matter in March. Export at the page-path level, deduplicate query strings, and be explicit about what the numbers can and cannot tell you.

Pageviews measure findability and demand together, and you cannot separate them from the report. A page with no views may be useless, or it may be a correct answer that search never surfaces. Before archiving anything on view count alone, check whether it has inbound internal links and whether support ever sends it to customers. Ask the support team for their five most-pasted links; that list rarely matches the analytics top five, and both are true.

Turning rows into decisions​

Score, then decide. A workable first pass:

  • Keep as-is — accurate, used, correctly typed. It converts mechanically.
  • Merge — two to five pages that are one page. Very common around "getting started", "quickstart", "installation" and "setup".
  • Rewrite — the topic is needed, the page is not correct or not the right content type. This is the row that costs real time; count it separately.
  • Archive — no longer true, or true of a version nobody runs. Archived is not deleted: it still needs a redirect target, usually the nearest surviving parent.

Run the pass with the person who owns the product area, not alone, and do it in one sitting per area with a timebox. Decisions get faster over the first batch of pages and then level off.

How long does a content inventory take?​

It takes as long as the disposition column takes. The crawl and join are minutes of compute; the disposition column is where all the time goes, because it needs someone who knows whether the content is still true, plus a review session per product area. Time the disposition pass on your first area and extrapolate from your own rate rather than from anyone else's.

Two ways to compress it: type-check in bulk before you read individual pages (sorting by title pattern catches most content types), and let word count carry the effort estimate instead of reading every page to judge.

What the inventory unlocks​

Three downstream artifacts come straight out of these columns and cannot be built without them: the effort estimate in scoping and sequencing, the content-type counts that drive templates, and the url to new_path pairs that become your redirect map.

Keep the CSV updated during the migration itself. Add a status column, move rows through it, and you have a burndown that a skeptical stakeholder can read without a status meeting.

Last updated on