Our Most Expensive Search Index Served Zero Queries

We got a CPU alert on the database cluster that backs our provider search. The obvious hypothesis was a traffic spike. The obvious hypothesis was wrong — and the real answer took about twenty minutes of grep to find.

Here is what happened, because the underlying mistake is easy to make and genuinely hard to see.

The alert wasn't about traffic

The first instinct with any search CPU alert is to look for load. So we did.

Authenticated API traffic for the period was 778 requests. Not per second — total, across two weeks. Lookup volume told the same story: a flat ~1,000 provider records per day, all of it from a single customer's nightly batch job that fires at the same hour every day and has never thrown an error.

More to the point, none of that traffic touches the search engine. Provider lookups by NPI number are ordinary indexed find operations. They never reach Atlas Search. The search-backed endpoint existed and worked fine, but it wasn't where the volume was.

So we had meaningful search CPU and effectively no search queries. Those two facts only reconcile one way: the cost wasn't coming from serving queries. It was coming from having the index.

What edgeGram actually costs

Our NPI collection holds 9.6 million providers — every individual and organization in the NPPES registry, about 12 GB of documents. It carried two Atlas Search indexes.

The first, npi_search, is a conventional static mapping over names, addresses, and taxonomy codes. It backs the search endpoint and earns its keep.

The second was an autocomplete index:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "display": {
        "type": "document",
        "fields": {
          "full_name": {
            "type": "autocomplete",
            "tokenization": "edgeGram",
            "minGrams": 2,
            "maxGrams": 15,
            "foldDiacritics": true
          }
        }
      },
      "organization_name": { "type": "autocomplete", "tokenization": "edgeGram", "minGrams": 2, "maxGrams": 15 },
      "person_name": {
        "type": "document",
        "fields": {
          "last": { "type": "autocomplete", "tokenization": "edgeGram", "minGrams": 2, "maxGrams": 15 }
        }
      }
    }
  }
}

An edgeGram field doesn't store one term per token. It stores every prefix of that token between minGrams and maxGrams. The term count for a single token is min(length, maxGrams) - minGrams + 1.

Work it through on a boring name. "John Smith" in display.full_name tokenizes to two tokens: John produces jo, joh, john — three terms. Smith produces four. That's seven terms for one field. The same surname is indexed again under person_name.last for another four. Call it roughly a dozen index terms for one unremarkable individual provider.

Now multiply by 9.6 million rows. Organization names are worse, because "Cedar Ridge Family Medicine Associates" is four long tokens instead of two short ones.

The document count is 9.6 million. The term count is north of a hundred million. That index was, by a wide margin, the most expensive object in the database.

Index cost is decoupled from query volume

This is the part worth internalizing, and it's why the alert was so counterintuitive.

Atlas Search runs mongot, a process separate from mongod. It maintains its indexes by tailing the change stream on the indexed collections, and it holds index structures resident to answer queries with low latency. That work is a function of corpus size, field count, and analyzer configuration. It is not a function of how many queries you send.

An index nobody queries is not free and idle. It is fully built, fully maintained, and fully resident — it simply never returns anything to anyone. On a cluster without dedicated Search Nodes, mongot shares CPU with mongod, so that permanent overhead shows up as cluster CPU that no request in your logs can explain.

Query-volume dashboards will never surface this. You are looking for a spike; the cost is a plateau.

How we found it: grep, not metrics

Once "the cost is structural, not load" was on the table, the question became which index and whether it earns its cost. That's a code question, not a metrics question.

Start by enumerating what actually exists. This is the snippet we ran, and it's worth keeping:

// mongosh — list every Atlas Search index in a database
db.getCollectionNames()
  .filter((name) => !name.startsWith("system."))
  .forEach((name) => {
    try {
      db[name]
        .aggregate([{ $listSearchIndexes: {} }])
        .toArray()
        .forEach((idx) => {
          print(`${name}.${idx.name}  status=${idx.status}  queryable=${idx.queryable}`);
        });
    } catch (e) {
      // collection doesn't support search indexes — skip
    }
  });

That gave us 17 indexes. Then we traced each one to the code path that queries it. The autocomplete index resolved to exactly one function:

export async function autocompleteNpiProviders(
  query: string,
  options: { limit?: number; state?: string } = {}
): Promise<NpiAutocompleteResult> {
  const compound = {
    should: [
      { autocomplete: { query, path: "display.full_name", fuzzy: { maxEdits: 1, prefixLength: 2 } } },
      { autocomplete: { query, path: "organization_name", fuzzy: { maxEdits: 1, prefixLength: 2 } } },
      { autocomplete: { query, path: "person_name.last", fuzzy: { maxEdits: 1, prefixLength: 2 } } },
    ],
    filter: [{ equals: { value: true, path: "is_active" } }],
    minimumShouldMatch: 1,
  };
  // ... $search against the autocomplete index
}

It's correct code. It's decent code. It has no callers.

No route registers it. No MCP tool invokes it. No SDK method reaches it. A typeahead endpoint was scaffolded, the index was built to support it, and the endpoint itself never shipped. The index went to production alone and has been faithfully maintaining a hundred million-plus terms for a caller that doesn't exist.

The search endpoint customers actually use runs against the other index and was never affected.

The audit nobody schedules

Dead code in an application is cheap. You ship a function nobody calls and it costs you disk space and a little confusion. Dead code in a search index bills you monthly, forever, in a metric that doesn't point back at it.

The check is not complicated, which makes it easier to skip:

• Enumerate your search indexes. Most teams cannot list theirs from memory. Run the snippet above.
• Trace each one to a live call path. Not "a function that queries it" — an actual route, tool, or job that reaches that function. Grep for callers, then grep for callers of the callers. This is where ours died.
• Check maxGrams against your real query length. Nobody types a 15-character prefix before results appear. Capping at 8 removes roughly the top half of the gram range on longer tokens and costs you nothing a user will notice.
• Count your autocomplete fields. We indexed the same surname twice — once inside display.full_name, once as person_name.last. Overlapping paths multiply terms without adding much recall.
• Ask whether you need autocomplete at all. A plain text index with a prefix operator, or a small curated set of common queries, covers a lot of typeahead use cases at a fraction of the index cost.

We're removing the index rather than shipping the endpoint. If we build provider typeahead later, it comes back scoped to one field with a sane maxGrams — and it ships together with the route that calls it.

Key Takeaways

• Search index cost tracks corpus size and analyzer config, not query volume. An index with zero traffic can be your most expensive one.
• edgeGram autocomplete multiplies terms per token. min(length, maxGrams) - minGrams + 1 terms for every token, in every mapped field, in every document. Over millions of rows that compounds fast.
• mongot is a separate process sharing your cluster's CPU. Without dedicated Search Nodes, index maintenance surfaces as cluster CPU that your request logs cannot explain.
• Indexes outlive the features that justified them. Ours was built for an endpoint that never shipped and ran unquestioned in production.
• Audit by tracing call paths, not by reading dashboards. The metric told us that something was expensive. Only grep told us what.

Further Reading

• NPI Provider API — lookup by number, batch, and response shapes
• NPI Search API — the search endpoint, and the index that does earn its keep
• MongoDB: How to Index Fields for Autocompletion — official guidance on edgeGram, minGrams, and maxGrams