Every data catalog vendor demo starts the same way: a polished UI, a pre-seeded environment, a smooth lineage graph, and a confident slide deck. None of those tell you whether the product will survive contact with your warehouse. The differences that decide whether a catalog earns or burns its line item live one layer below the demo. Force these four questions to the front of the conversation before you let anyone screen-share.

What does ingestion break on?

Every catalog claims connectors for Snowflake, BigQuery, Databricks, dbt, Tableau, Looker, Power BI, and “200+ more.” The honest comparison is what each connector silently drops or mangles. Push for specifics:

  • Schema drift. When a column’s type changes mid-week, does the catalog version the change, overwrite, or quietly desync from the warehouse? Ask for the actual reconciliation policy, not the marketing word for it.
  • External tables and zero-copy clones. Iceberg, Delta, and Snowflake-on-S3 setups break catalogs that assumed a single physical store. Ask how the connector handles a table whose data lives in a bucket the catalog cannot list.
  • Views referencing missing objects. A common warehouse state. Some catalogs error out the entire ingestion pass; better ones flag the broken reference and continue. The difference is whether your catalog stays current after one bad release.
  • Permission scope. What minimum role does the connector need? Many vendors demo with ACCOUNTADMIN and discover at deployment time that their parser silently skips objects the service role can’t DESCRIBE. You want documented behavior under least-privilege, not a workaround invented during onboarding.
  • Throughput and incrementality. A 50,000-table warehouse is not exotic. Ask how long a full re-scan takes, whether incremental ingestion exists, and what triggers a forced full re-scan. “Daily full crawl” is a maintenance bill disguised as a feature.

If a vendor cannot give you a written failure-mode matrix for their connectors, you’re buying a demo, not a product.

How is lineage actually computed?

Lineage is the headline screenshot. It’s also the feature most likely to be smoke and mirrors. The right question isn’t “do you have column-level lineage?” — every vendor will say yes — but “what’s your parser, and what does it not understand?”

  • SQL parsing depth. Catalog lineage is usually one of: query log heuristics, AST-based static parsing, or a hybrid. Each fails on different SQL: heuristics miss CTEs and dynamic SQL; AST parsers stumble on warehouse-specific dialects; hybrids inherit both failure modes unevenly. Ask for the parser’s open-source provenance. “Built in-house” usually means “lags Snowflake feature releases by 6–12 months.”
  • dbt-aware lineage. If they parse manifest.json, ask whether they reconcile dbt’s logical lineage with the warehouse’s physical lineage when they disagree. They will disagree. The answer should not be “we show whichever was ingested last.”
  • Cross-tool stitching. Lineage from Snowflake → dbt → Tableau → Slack is the demo. Ask which of those edges are inferred from query logs, which from native APIs, and which from name-matching heuristics. Name-matching is fragile and usually undisclosed.
  • Refresh latency. A lineage graph that’s accurate on a 24-hour delay is a different product from one accurate within 15 minutes. Both are defensible; conflating them is not.

The healthiest tell: a vendor who shows you a lineage edge their parser would miss and explains why. Anyone who claims complete coverage is either wrong or hasn’t looked.

How do they handle PII auto-tagging precision?

PII auto-classification is where catalog economics get interesting. Vendors quote precision and recall numbers from internal benchmarks against synthetic datasets. Ask three follow-ups:

  • What’s the false positive rate against your real data? Run the classifier against a sample of your actual warehouse before you sign. If they refuse a paid POC against your data, that’s the answer.
  • What does a misclassification cost? A column flagged as email but actually a free-text comments field will, in some catalogs, automatically restrict access via downstream policy enforcement. Ask exactly which downstream actions the classifier triggers and how a steward overrides them.
  • What’s the model lifecycle? Is the classifier updated, retrained, or static since 2023? Static models silently degrade as your data evolves.

A 95% precision number with no false-positive cost analysis is marketing. A 92% precision number with a documented override workflow and weekly retraining cadence is engineering.

What’s the actual user-onboarding cost?

Catalogs are sold as productivity tools and deployed as governance projects. The gap is the user-onboarding tax — and it’s the line item most vendors will not surface unprompted.

  • Time to first useful query. From login, how long until a typical analyst finds a table they didn’t know existed and trusts what they find? If the answer is “after 30 days of curation work by your stewards,” the catalog is a CMS, not a discovery tool.
  • Stewardship load. How many full-time equivalents does the vendor’s reference customer dedicate to keeping descriptions, ownership, and certifications fresh? If they don’t quote a number, ask for a customer reference who will.
  • Search quality on day one. Ask to search the demo environment for a term that isn’t in the curated descriptions. If the search returns garbage until a human writes documentation, the product depends on labor you’ll have to fund indefinitely.
  • Adoption telemetry. Does the catalog tell you which teams use it and which don’t? You need this to defend the renewal in eighteen months. “We can build that for you” is not the same as “it ships.”

The catalogs that survive in real organizations are the ones whose vendors answer these four questions in writing before the procurement conversation starts. Make them.