Every BI selection we walk into starts with a vendor shortlist and ends with a CFO question we should have asked first: what is this going to cost over three years, and will the team we have actually use it? The shortlist is rarely the problem. The framing is. Most matrices we see weight the wrong axes — feature checklists from a 2022 RFP, a perfunctory “supports SSO” row, and a column for “AI” that nobody knows how to score. Here is the matrix we actually use, and the reasoning behind each axis.
Semantic layer fit
This is the axis that most decisively splits the market, and it is the one most often glossed over. A semantic layer is not a calculated-field repository; it is the contract between your data model and every consumer of it. The question is not “does the tool have a semantic layer” — they all claim one — but “where does it live, who can change it, and does it federate with the layer my data team already maintains?”
If your transformation tool already exposes metrics (dbt Semantic Layer, Cube, MetricFlow), a BI vendor that insists on its own parallel definition is not a feature; it is a second source of truth and a guaranteed drift problem within twelve months. Tools that consume external semantic layers cleanly (Lightdash, Hex, Omni, Mode for some workloads) score differently on this axis than tools that prefer their own (Looker historically, Tableau via Pulse, Power BI via its dataset model). Neither approach is wrong in the abstract; the wrong one for your stack is wrong in the concrete.
Embedded analytics need
Be honest about whether you actually have an embedded use case. “We might want to embed dashboards for customers eventually” is not an embedded use case; it is a hedge that doubles your license cost. If the answer is genuinely yes — multi-tenant white-label analytics shipped to paying customers — the shortlist collapses fast: Sigma, Looker, ThoughtSpot Embed, Cube + a custom front end, or a purpose-built embedded vendor like Embeddable or Luzmo. If the answer is no, paying for embed-tier licensing is a tax on a feature you will not use, and several of the strongest internal-BI tools (Hex, Lightdash, Omni) deprioritize embedding for good reason.
Governance
Governance is where vendor demos consistently oversell. The questions to score are not “does it have row-level security” but: how is access lineage exposed, can a non-admin trace why a number in dashboard A differs from dashboard B, and what does an audit log look like when an executive asks for one at 4pm on a Friday? Tools with strong git-backed definition workflows (Lightdash, Looker via LookML, Omni via its modeling layer) make this materially easier than tools where artifacts are clicked into existence and live only in the application database.
Total cost across three years
Year-one license is the smallest piece of three-year cost in almost every BI deployment. The matrix axis we use sums: licenses (fully loaded across viewer, explorer, and creator tiers), warehouse compute generated by the tool’s query patterns (a 3x range across vendors for the same dashboard), implementation services, and the headcount required to maintain whatever proprietary modeling layer the tool imposes. Per-seat pricing that looks reasonable at fifty users is rarely reasonable at five hundred, and consumption-based pricing that looks reasonable at five hundred users can become unbounded when a single dashboard goes viral internally. Model both curves at your projected three-year size before signing.
Hiring market
Whatever you pick, you will need to hire for it. The hiring-market axis scores how easy it is to find someone in your geography who can do real work in the tool within their first month. Tableau, Power BI, and Looker have deep talent pools. Sigma and ThoughtSpot are growing. Hex, Lightdash, Omni, and Mode are smaller markets where you are likely hiring data engineers and training them on the tool — which is fine if your team’s center of gravity is engineering, and a tax if it is not. This axis correlates strongly with implementation cost and with the speed at which a replacement hire can ramp.
AI features: substance vs surface
Every vendor has shipped an “ask your data” interface. Most of them are demoware against a curated dataset and degrade sharply on real schemas. The matrix axis we score is not “has AI features” but: does the AI feature ground in the semantic layer (so it cannot hallucinate a metric you have not defined), does it cite the underlying SQL it generated, and does it improve measurably with feedback over months of use? ThoughtSpot’s natural-language search has the longest track record. Snowflake Cortex and Databricks Genie ground in the warehouse’s own semantic objects. Most application-layer chatbots do not pass the grounding test and should be scored as marketing rather than product.
The matrix sketch
A workable scoring sheet has these seven columns and weights that reflect your situation: Semantic-layer fit (20%), Embedded need match (10–25% if relevant, 0% if not), Governance depth (15%), Three-year TCO at projected scale (20%), Hiring market in your geography (10%), AI feature substance (10%), and Migration cost from your current tool (10%). Score each vendor 1–5 per axis, document the rationale per cell, and re-run the math when any input changes. The output is rarely surprising once the weights are honest. The discipline is in keeping the weights honest.
The matrix does not pick the vendor. It forces the conversation about which axes you are willing to lose on, which is the conversation a serious BI selection turns on.