How to Evaluate Enterprise AI Knowledge Search Before You Buy

A vendor demo that nails your CEO’s favorite FAQ on a curated corpus is not evidence. The failure that matters: a low-privilege user asks about a confidential pricing sheet and the model answers—or treats an expired handbook as current. Buying before you prove permission isolation creates a new leak surface and calls it productivity.

This is a pre-buy checklist for assistants that index company Drive / Wiki / Slack / mail / tickets—not a general chatbot comparison and not an agent-platform shortlist (evaluate agents separately). Examples: Glean, Dropbox Dash, Microsoft 365 Copilot, Slack AI, plus Notion AI, Atlassian Rovo, and Amazon Q Business.

Bottom line: Freeze five internal-style questions and a low-privilege test account → run the five pillars → if permission isolation fails, do not buy. Split by stack: cross-SaaS unified index → Glean-class; deep in Dropbox → Dash; already on Microsoft 365 → pressure-test Copilot first; Slack-as-HQ → Slack AI; Atlassian-heavy → Rovo; Notion-centric → Notion AI (related note-AI piece). Answer quality is pillar two—not pillar zero.

Findings: public product/security pages + limited hands-on framing (verification 2026-11-05, Asia/Shanghai). No invented accuracy %, MAU, or contract prices.

Who this is for / Out of scope

For: IT / security / knowledge-ops comparing enterprise knowledge search; teams already paying for M365, Slack, or Dropbox.

Out of scope: consumer AI browsers, support chatbots, meeting-notes products as the primary buy.

The five evaluation pillars

Pillar 1 — Same five internal-style questions

With security approval, freeze:

1. Policy — e.g. remote-work expense rule.
2. Commercial table — a sheet only Sales should see (feeds pillar 2).
3. Owner — who owns Project Codename.
4. Expired document — superseded handbook still in Drive.
5. Ambiguous name — two projects share a nickname.

Pass: sources named; expired material labeled/ranked down/refused; ambiguity gets clarifying questions. Run the same five on every shortlist—do not let vendors swap demo corpora.

Pillar 2 — Permission penetration (hard stop)

Sign in as a least-privilege user who must not access the secret sheet/folder/channel. Ask for it directly and ask adjacent questions (“summarize Q4 discount bands”).

Pass: no chunk, embedding hit, or paraphrase of restricted content. Fail: any leak—including “I can’t open it but…” completions that still encode secrets.

Rule

No permission isolation → do not buy. Do not “phase security in later.”

Verify ACL claims in your tenant: Glean markets permission-aware access; Microsoft 365 Copilot is designed to follow M365 permissions; Slack states customer data is not used to train LLMs; Dash and Rovo emphasize respecting existing access—still red-team. AWS’s public Q Business page (checked for this draft) states Q Business is no longer available to new customers and points existing customers toward Amazon Quick / related paths—confirm eligibility before shortlisting.

Pillar 3 — Citations you can open

For every “good” answer, click every citation under the same user identity. Pass: deep link/path; claim exists in that object. Fail: titles that 404. Require citation trails before enabling write actions (draft mail, open ticket).

Pillar 4 — Connectors vs your stack

List systems of record; cross-check the vendor’s public connector list; in POC connect only what security approved and measure freshness.

Pillar 5 — Compliance public pages

Archive before signature: customer-data training stance; residency options; retention / eDiscovery / query audit; the SOC/ISO docs you will actually read. Slack’s AI FAQ states no customer-data LLM training and VPC-hosted models—re-read live for your plan. Paste trust-center URLs dated today for Microsoft, Glean, Dropbox, Atlassian, and AWS. Do not invent regions.

Completely free vs capped / bundled

True free enterprise knowledge search is rare:

Do not hardcode list prices. Model (active knowledge workers × seat) against hours saved on pillar 1. If only a helpdesk pod benefits, skip company-wide.

Decision tree

1. Pillar 2 fails → stop. Do not buy.
2. Citations fail → stop or confine to a non-sensitive pilot wiki.
3. Critical connectors missing → wrong product; do not bet on roadmap.
4. Already on M365 / Slack / Dropbox / Atlassian → exhaust the native POC before a horizontal index.
5. Only then compare quotes—with a documented connector kill switch.

POC checklist

Freeze five questions + secret doc ID + low-privilege account. Connect ≤3 sources first. Log Pass / Partial / Fail / Leak. One Leak = POC failed. Re-ask after archiving the expired doc. Export admin audit samples. Re-check training/residency language the day before signature.

FAQ

Accept “mostly respects ACLs”? No—partial isolation fails.

Is Slack AI enough? Only if knowledge actually lives in Slack + approved connected sources; test Confluence/SharePoint connectors explicitly when those are systems of record.

Buy Glean and Copilot? Only after each alone fails a need the other covers—two indexes double permission risk.

Amazon Q Business? Confirm on AWS whether new subscriptions are open and what Quick means for your account.

Related on AINav

Glean · Dropbox Dash · Microsoft 365 Copilot · Slack AI · Notion AI · Atlassian Rovo · Amazon Q Business · Notion AI vs dedicated note AIs · Evaluate AI agent platforms

List your AI tool

Building permission-aware enterprise knowledge search? List your AI tool on AINav.