Evaluate a search solution with the queries your users actually need to resolve. A small, documented test set makes relevance, filters, freshness, and access rules concrete before a vendor demonstration or configuration change. Start with a copy of representative content and a written definition of a successful result.
Separate product discovery from internal document search
A shopper may need an in-stock drill bit with a particular diameter and price limit. An employee may need an approved procedure for the correct office and date. Both need relevant results, but their constraints differ. Record which fields are authoritative and which conditions must exclude a result.
For product search, distinguish part number, compatibility, stock, price, currency, and variants. Search cannot reliably filter a specification that was never captured. For documents, record owner, version, source system, modification date, and access rules. Establish how the index receives updates and deletions from those systems.
Authorization needs its own acceptance checks. Microsoft's document-level access documentation distinguishes methods for enforcing permissions and their support conditions. Check the exact connector, service version, and permission model you propose to use. A good relevance score cannot compensate for a restricted document appearing to an unauthorized user.
Build a small judgment set
Collect examples from search logs, support conversations, and common tasks. Remove unnecessary personal information. Include exact identifiers, ordinary language, common spelling errors, constrained queries, ambiguous requests, and requests for items that genuinely do not exist. Reserve some queries for later testing so you can detect tuning that only helps the development examples.
| Query | Expected behavior | Failure to investigate |
|---|---|---|
| AB-120 | Show the exact available part prominently | Similar identifiers crowd out the match |
| 6 mm masonry bit under $10 | Respect diameter, use, and the chosen currency's price bound | Text relevance overrides a hard constraint |
| dril bit | Offer a useful correction or relevant results | Silent unrelated substitution |
| travel policy | Show the current policy available to this test employee | Old versions or restricted offices' documents appear |
Have knowledgeable reviewers label which results are useful before comparing systems. Resolve disagreements about “relevant” by returning to the task. Elastic's ranking-evaluation documentation uses typical queries and manually rated documents to calculate retrieval metrics. You can begin with the same principle in a simple worksheet.
Measure the first results and the full task
Also record the position of the first useful result, zero-result cases, incorrect filter behavior, and successful task completion. Report results by query type so common easy queries do not hide failures on exact identifiers or restricted content. A timed user session can reveal whether people understand the filters and result snippets.
The search evaluation sheet includes query intent, test identity, expected results, top-five judgments, timing, and hard pass/fail conditions. Preserve the baseline and configuration with each run.
Test what happens after content changes
- Change a price or document version in the source and measure when search reflects it.
- Delete an item and check results, snippets, cached previews, and direct links.
- Revoke a test user's permission and confirm that the complete search experience follows the intended rule.
- Run a representative workload and record latency under the same conditions for each system.
- Check monitoring, failed-ingestion alerts, recovery procedures, and the staff effort needed to maintain data quality.
For AI-generated answers, add checks that the response is supported by retrievable, current sources available to that user. Test how it handles conflicting documents and missing evidence. Keep retrieval, answer quality, and authorization as separate evaluation dimensions.
Choose a solution after the responsible team accepts both the query results and operating requirements. Repeat the tests when the schema, connector, ranking configuration, or permissions change. Pair search improvements with clear navigation and task testing so users have more than one useful route to the answer.