A useful spam filter keeps unwanted mail from interrupting people while delivering the messages they need. Evaluation therefore needs to look at mistakes in both directions, the cost of delay and the work of correcting a verdict.
SpamSquelcher’s 2003 announcement promoted network-perimeter filtering and asserted that it worked without false positives. That is a historical vendor claim. A present-day assessment should define the messages examined, the independent review method and the observed result.
Define the two kinds of error
A false positive is legitimate mail treated as unwanted. A false negative is unwanted mail treated as legitimate. Define the categories for the review: an unwanted newsletter, ordinary spam, phishing and a malicious attachment have different consequences.
Keep “unknown” available when reviewers cannot confidently classify a message. A user complaint is a valuable lead, but the message may need examination through the organization's approved reporting process. Microsoft’s submission documentation illustrates how administrators can send suspected threats or wrongly blocked messages for analysis.
Choose a sample that answers the question
Record the time window, the mailboxes included and how messages were selected. A sample containing only quarantined mail can tell you about quarantine mistakes; it cannot by itself measure unwanted mail that reached inboxes. Include appropriate evidence from delivered mail and user reports.
Use the same review rules when comparing two settings. Preserve uncertainty and avoid presenting a small convenience sample as the whole organization. Have authorized reviewers handle sensitive mail within the organization's privacy and retention rules.
Fictional example: 1,000 reviewed messages
This invented example demonstrates denominators, not a product benchmark. Reviewers confidently classify all 1,000 messages: 800 legitimate and 200 unwanted.
| Reviewed category | Delivered | Blocked or quarantined | Total |
|---|---|---|---|
| Legitimate | 792 | 8 | 800 |
| Unwanted | 20 | 180 | 200 |
| Total | 812 | 188 | 1,000 |
False-positive rate: 8 ÷ 800 = 1% of legitimate mail was blocked. False-negative rate: 20 ÷ 200 = 10% of unwanted mail was delivered.
A different question is how much delivered mail was unwanted: 20 ÷ 812 ≈ 2.46%. The numbers describe the same sample but use different denominators. Label each result with the question it answers.
Also record how long the eight legitimate messages waited and how much staff time recovery took. One delayed order can matter more to a team than several easily dismissed newsletters. If a message is harmful rather than merely unwanted, assess that severity separately.
Resolve mistakes with narrow changes
First establish why the message received its verdict: content, sender authentication, a policy rule or another feature. Review the original message through the supported workflow and correct the relevant cause. Record the scope and expiry of any temporary exception.
Microsoft’s allowlist guidance explains that broad allow mechanisms can bypass important checks and recommends reporting filtering mistakes for analysis. The exact behavior depends on the product and method. An exception that helped one sender should be reviewed again when its purpose ends.
Use a known contact route to resolve a supplier's delivery problem. A request inside a suspicious message to disable filtering is part of what needs verification.
Make the review repeatable
Choose an owner and record a baseline before changing settings. Compare the same measures after the change: both error rates, unresolved classifications, delay and handling effort. Check a later sample to see whether the result persists.
The downloadable worksheet preserves the example and leaves space for a team's own definitions and counts. Use the gateway guide for the wider filtering and quarantine workflow, and sender authentication when an identity check affects the verdict.