Useful utility-computing research helps you decide whether a result applies to your workload. Begin with the decision, then inspect the evidence behind the claim: what was measured, under which conditions, against which baseline, and with which costs included. A forecast, a product announcement, and an observed test result answer different questions.
This guide is for technology buyers, analysts, and IT teams comparing reports or proposals. You need the source document, the relevant methodology or test description, and a short description of your own workload. Treat missing evidence as a question to resolve before committing to a conclusion.
Classify the claim before assigning it weight
On small screens, scroll the table sideways. Keyboard users can focus it and use the arrow keys.
| Claim type | What it can establish | What you still need |
|---|---|---|
| Product specification | The documented feature or limit for a stated version | Availability, configuration, support boundaries, and workload fit |
| Benchmark | Performance under a described test | Inputs, configuration, repetition, variation, and comparability |
| Customer case study | An attributed result in one deployment | Baseline, selection effects, complete costs, and local differences |
| Market forecast | An estimate based on a model and assumptions | Definitions, date, uncertainty, and later observed results |
| Partnership announcement | The stated intention or agreement | Evidence of delivered integration, support, use, or outcomes |
Keep publication date, measurement date, and product version separate. A newly posted PDF can describe old tests. A revised webpage can retain a forecast whose target year has already passed. Follow the document’s internal dates and methodology, then compare them with the decision period.
Define the metric so another person can reproduce it
NIST’s Cloud Computing Service Metrics Description, SP 500-307 treats a metric as a defined expression, unit, rules, and measured values within stated constraints. That makes the measurement boundary a practical starting point for comparing services.
- For latency: record the operation, client location, concurrency, sample window, and percentile.
- For throughput: record completed units of useful work, time interval, correctness checks, and error rate.
- For availability: record the observation point, qualifying failures, measurement window, and exclusions.
- For utilization: specify the resource and whether the figure is an average, peak, allocation, or actual consumption.
- For cost: state the period, currency, quantities, rates, support, licenses, data transfer, retained storage, and implementation effort.
Compare equivalent service outcomes. A cheaper run with fewer validated results or weaker recovery requirements is a different service. Document the differences before interpreting the price ratio.
Reconstruct a savings claim from its boundary
This comparison does not settle the decision. A service may offer valuable improvements that the cost table does not measure. Record those outcomes explicitly and test them, so the reader can see which part of the case rests on price and which part rests on capability.
Check announcements against later evidence
In a 2012 first-person account of Opsware partnerships, John O’Farrell reported that the 2003 HP deal produced no revenue, while a later Cisco distribution relationship produced substantial sales. This is a participant’s retrospective account of commercial outcomes, with that perspective and scope. It illustrates why the existence of an announcement is a different observation from a delivered result.
For an old forecast, find the original definition and target year before looking for subsequent measurements. Market categories can change, making a later total incomparable. If an exact comparison cannot be established, explain that limitation and retain the historical forecast as historical evidence of expectations.
Write an evidence record and a bounded conclusion
Download the evidence review worksheet (plain text). Use one record per consequential claim. Preserve the source URL and document version, state the claim in your own words, and identify the observations that would change your decision.
- Record the decision and the minimum evidence it requires.
- Trace each consequential claim to a primary source or clearly identified first-person account.
- Reconstruct calculations and list omitted costs or measurements.
- Ask for missing test inputs, raw results, or contractual details where they determine applicability.
- Conclude: supported for these conditions, promising but requiring this test, or insufficient for this decision.
A strong evidence record is specific about what remains unknown. “Measure restore time on our dataset” gives an operator a task; “do more research” leaves the decision unchanged. Revisit the conclusion when the product version, workload, service terms, or underlying evidence changes.