Explainable AI (XAI)

The field of making model behavior and outputs easier for people to understand.

Three ways to investigate a sorting model

A print archive uses a model to suggest which scans need a quality check. The team suspects it reacts strongly to dark edges.

  1. Local explanation

    For one flagged scan, an attribution method highlights a dark strip. Treat that as a hypothesis about this prediction.

  2. Global pattern

    Across a held-out set, compare flags and errors on scans with and without dark borders. Check for other differences between those groups.

  3. Sensitivity test

    Repeat the prediction on a carefully edited copy with the border removed. Record whether the score changes and whether the edit also altered useful image information.

What would make an explanation faithful?

Compare the explanation with observed model behavior under controlled changes. Methods can disagree, and an edited image may be unlike real scans. When the evidence is mixed, report the test and its limits alongside the explanation.

Different Kinds of Explanation

No single explanation method works for every system. Some methods highlight influential inputs. Some compare similar examples. Some describe model behavior at a global level. Some, such as Activation Patching, intervene inside a model to study what internal components are doing. The best method depends on what question a person is trying to answer.

It is also important to remember that an explanation can be persuasive without being sufficient. A good XAI approach should help people inspect reality, not simply create a comforting story after the fact.

Follow the idea

Interpretability asks how directly a person can follow model behavior.

Model explainability applies those questions to a particular result.

Related concepts (6)
Related Yenra articles (12)

Source · NIST’s four principles of explainable artificial intelligence

Return to Explainable AI (XAI) in the A–Z glossary