Document Comparison

note

This feature is experimental and the API may change in future versions.

The Relay compiler includes an experimental feature for comparing the intermediate representations (IR) of two GraphQL documents. This feature determines whether one document is a subset of another and reports any missing selections.

This feature is particularly useful for comparing documents generated by LLMs where exact match of LLM generated documents and static expected documents is neither attainable or desirable, due to the nature of LLMs. In those cases, you could:

• Specify (via --left) an expected document with minimally required selections, and determine if the LLM generated document (via --right) contains all required selections, via the subset test that the command performs.
• Optionally, pass exepected document via --right and LLM generated document via --left to determine whether LLM generated documents contains extra selections than expected, aka. overfetching.

Usage

> relay experimental-compare-document-ir --help
EXPERIMENTAL! Compare intermediate representations (IR) of documents passed via left and right, outputs selections that exists in left but not in right.

Usage: relay experimental-compare-document-ir [OPTIONS] --left <LEFT> --right <RIGHT>

Options:
      --left <LEFT>                    Document in string, must contain exactly 1 operation
      --right <RIGHT>                  Document in string, must contain exactly 1 operation
      --schemaPaths <SCHEMA_PATHS>...  Path(s) to the full schema file(s) to convert documents to intermediate representation (IR) for comparison
  -h, --help                           Print help

You can also run the comparison pragramatically via Rust API, e.g. in the context of an eval grader, where you can report a numerical score and adjust the output as needed to better integrate with your eval setup.

use graphql_ir_diff::compare;
use anyhow::Result;

impl Grader for ComparisonGrader {
  fn grade(schema_paths: Vec<String>, expected: String, actual: String) -> Result<Output> {
      graphql_ir_diff::compare(schema_paths, &expected, &actual).map(|(score, message)| {
        ... // process score/message as needed
      }).map_err(|err| {
        ... // process error as needed
      })
  }
}

Output

Detailed information about missing selection(s) from the right document, including line number and path.

How It Works

The 2 documents (left and right) are first transformed to their Intermediate Representation (IR), then to Normalized Tree form, for comparison. Each node of the tree corresponds to a selection in the original document, and is checked for its existence in the other tree.

Normalized Trees

Each IR is transformed into a normalized tree structure that:

• Inlines fragments: Fragment spreads and inline fragments are inlined, while type conditions specified by the fragment are encoded in the node.
• De-aliases fields: Converts aliased fields to their original names (e.g., my_id: id → id)
• Deduplicates fields: Removes duplicate identical fields
• Tracks type conditions: Records which concrete object types can reach each scalar field at the leaf level

The normalized tree mirrors the JSON response structure of the query.

Tree Node Identity

Consists of

• Path from operation root to the corresponding selection
• Field arguments of the corresponding selection (if applicable)
• Type condition: the type that the enclosing object must be, for the corresponding selection to be included in the response. E.g. with ... on A { id }, the type condition of selection id is A if A is a concrete type, or a set of types that are A if A is an abstract type.

Tree Node Existence

A selection exists in a tree if:

• An identical selection is found
• A superset of the selection is found. A selection is a superset of another if:
  • It allows the same selection and more, with query variables. E.g. user(id: $id) is a superset of user(id: 100) because variable $id can be set to arbitrary values, including 100, at runtime.
  • Its type condition is a superset of another.

Example

Left document:

query A {
  viewer {
    timezone_estimate {
      display_name
      gmt_offset
    }
  }
}

Right document:

query A {
  viewer {
    timezone_estimate {
      display_name
    }
  }
}

Output

[INFO] Found 1 missing selection(s):
    4 │       display_name
    5 │       gmt_offset
      │       ^^^^^^^^^^
    6 │     }
* query -> viewer -> timezone_estimate -> gmt_offset

Explore tests for more examples.