SPARQL 1.1 conformance

Claim boundary — read this before quoting any number on this page. Quipu does not pass every approved W3C SPARQL 1.1 Query, Update, Protocol and Results test at rdf-tests 369a90d: query syntax 86/86, query evaluation 167/168, update 93/93, protocol 34/34, result format 10/10, SPARQL 1.0 query 199/242. Exceptions, each named below: federated query (SERVICE) passes 6/7, with 1 refused by policy (variable endpoints); entailment regimes are scored separately (35/70 passed, 0 failing, 35 declared non-goals); SHACL-SPARQL, OWL, RIF and D entailment are declared non-goals. What these counts are. Working Group–approved tests only. The query-evaluation manifests list 225 tests, and the 168 approved ones are scored; the 57 Proposed or unclassified are not run. The update-syntax suites are not run yet. The SPARQL 1.0 query tests are scored separately. The SPARQL 1.1 manifests hold what 1.1 added; the 1.0 tests (rdf-tests sparql/sparql10) also bear on SPARQL 1.1 Query conformance. Quipu passes 199/242 of the approved ones, with 33 failing, 0 errors and 10 not comparable; see SPARQL 1.0 query tests. This score is fitted to this suite. Quipu’s failures here were found by running this suite and fixed against it, case by case, so a perfect score is partly a record of that work rather than an independent sample. Other stores measured with the same harness were not tuned to it. Every class below is scored separately and is never combined into a single compliance percentage, because a blended figure would hide exactly the classes that are not implemented at all.

These results come from the W3C RDF Tests suite at a pinned revision, run against throwaway stores by a checked-in runner. You can re-derive every number on this page yourself — the commands are below.

For what Quipu does with a graph once it is correct — handing it to another store, and composing another store’s without trusting it — see Sharing & Federation. That page states its own claim boundary for SERVICE, including the configured-endpoint policy deviation scored below.

What was measured

Results by class

unsupported means the harness cannot execute the test at all — the capability is missing, not merely wrong. Those cases stay in the denominator and each one carries a named reason further down this page.

The final row is an arithmetic total, not a score. It is here so the class rows can be checked against the ledgers, not so it can be quoted as a percentage.

Other stores, same harness

The same discovery, test selection and result comparison, run against other stores at the same rdf-tests revision (369a90d1). Scores use RDF term equality, the rule quipu is held to. “Same value” counts failures whose answer had the right values in a different lexical form; they stay failures and are shown separately, so a design choice is not presented as a wrong answer.

The quipu row is this page’s own ledger. Quipu’s runner compares exact labels and has no same-value tag, so that cell is empty rather than zero.

Disclosure. Quipu parses SPARQL with spargebra and models RDF with oxrdf, both from the Oxigraph project. Where the two agree, part of that agreement is shared code.

Quipu’s score is fitted to this suite. Its failures were found by running this suite and fixed against it, case by case. The other stores were not tuned to this harness.

Pinned versions, the fairness rules, every competitor deviation checked by hand, and the per-case ledgers are in benchmark/competitors.

RDF syntax

The W3C RDF 1.1 and RDF 1.2 syntax suites at the same rdf-tests revision (369a90d1). Every manifest case is counted, including cases the manifests have not marked approved.

RDF 1.2 is measured and not supported. Quipu is built without RDF 1.2, so it cannot parse a triple term. The RDF 1.2 cases are enumerated from the pinned manifests and not run: a loader that rejects all RDF 1.2 input would “pass” every negative-syntax case, and those passes would read as partial support. No RDF 1.2 case is scored as a pass until the support exists.

Known deviation: literal lexical identity

The current loader canonicalizes integer and double literals: for example, "01"^^xsd:integer is stored and exported as "1"^^xsd:integer. It also converts an ill-typed boolean such as "z"^^xsd:boolean to false, and rejects some ill-typed numeric literals during loading. These are known deviations, not the intended storage contract; affected cases remain failures.

The chosen contract is to preserve lexical form and datatype as RDF term identity, including ill-typed literals, and derive numeric values separately. Distinct terms such as "01" and "1" with the same integer datatype must coexist even though their numeric values compare equal. This follows RDF 1.1 literal term equality. An expression’s effective boolean value must not replace the stored term.

The preservation fix is not implemented yet. Existing data cannot recover discarded spellings; a future compatibility plan must preserve legacy term identity, exact retraction and history without inventing missing lexical forms.

Ledgers: rdf11-syntax.json and rdf12-syntax.json.

SPARQL 1.0 query tests

The approved W3C SPARQL 1.0 query-evaluation tests (sparql/sparql10) at the same rdf-tests revision (369a90d1), run by the same runner: 199/242 pass.

Most SPARQL 1.0 answers are RDF result-set graphs (rs:ResultSet). They are read with rdflib, pinned, with literal normalisation off, so "01"^^xsd:integer stays 01. Quipu never reads its own expected answers. Where the answer numbers its solutions (rs:index), order is compared, not just the multiset. The unsupported cases have RDF/XML answers, which the runner does not read.

Some failures are the lexical-form design choice described above (a number is stored by value, so "01" reads back as 1); others are engine defects being fixed. Every case, with its diagnostic, is in sparql10-evaluation.json.

Named SPARQL 1.0 deviations

These are the current ledger’s non-passing cases, including loader errors and runner limitations. Listing a case does not claim its engine defect is fixed.

SPARQL 1.2 query tests

The W3C SPARQL 1.2 query tests (sparql/sparql12) at the same rdf-tests revision (369a90d1). No SPARQL 1.2 case is Working Group–approved yet; every one is counted anyway. 3 of 269 pass, and 263 are not run.

Most of SPARQL 1.2 needs grammar or terms Quipu does not have: triple terms, the VERSION declaration, base direction, new codepoint escapes. Those cases are listed and never run, because a parser that rejects all SPARQL 1.2 input would “pass” every negative case. The few cases that need nothing new are run by the same runner as everything above.

Ledger: sparql12.json.

Query evaluation, by feature family

The family is the pinned suite’s own directory for each manifest, so this grouping is re-derivable by anyone holding the same suite revision. It is the map of where the work is: the largest failing families are where conformance moves the most per fix.

The not-yet-passing tests, named

Every test that does not pass is listed here with its W3C identifier, so a claim of progress can be checked against a specific case rather than a count.

Federated query (SERVICE)

Quipu passes 6/7 approved W3C Basic Federated Query cases. SERVICE is a query-planned subquery path using the same declarations and labels as RemoteProvider; it is separate from GraphProvider whole-query fanout and is not open federation. The variable-endpoint case is a deliberate policy deviation because query data cannot widen the configured remote allowlist.

Why a class is unsupported

Grouped by the reason the runner recorded.

Corrections

Kept on the page so a changed number never changes silently.

• 2026-09-24: update was 37 of 93. Until this date the page reported update 37/37. The runner discovered only 37 of the 93 approved update tests: the delete, delete-data, delete-insert, delete-where, clear and drop manifests declare their tests with Turtle’s a rather than rdf:type, and the parser matched only the latter. On the full 93, quipu passes 93/93. The runner now pins the approved count per class at the pinned suite revision and refuses a run that discovers a different number.

Re-derive these numbers

git clone https://github.com/w3c/rdf-tests /tmp/rdf-tests
git -C /tmp/rdf-tests checkout 369a90d1a60c021b746df2e411da0ff36258a758
cargo build --release --bin quipu --bin quipu-server --features shacl,onnx,server
QUIPU_BIN="$(cargo metadata --no-deps --format-version 1 | python3 -c 'import json,sys; print(json.load(sys.stdin)["target_directory"])')/release/quipu"
SUITE="/tmp/rdf-tests/sparql/sparql11"
python3 benchmark/public/sparql11_syntax.py --suite "$SUITE/syntax-query" \
  --quipu "$QUIPU_BIN" --output /tmp/sparql11-syntax.json
python3 benchmark/public/sparql11_evaluation.py --suite "$SUITE" \
  --quipu "$QUIPU_BIN" --output /tmp/sparql11-evaluation.json
python3 benchmark/public/sparql11_federated.py --suite "$SUITE" \
  --quipu "$QUIPU_BIN" --output /tmp/sparql11-federated-query.json

A nonzero exit from an incomplete runner is expected while any test fails: it writes the complete ledger first, then reports that not everything passed. Use the regression gate below to tell worse than committed apart from not yet perfect.

Per-class reproduction commands are embedded in the ledger itself, under reproduce.per_class.

The regression gate

python3 benchmark/public/check_regression.py \
  --baseline benchmark/public/results/sparql11-evaluation.json \
  --candidate /tmp/sparql11-evaluation.json

It exits nonzero when a class’s pass count drops or when a test that passed in the committed ledger stops passing, and it names the tests. Improvements exit zero and print the tests that newly pass — a prompt to refresh the baseline. This is what runs in CI on every release, so a published number cannot silently get worse.

Full ledgers

• benchmark/public/results/sparql11-syntax.json
• benchmark/public/results/sparql11-evaluation.json
• benchmark/public/results/sparql11-federated-query.json

Each row records its class, test identifier, manifest, query and result paths, status, and diagnostic or unsupported reason.

W3C SHACL conformance

Quipu passes 98/98 manifest-reachable SHACL Core cases. SHACL-SPARQL remains a deliberate non-goal: 0/22 pass and 22 are unsupported. Quipu does not advertise SHACL-SPARQL support; the upstream SPARQL validator is incomplete, and support requires the full 22-case manifest gate rather than a partial claim. This score uses the context-free native validator on whole graphs; the write path is measured below.

Pinned W3C Data Shapes revision: 9c863967bceaef1a87c24e4dd761eda763823120. The pinned manifest exposes 120 approved cases (98 Core + 22 SHACL-SPARQL). nodeValidator-001.ttl exists in the checkout but is not manifest-reachable and is not scored.

Write-path conformance

The same 98 Core cases are also written through the real /knot handler, whole and split across writes (subject halves, one write per subject, one write per triple, each in both orders). 90/98 agree with the W3C verdict under every split; 7 diverge under at least one; 1 could not be judged under at least one split (the write gate answered cannot verify instead of a SHACL verdict). A divergence is a case whose final graph violates (or conforms) while the write gate accepted (or refused) every write. These are known defects, listed in the ledger with the splits that expose them; the CI gate fails on any new divergence and on any entry that no longer diverges, so this number can only fall.

Ledger: shacl-write-path.json.

Entailment-regime commitments

2 of 6 regimes are goals (RDF, RDFS): 35/35 of their cases pass. The remaining 4 are deliberate non-goals. Ledger re-derived 2026-10-21T15:07:06Z by CI run, from quipu 8fb57af0472a. Local RDFS and OWL extensions beyond a goal regime are not standards-regime claims.

Do not read the goal-regime fraction as “nearly done”. The two numbers have different characters. Most RDF-regime cases are bind* tests answerable under simple entailment, so they pass without any additional inference — a high RDF score is not evidence of an entailment engine. The RDFS score DOES reflect one: an RDFS closure (rdfs2/3/5/7/9/11) is materialised into the graph’s companion inferred graph and composed into the default graph when the regime is in force, which is what a query like SELECT ?x WHERE { ex:a ?x ex:c } needs — its predicate is a variable, so the entailed triple has to EXIST and cannot be produced by rewriting the pattern. What remains failing is not more of the same closure: it is container and axiomatic shapes beyond those six rules, and OWL-flavoured cases filed under RDFS.

Machine ledgers: shacl-core.json and sparql11-entailment.json.