Loading documents

The catalog, resource ids and fragments, the request/provide protocol, loader patterns for disk, HTTP and embedded assets, caching and path safety.

dxn_dtcg performs no I/O. Documents enter the runtime as strings under logical names, and the runtime tells you which ones it still needs. This page covers the catalog, resource ids, the protocol, loader patterns and path safety.

#The catalog

Each DtcgRuntime owns a catalog mapping ResourceIds to document text (Arc<str>), a revision counter and the id of the root document. Every entry has a DocumentKind: a JSON document (token document or resolver), or the source text of a JavaScript or TypeScript module that expressions import (see module documents).

• On insert the runtime checks that the id is valid and the text is well-formed JSON (module text is not checked), nothing more. A valid JSON document that is wrong DTCG is accepted and reported when a resolution reaches it.
• On resolve the documents involved are parsed from their text; nothing parsed is kept between calls.
• The root is where resolution starts. from_json sets it; otherwise call set_root. The root need not be in the catalog yet; if it is not, the first resolve requests it.

#Resource ids

A ResourceId is a logical, portable, normalized document name such as themes/dark.json. Every valid id is an ordinary relative file name on Windows and Unix alike.

Invalid ids fail with InvalidResourceId. References resolve against the directory of the document that contains them, like relative URLs:

use dxn_dtcg::{DtcgError, ResourceId};

fn main() -> Result<(), DtcgError> {
    let root = ResourceId::parse("themes/base/root.json")?;

    let reference = root.resolve("../shared/palette.json#/colors/primary")?;
    assert_eq!(reference.resource_id().as_str(), "themes/shared/palette.json");
    assert_eq!(reference.fragment(), Some("/colors/primary"));
    assert_eq!(reference.original(), "../shared/palette.json#/colors/primary");

    // A fragment-only reference stays in the same document.
    assert_eq!(root.resolve("#/tokens/button")?.resource_id(), &root);

    // An id never carries a fragment, and must be portable.
    for invalid in ["library.json#/palette", "con.json", "what?.json", "dir./a.json"] {
        assert!(ResourceId::parse(invalid).is_err(), "{invalid}");
    }

    // Escaping the root, absolute paths and drive letters are rejected.
    for invalid in ["", "/absolute.json", "C:/x.json", "../../../escape.json"] {
        assert!(root.resolve(invalid).is_err(), "{invalid}");
    }
    Ok(())
}

URLs are not valid references: the path contains :, so the reference fails with InvalidResourceId and no document is requested. Map remote documents onto relative ids and fetch them yourself.

#Fragments

A fragment selects part of a document: library.json#/palette makes the palette member the source root. Fragments are JSON Pointers in URI-fragment form, parsed by the same rules as value-level $refs (pointer syntax). A malformed fragment fails with InvalidPointer when the reference is parsed; a well-formed one that addresses nothing fails when the document is loaded.

#The request/provide protocol

// As defined by dxn_dtcg (simplified).
pub enum Resolution<T> {
    Ready(T),
    Missing(Vec<DocumentRequest>),
}

pub struct DocumentRequest {
    pub resource_id: ResourceId,      // the normalized id to provide
    pub referenced_from: ResourceId,  // the document that contains the reference
    pub original_reference: Arc<str>, // the reference as written, fragment included
    pub kind: DocumentKind,           // what to provide: Json, or Module (expressions)
}

pub enum DocumentKind { Json, Module } // non_exhaustive

Resolution has ready() (Some for Ready), map(f), is_ready(), is_missing() and missing() (borrows the requests). These methods can return Missing: resolve, project, modifiers, normalize_selection, validate_complete, replace_json_and_resolve_affected, source_references, inspect_source, validate_source, preview_source and preview_occurrence.

A Missing response is:

• Complete for the pass: every missing document reachable without the missing documents’ own contents.
• De-duplicated and sorted by resource id, so the same catalog gives the same requests.
• Scoped to the active graph: resolve requests only the selected contexts’ documents; validate_complete, validate_source and source_references request every context’s. With expressions enabled, validate_complete also requests the modules every context’s sources declare.
• Possibly incremental: a missing nested resolver reveals more references once provided. Loop until Ready.
• Before input errors: a missing nested resolver may declare modifiers, so selection inputs are checked only once every reached document is present.

When the root itself is missing, the request has referenced_from and original_reference equal to the root’s id. provide_requested_json(&request, json) is provide_json(request.resource_id.as_str(), json) without re-parsing the id, and fails with InvalidDocument when the request asks for a module. Without expressions, every request has kind: DocumentKind::Json.

#Loader patterns

#Everything up front

When all documents are known in advance, such as a design system embedded in a binary or a WebAssembly build, provide them in one batch. The batch is all-or-nothing and bumps the revision once.

use dxn_dtcg::{DtcgError, DtcgRuntime};

fn embedded_runtime() -> Result<DtcgRuntime, DtcgError> {
    let mut runtime = DtcgRuntime::from_json(
        "acme.resolver.json",
        include_str!("../design/acme.resolver.json"),
    )?;
    runtime.provide_json_batch([
        ("tokens/foundation.json", include_str!("../design/tokens/foundation.json")),
        ("tokens/dark.json", include_str!("../design/tokens/dark.json")),
    ])?;
    Ok(runtime)
}

fn main() -> Result<(), DtcgError> {
    let runtime = embedded_runtime()?;
    assert!(runtime.resolve(&[])?.is_ready());
    Ok(())
}

#File system

A synchronous loader that keeps reads inside one directory. It touches the file system, so it needs a design directory with the documents:

use std::{
    error::Error,
    fs,
    path::{Path, PathBuf},
};

use dxn_dtcg::{DocumentRequest, DtcgRuntime, LocalReferencePolicy, Resolution, ResolvedTokens};

struct FileLoader {
    base: PathBuf,
}

impl FileLoader {
    fn new(base: impl AsRef<Path>) -> std::io::Result<Self> {
        Ok(Self { base: fs::canonicalize(base)? })
    }

    fn read(&self, request: &DocumentRequest) -> Result<String, Box<dyn Error>> {
        // Reject absolute paths, drive letters, URI schemes and backslashes.
        LocalReferencePolicy::check(&request.referenced_from, &request.original_reference)?;
        let path = fs::canonicalize(self.base.join(request.resource_id.as_str()))?;
        // Also refuse symbolic links that lead out of the token directory.
        if !path.starts_with(&self.base) {
            return Err(format!("{} is outside the token directory", request.resource_id).into());
        }
        Ok(fs::read_to_string(path)?)
    }

    fn resolve(
        &self,
        runtime: &mut DtcgRuntime,
        selection: &[(&str, &str)],
    ) -> Result<ResolvedTokens, Box<dyn Error>> {
        loop {
            match runtime.resolve(selection)? {
                Resolution::Ready(tokens) => return Ok(tokens),
                Resolution::Missing(requests) => {
                    for request in &requests {
                        runtime.provide_requested_json(request, self.read(request)?)?;
                    }
                }
            }
        }
    }
}

fn main() -> Result<(), Box<dyn Error>> {
    let loader = FileLoader::new("design")?;
    let mut runtime = DtcgRuntime::default();
    runtime.set_root("acme.resolver.json")?;
    let tokens = loader.resolve(&mut runtime, &[("theme", "dark")])?;
    println!("{} tokens", tokens.len());
    Ok(())
}

#HTTP and other asynchronous sources

The runtime has no async API because resolution is CPU-only and fast. Run the loop in your async code and await the fetches between calls. Fetching all requests of a round concurrently saves round trips. In a browser (wasm32-unknown-unknown) the same loop works with fetch through wasm-bindgen. This snippet touches the network once you replace the stub:

use std::error::Error;

use dxn_dtcg::{DtcgRuntime, Resolution, ResolvedTokens};

async fn fetch_text(url: &str) -> Result<String, Box<dyn Error>> {
    // Use your HTTP client here, for example `reqwest::get(url).await?.text().await?`.
    unimplemented!("GET {url}")
}

async fn resolve_remote(
    base_url: &str,
    selection: &[(&str, &str)],
) -> Result<ResolvedTokens, Box<dyn Error>> {
    let mut runtime = DtcgRuntime::default();
    runtime.set_root("acme.resolver.json")?;
    loop {
        match runtime.resolve(selection)? {
            Resolution::Ready(tokens) => return Ok(tokens),
            Resolution::Missing(requests) => {
                for request in requests {
                    let text = fetch_text(&format!("{base_url}/{}", request.resource_id)).await?;
                    runtime.provide_requested_json(&request, text)?;
                }
            }
        }
    }
}

fn main() {
    let _ = resolve_remote("https://design.example.com/tokens", &[]);
}

#Preloading every context

To fetch everything any selection could need, for an offline cache or a bundle, drive the loop with validate_complete instead of resolve:

use std::collections::HashMap;

use dxn_dtcg::{DtcgError, DtcgRuntime, Resolution};

fn preload_all(runtime: &mut DtcgRuntime, files: &HashMap<&str, &str>) -> Result<(), DtcgError> {
    loop {
        match runtime.validate_complete()? {
            Resolution::Ready(()) => return Ok(()),
            Resolution::Missing(requests) => {
                for request in requests {
                    runtime.provide_requested_json(&request, files[request.resource_id.as_str()])?;
                }
            }
        }
    }
}

fn main() -> Result<(), DtcgError> {
    let files = HashMap::from([
        ("light.json", r#"{ "bg": { "$type": "number", "$value": 0 } }"#),
        ("dark.json", r#"{ "bg": { "$type": "number", "$value": 1 } }"#),
    ]);
    let mut runtime = DtcgRuntime::from_json(
        "r.json",
        r##"{
          "version": "2025.10",
          "modifiers": { "theme": { "default": "light", "contexts": {
            "light": [{ "$ref": "light.json" }], "dark": [{ "$ref": "dark.json" }]
          } } },
          "resolutionOrder": [{ "$ref": "#/modifiers/theme" }]
        }"##,
    )?;
    preload_all(&mut runtime, &files)?;
    assert!(runtime.contains_document("dark.json")?); // fetched although "light" is the default
    Ok(())
}

#Module documents

With the expressions feature and RuntimeOptions::with_expressions, a token document may declare JavaScript and TypeScript modules at its root ($extensions["dxn.dtcg.expressions"].modules), and token expressions import them (modules in token documents). Module text loads through the same protocol: when an expression imports a declared module the catalog lacks, resolution returns Missing with a DocumentRequest of kind: DocumentKind::Module. A loader that serves both kinds dispatches on kind:

use dxn_dtcg::{DocumentKind, DtcgError, DtcgRuntime, Resolution, ResolvedTokens};

fn resolve_all(
    runtime: &mut DtcgRuntime,
    load: impl Fn(&str) -> String,
) -> Result<ResolvedTokens, DtcgError> {
    loop {
        match runtime.resolve(&[])? {
            Resolution::Ready(tokens) => return Ok(tokens),
            Resolution::Missing(requests) => {
                for request in requests {
                    let text = load(request.resource_id.as_str());
                    match request.kind {
                        DocumentKind::Module => runtime.provide_requested_module(&request, text)?,
                        _ => runtime.provide_requested_json(&request, text)?,
                    }
                }
            }
        }
    }
}

• A module’s relative imports (./units.ts) resolve against its resource id with the same resource id rules and are requested in later rounds, until the whole static import graph is loaded; referenced_from is then the importing module.
• The language follows the extension: .js and .mjs are JavaScript, .ts and .mts TypeScript; any other extension fails with DtcgError::Expression.
• Module text is not checked when it is provided, only when a resolution imports it.
• Only modules that expressions reach are requested. validate_complete requests the modules the sources of every context declare, and what those import, so a catalog it accepts resolves in every context. The preview APIs request modules like resolve.
• Module documents are a separate count: one resolution loads at most max_documents of them, besides its JSON documents (LimitExceeded { kind: "documents" }).
• Kinds never mix. An entry is either JSON or a module. Providing JSON at an id that holds a module, a module at an id that holds JSON, or answering a request with the other kind (provide_requested_json for a module request, or the reverse) fails with InvalidDocument and leaves the catalog unchanged. A JSON reference (a resolver source, a $ref) to a module document fails with InvalidDocument; a module declaration or import that reaches a JSON document fails with DtcgError::Expression.
• Module edits are not incremental. Affected resolution does not follow module edits: after replace_module, resolve fully.

The module methods exist in every build. Without the feature or the option, nothing requests or reads modules.

#Catalog API

A mutation that fails (invalid id, malformed JSON, duplicate in a batch, unknown document, kind mismatch) changes nothing, revision included. The revision increases on every successful mutation, even if the new text equals the old. Debug for DtcgRuntime prints the root, options, revision and document ids, never document text.

#Caching and sharing

resolve recomputes from the catalog every time. In a UI, cache by catalog revision plus normalized selection; the cookbook has a complete cache. To skip downstream work, compare ResolvedTokens::fingerprint() values.

DtcgRuntime is Send + Sync and every reading method takes &self, so an Arc<RwLock<DtcgRuntime>> allows concurrent resolution with exclusive updates. Cloning is cheap since documents are reference-counted, so clone, modify and swap works too.

#Path safety

Resource ids come from documents, which may not be trustworthy. The id rules prevent the usual escapes: no leading /, no \, no :, no .. above the root, and no Windows-special names. Joining an id onto a base directory with Path::join therefore stays inside it on Windows and Unix, symbolic links aside. A reference that breaks the rules fails with InvalidResourceId and never becomes a DocumentRequest.

A loader that reads files should still check each request:

1. LocalReferencePolicy::check(&request.referenced_from, &request.original_reference) rejects absolute paths, backslashes and anything starting like a URI scheme. The runtime does not produce such requests, so this is a second line of defense.
2. To also refuse symlinks that lead elsewhere, canonicalize the joined path and check it is under the canonical base directory, as the file loader above does.

LocalReferencePolicy is an opt-in document-reference policy for hosts that load documents from a local tree: a resolver may only reference relative local documents, and absolute paths, backslashes and URI schemes are rejected before any document is requested. from_json and resolve do not apply it; apply it in your loader. LocalReferencePolicy::check_resolver(resource, &value) applies check to every $ref in a resolver’s sets.*.sources and modifiers.*.contexts.*; inline resolutionOrder steps and nested resolvers are not walked. Both fail with InvalidReference.

Next: Typed projection, or the Cookbook.