Skip to content

Cache template sources instead of composed templates #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RobAndrewHurst wants to merge 9 commits into
base: recursive-templates
Choose a base branch
from
Open

Cache template sources instead of composed templates #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e024f4a
workspace template src caching
RobAndrewHurst Jul 15, 2026
3738d6f
Merge branch 'recursive-templates' into src-map
RobAndrewHurst Jul 16, 2026
4e60423
Merge branch 'recursive-templates' into src-map
RobAndrewHurst Jul 17, 2026
6b69c0f
cache scopes
RobAndrewHurst Jul 17, 2026
29b571f
Merge branch 'recursive-templates' into src-map
RobAndrewHurst Jul 17, 2026
0c48c44
Refactor workspace caching and scope handling; improve documentation …
dbauszus-glx Jul 20, 2026
a69ba41
correct workspace caching
RobAndrewHurst Jul 20, 2026
ffbc43b
update srcMap envReplace
RobAndrewHurst Jul 20, 2026
406155e
update getTemplate and srcMap
RobAndrewHurst Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 21 additions & 167 deletions mod/workspace/_workspace.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,11 @@ The workspace object defines the mapp resources available in an XYZ instance.
*/

import { createHash } from 'node:crypto';
import envReplace from '../utils/envReplace.js';
import logger from '../utils/logger.js';
import workspaceCache from './cache.js';
import generateScopes from './generateScopes.js';
import getLayer from './getLayer.js';
import getLocale from './getLocale.js';
import getTemplate from './getTemplate.js';

const keyMethods = {
layer,
Expand All @@ -41,6 +40,9 @@ const keyMethods = {

let workspace;

// Scope generation composes every locale, layer, and nested template. Cache the resulting promise by workspace identity so concurrent requests share the same traversal and a refreshed workspace invalidates the result automatically.
const workspaceScopes = new WeakMap();

/**
@function getKeyMethod
@async
Expand Down Expand Up @@ -264,31 +266,27 @@ async function scopes(req, res) {
return;
}

const cachedWorkspace = await workspaceCache(true);
const cachedWorkspace = await workspaceCache();

// TODO test workspace without locales property. Should the scopes method still return the scopes of the templates in the workspace.templates object?
for (const localeKey of Object.keys(cachedWorkspace.locales)) {
const locale = await getLocale({
locale: localeKey,
layers: true,
user: { roles: true },
});
await nestedLocales(locale, { roles: true });
if (Array.isArray(cachedWorkspace.cachedScopes)) {
res.send([...cachedWorkspace.cachedScopes]);
return;
}

const scopesStringsSet = new Set();
let scopesPromise = workspaceScopes.get(cachedWorkspace);

cachedWorkspace.scopes.forEach((scope) => {
if (!Array.isArray(scope)) return;
scopesStringsSet.add(scope.filter(Boolean).join('.'));
});
if (!scopesPromise) {
scopesPromise = generateScopes(cachedWorkspace);
workspaceScopes.set(cachedWorkspace, scopesPromise);

const scopesArray = Array.from(scopesStringsSet)
.filter(Boolean)
.sort((a, b) => a.localeCompare(b));
// Do not retain a rejected promise for the lifetime of the workspace.
scopesPromise.catch(() => workspaceScopes.delete(cachedWorkspace));
}

const scopesArray = await scopesPromise;

if (req.params.tree) {
scopesArrayToTree(res, scopesStringsSet);
scopesArrayToTree(res, scopesArray);
return;
}

Expand All @@ -302,12 +300,12 @@ async function scopes(req, res) {
The scopesArrayToTree method converts an array of scopes strings into a tree structure.

@param {res} res HTTP response.
@param {Set} scopesStringsSet Set of scopes strings.
@param {Array} scopesArray Array of scopes strings.
*/
function scopesArrayToTree(res, scopesStringsSet) {
function scopesArrayToTree(res, scopesArray) {
const scopesTree = {};

for (const scope of scopesStringsSet) {
for (const scope of scopesArray) {
if (scope === '') continue;

const rolesArr = scope.split('.');
Expand Down Expand Up @@ -560,150 +558,6 @@ function processTestResults(testConfig) {
return results;
}

/**
@function cacheWorkspaceTemplates
@async

@description
Build-time workspace cache method used by the workspace generation script. The
method lives in the workspace module so it can reuse the same workspace cache and
getTemplate behaviour as the API without going through an HTTP-only response
path.

The workspace is parsed for every non-module src property. All matching source
templates found in the current pass are cached, then only those newly cached
templates are parsed for additional src properties. This repeats until no new src
templates are discovered.

Each unique src is fetched once (de-duplicated via srcMap). References within a
circular src path are skipped to prevent infinite expansion. Module templates
keep their src because render functions cannot be serialised into JSON.

@returns {Promise<Object|Error>} Generated workspace or an Error.
*/
export async function cacheWorkspaceTemplates() {
workspace = await workspaceCache(true);

const errors = [];
const srcMap = new Map();
const scannedSrc = new Set();
const queue = [{ obj: workspace, srcPath: new Set() }];

while (queue.length) {
const srcRefs = new Map();

for (const item of queue.splice(0)) {
collectSrcRefs(item.obj, item.srcPath, srcRefs, new WeakSet());
}

for (const [src, refs] of srcRefs) {
await inlineSrc(src, refs, srcMap, scannedSrc, errors, queue);
}
}

if (errors.length) {
return new Error(errors.join('\n'));
}

return workspace;
}

/**
@function inlineSrc
@async

@description
Loads the template for a single src, inlines it into every referencing object,
and queues the newly populated objects for the next BFS pass. Refs that already
contain this src in their branch path are skipped to prevent circular expansion.

@param {string} src Normalised source reference.
@param {Array} refs Objects that reference this src with their current srcPath.
@param {Map} srcMap Cache of already-loaded templates for this generation run.
@param {Set} scannedSrc Sources whose children have already been queued.
@param {Array} errors Accumulated error messages.
@param {Array} queue BFS queue for the next pass.
*/
async function inlineSrc(src, refs, srcMap, scannedSrc, errors, queue) {
// Skip refs where this src already appears in the branch path (circular).
const refsToExpand = refs.filter((ref) => !ref.srcPath.has(src));
if (!refsToExpand.length) return;

let template = srcMap.get(src);
if (!template) {
// getTemplate caches anonymous src objects under the src key; remove
// that temporary entry unless the key already existed before this run.
const hadTemplate = Object.hasOwn(workspace.templates, src);
template = await getTemplate({ src });
if (!hadTemplate) delete workspace.templates[src];
srcMap.set(src, template);
}

if (template instanceof Error) {
errors.push(template.message);
return;
}

for (const ref of refsToExpand) {
Object.assign(ref.obj, template);
delete ref.obj.src;
}

// Only scan a src once for nested src values; repeated references in the
// same pass are already inlined above.
if (!scannedSrc.has(src)) {
scannedSrc.add(src);
queue.push(
...refsToExpand.map((ref) => ({
obj: ref.obj,
srcPath: new Set([...ref.srcPath, src]),
})),
);
}
}

/**
@function collectSrcRefs

@description
Walks an object tree and groups non-module src references by normalized src
value. The object that owns the src is retained so the cached template can be
assigned back to the same location in the generated workspace.

@param {Object|Array} obj Object or array to inspect.
@param {Set<String>} srcPath Source chain for circular reference detection.
@param {Map<String, Array<Object>>} srcRefs Grouped source references.
@param {WeakSet<Object>} objects Objects already visited during this scan.
*/
function collectSrcRefs(obj, srcPath, srcRefs, objects) {
if (!obj || typeof obj !== 'object') return;

// Avoid walking the same object twice in one scan if objects are shared.
if (objects.has(obj)) return;
objects.add(obj);

if (Array.isArray(obj)) {
obj.forEach((item) => collectSrcRefs(item, srcPath, srcRefs, objects));
return;
}

if (typeof obj.src === 'string') {
obj.src = envReplace(obj.src);

// Module templates stay as runtime imports; skip them entirely so nested
// src properties inside a module object are not inlined either.
if (obj.module) return;

const refs = srcRefs.get(obj.src) || [];
refs.push({ obj, srcPath });
srcRefs.set(obj.src, refs);
}

Object.values(obj).forEach((value) =>
collectSrcRefs(value, srcPath, srcRefs, objects),
);
}

/**
@function assignChecksum

Expand Down
13 changes: 12 additions & 1 deletion mod/workspace/cache.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Default templates can be overwritten in the workspace or by providing a CUSTOM_T
import getFrom from '../provider/getFrom.js';
import logger from '../utils/logger.js';
import merge from '../utils/merge.js';
import { cacheWorkspaceSources, srcMap } from './srcMap.js';

let cache = null;
let timestamp = Infinity;
Expand All @@ -26,7 +27,7 @@ The method checks whether the module scope variable cache has been populated.

The timestamp set by cacheWorkspace is checked against the current time. The [workspace] cache will be invalidated if the difference exceeds the WORKSPACE_AGE xyzEnvironment variable.

Setting the WORKSPACE_AGE to 0 is not recommended as this could cause the cache to be flushed while a request is passed through the XYZ API. A layer query processed by the [Query API module]{@link module:/query~layerQuery} will request the layer and associated locale which could be defined in remote templates. Each request to the [Workspace API getTemplate]{@link module:/workspace/getTemplate~getTemplate} method for the locale, layer, and query templates will call the checkWorkspaceCache method which will cause the workspace to be flushed and templates previously cached from their src no longer available.
Setting the WORKSPACE_AGE to 0 is not recommended because the source promise map is flushed whenever the workspace is rebuilt. Constantly replacing the workspace prevents source responses from being reused between requests.

The cacheWorkspace method is called if the cache is invalid.

Expand Down Expand Up @@ -151,5 +152,15 @@ async function cacheWorkspace() {

cache = workspace;

// Cached source responses may be stale after the workspace is rebuilt.
srcMap.clear();

cacheWorkspaceSources(cache).then((result) => {
if (result instanceof Error) {
// TODO needs test
console.error(result);
}
});

return workspace;
}
Loading