Skip to content

datavzrd: auto-generated tool wrapper (issue #1966) #1976

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bgruening merged 6 commits into from
Sep 1, 2026
Merged

datavzrd: auto-generated tool wrapper (issue #1966) #1976

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d0ef6c0
Add datavzrd render tool with auto-config, user configs, and tests
Aug 31, 2026
7de21e5
Remove redundant webview_url param, clarify auto-config help, set pro…
Aug 31, 2026
cccd766
Address review: fix extensions, dataset naming, required_files, tests
Aug 31, 2026
9f3cf71
Address review feedback and fix CI failures
Sep 1, 2026
2603814
Update tools/datavzrd/datavzrd_render.xml
SaimMomin12 Sep 1, 2026
bdbeb7c
Update tools/datavzrd/macros.xml
SaimMomin12 Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions tools/datavzrd/.shed.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
categories:
- Visualization
description: Render visual and interactive HTML reports from tabular data with datavzrd
homepage_url: https://github.com/datavzrd/datavzrd
long_description: |
Datavzrd creates visual and interactive HTML reports from collections of tabular
data (CSV, TSV, JSON or Parquet). Reports include automatically generated vega-lite
plots per column, searching, sorting and filtering, and can be fully customized via a
YAML configuration file supporting custom plots, heatmaps, linkouts and inter-table
linking.
name: datavzrd
owner: bgruening
remote_repository_url: https://github.com/bgruening/galaxytools/tree/master/tools/datavzrd
type: unrestricted
94 changes: 94 additions & 0 deletions tools/datavzrd/datavzrd_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""Helper script to adapt datavzrd report configuration files for Galaxy.

Subcommands:
rewrite: rewrite the dataset paths of a config file to point to the
Galaxy input files. Datasets are matched to the input files
in the order in which they appear in the config file.
set: set one or more top-level key-value pairs in a config file.
"""

import argparse
import sys

import yaml


def load_config(path):
with open(path, "r", encoding="utf-8") as fh:
return yaml.safe_load(fh) or {}


def dump_config(config, path):
with open(path, "w", encoding="utf-8") as fh:
yaml.safe_dump(
config,
fh,
default_flow_style=False,
sort_keys=False,
allow_unicode=True,
)


def command_rewrite(args):
config = load_config(args.config)
inputs = [value for value in args.inputs.split(",") if value]
datasets = config.get("datasets") or {}
if len(datasets) != len(inputs):
sys.exit(
"Error: the config file defines %d dataset(s) (%s), but %d input "
"file(s) were provided. Please provide exactly one input file "
"per dataset, in the order in which the datasets appear in the "
"config file." % (len(datasets), ", ".join(datasets), len(inputs))
)
for (name, dataset), path in zip(datasets.items(), inputs):
dataset["path"] = path
dump_config(config, args.output)


def command_set(args):
config = load_config(args.config)
for assignment in args.assignments:
if "=" not in assignment:
sys.exit("Error: invalid assignment '%s', expected key=value." % assignment)
key, value = assignment.split("=", 1)
config[key] = yaml.safe_load(value)
dump_config(config, args.output)


def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__)
subparsers = parser.add_subparsers(dest="command", required=True)

parser_rewrite = subparsers.add_parser(
"rewrite", help="rewrite dataset paths to point to Galaxy input files"
)
parser_rewrite.add_argument("config", help="path of the datavzrd config file")
parser_rewrite.add_argument(
"--inputs", required=True, help="comma-separated list of input files"
)
parser_rewrite.add_argument(
"--output", required=True, help="output path of the rewritten config file"
)
parser_rewrite.set_defaults(func=command_rewrite)

parser_set = subparsers.add_parser(
"set", help="set top-level key-value pairs in a config file"
)
parser_set.add_argument("config", help="path of the datavzrd config file")
parser_set.add_argument(
"--output", required=True, help="output path of the updated config file"
)
parser_set.add_argument(
"assignments",
nargs="+",
help="key=value pairs; values are parsed as YAML scalars",
)
parser_set.set_defaults(func=command_set)

args = parser.parse_args(argv)
args.func(args)


if __name__ == "__main__":
main()
252 changes: 252 additions & 0 deletions tools/datavzrd/datavzrd_render.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
<tool id="datavzrd_render" name="datavzrd" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="@PROFILE@">
<description>Render visual and interactive HTML reports from tabular data</description>
<macros>
<import>macros.xml</import>
</macros>
<expand macro="render_requirements"/>
Comment thread
SaimMomin12 marked this conversation as resolved.
<required_files>
<include path="datavzrd_config.py"/>
</required_files>
<expand macro="version_command"/>
<command detect_errors="aggressive"><![CDATA[
## Symlink the input tables into the working directory using safe names
## derived from the dataset names, so that they can be referenced from
## the datavzrd configuration file.
#import re
#set $data_files = []
#set $suggest_args = []
#set $used = []
#for $i, $data_file in enumerate($input_data):
#if $data_file.is_of_type('json'):
#set $ext = 'json'
#elif $data_file.is_of_type('parquet'):
#set $ext = 'parquet'
#elif $data_file.is_of_type('csv'):
#set $ext = 'csv'
#else:
#set $ext = 'tsv'
#end if
#set $stem = re.sub('[^\w\-]', '_', str($data_file.element_identifier).rsplit('.', 1)[0])
#if $stem in $used:
#set $stem = $stem + '_' + str($i)
#end if
#silent $used.append($stem)
#set $fname = $stem + '.' + $ext
#silent $data_files.append($fname)
ln -s '${data_file}' '${fname}' &&
#if $ext == 'tsv':
#silent $suggest_args.append("-f '" + $fname + "' -s $'\\t'")
#else:
#silent $suggest_args.append("-f '" + $fname + "' -s ','")
#end if
Comment thread
SaimMomin12 marked this conversation as resolved.
#end for

## Build the datavzrd configuration file
#if $config_mode.config_mode_selector == 'auto':
## Let datavzrd suggest a configuration from the given input tables
datavzrd suggest #echo ' '.join($suggest_args)# --name '${config_mode.report_name}' > config.yaml &&
#else:
## Rewrite the dataset paths of the user-provided configuration file
## to point to the input tables of this job
python '$__tool_directory__/datavzrd_config.py' rewrite '${config_mode.config_file}' --inputs '#echo ','.join($data_files)#' --output config.yaml &&
#end if

## Apply advanced options to the configuration file
#set $adv_args = []
#if $adv.max_in_memory_rows:
#silent $adv_args.append('max-in-memory-rows=' + str($adv.max_in_memory_rows))
#end if
#if $adv_args:
python '$__tool_directory__/datavzrd_config.py' set config.yaml --output config.yaml #echo ' '.join($adv_args)# &&
#end if

## Render the report
datavzrd config.yaml --output report_output --overwrite-output &&

## Collect the output
#if $output_format == 'html':
mkdir -p '${output_html.files_path}' &&
mv report_output/index.html '${output_html}' &&
mv report_output/* '${output_html.files_path}/'
#else:
cd report_output &&
zip -q -r ../report.zip . &&
cd .. &&
mv report.zip '${output_zip}'
#end if
]]></command>
<inputs>
<param name="input_data" type="data" format="csv,tsv,tabular,json,parquet" multiple="true" label="Input table(s)" help="One or more tabular datasets (CSV, TSV, JSON or Parquet) to include in the report. When using a custom configuration file, provide the tables in the same order as the datasets defined in that configuration file."/>
<conditional name="config_mode">
<param name="config_mode_selector" type="select" label="Configuration">
<option value="auto" selected="true">Auto-generate the configuration from the input tables</option>
<option value="provide">Provide a datavzrd configuration file</option>
</param>
<when value="auto">
<expand macro="report_name"/>
</when>
<when value="provide">
<param name="config_file" type="data" format="yaml" label="datavzrd configuration file" help="A datavzrd YAML configuration file. The dataset 'path' entries are rewritten to the input tables above (matched in the order in which the datasets appear in the configuration file), so any path can be used there. Note that relative references to external files (spec-path, script-path, custom-path, render-img paths) are not supported - use the inline 'spec' or 'custom' fields instead."/>
</when>
</conditional>
<param name="output_format" type="select" label="Output format">
<option value="html" selected="true">Interactive HTML report (displayed in Galaxy)</option>
<option value="zip">Zipped report directory</option>
</param>
<section name="adv" title="Advanced options" expanded="false">
<param name="max_in_memory_rows" type="integer" value="20000" min="1" label="Maximum in-memory rows" help="Maximum number of rows of a table that are rendered into a single page (default: 20000). Tables with more rows are split into searchable pages."/>
</section>
</inputs>
<outputs>
<data name="output_html" format="html" label="${tool.name} on ${on_string}: interactive report">
<filter>output_format == 'html'</filter>
</data>
<data name="output_zip" format="zip" label="${tool.name} on ${on_string}: zipped report">
<filter>output_format == 'zip'</filter>
</data>
</outputs>
<tests>
<!-- Test 01: auto-generated configuration from a single CSV -->
<test expect_num_outputs="1">
<param name="input_data" value="oscars.csv" ftype="csv"/>
<conditional name="config_mode">
<param name="config_mode_selector" value="auto"/>
<param name="report_name" value="Oscars Report"/>
</conditional>
<param name="output_format" value="html"/>
<output name="output_html" ftype="html">
<assert_contents>
<has_text text="&lt;script src=&quot;./static/bundle.js&quot;&gt;"/>
<has_text text="const name = 'Oscars Report';"/>
<has_text text="const views = {&quot;oscars&quot;:null};"/>
<has_text text="const table = null;"/>
<not_has_text text="movies"/>
</assert_contents>
</output>
<assert_command>
<has_text text="datavzrd suggest -f 'oscars.csv' -s ',' --name 'Oscars Report' &gt; config.yaml"/>
</assert_command>
</test>
<!-- Test 02: auto-generated configuration from multiple CSVs -->
<test expect_num_outputs="1">
<param name="input_data" value="oscars.csv,movies.csv" ftype="csv"/>
<conditional name="config_mode">
<param name="config_mode_selector" value="auto"/>
</conditional>
<param name="output_format" value="html"/>
<output name="output_html" ftype="html">
<assert_contents>
<has_text text="const name = 'Datavzrd Report';"/>
<has_text text="&quot;oscars&quot;:null"/>
<has_text text="&quot;movies&quot;:null"/>
<has_text text="&lt;script src=&quot;./static/bundle.js&quot;&gt;"/>
</assert_contents>
</output>
<assert_command>
<has_text text="-f 'oscars.csv' -s ',' -f 'movies.csv' -s ','"/>
</assert_command>
</test>
<!-- Test 03: user-provided configuration file with advanced options -->
<test expect_num_outputs="1">
<param name="input_data" value="oscars.csv,movies.csv" ftype="csv"/>

@bgruening bgruening Aug 31, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those names here need to match exactly the names in the config file ...

I think this deserves an example in the help section. Also raising the fact that those names need to be unique.

<conditional name="config_mode">
<param name="config_mode_selector" value="provide"/>
<param name="config_file" value="example-config.yaml" ftype="yaml"/>
</conditional>
<section name="adv">
<param name="max_in_memory_rows" value="500"/>
</section>
<param name="output_format" value="html"/>
<output name="output_html" ftype="html">
<assert_contents>
<has_text text="const name = 'My oscar report';"/>
<has_text text="&quot;oscars&quot;:"/>
<has_text text="&quot;movies&quot;:null"/>
<has_text text="&quot;oscar-plot&quot;:"/>
<has_text text="## All winning oscars beginning in the year 1929."/>
<has_text text="&lt;script src=&quot;./static/bundle.js&quot;&gt;"/>
</assert_contents>
</output>
<assert_command>
<has_text text="datavzrd_config.py' rewrite"/>
<has_text text="max-in-memory-rows=500"/>
</assert_command>
</test>
<!-- Test 04: TSV input with auto-generated configuration -->
<test expect_num_outputs="1">
<param name="input_data" value="genes.tsv" ftype="tsv"/>
<conditional name="config_mode">
<param name="config_mode_selector" value="auto"/>
<param name="report_name" value="Gene Report"/>
</conditional>
<param name="output_format" value="html"/>
<output name="output_html" ftype="html">
<assert_contents>
<has_text text="const name = 'Gene Report';"/>
<has_text text="const views = {&quot;genes&quot;:null};"/>
<has_text text="&lt;script src=&quot;./static/bundle.js&quot;&gt;"/>
</assert_contents>
</output>
<assert_command>
<has_text text="-s $'\t'"/>
</assert_command>
</test>
<!-- Test 05: zipped report output -->
<test expect_num_outputs="1">
<param name="input_data" value="oscars.csv" ftype="csv"/>
<conditional name="config_mode">
<param name="config_mode_selector" value="auto"/>
<param name="report_name" value="Oscars Report"/>
</conditional>
<param name="output_format" value="zip"/>
<assert_command>
<has_text text="zip -q -r ../report.zip"/>
</assert_command>
<output name="output_zip" ftype="zip">
<assert_contents>
<has_archive_member path="index.html"/>
<has_archive_member path="static/bundle.js"/>
<has_archive_member path="oscars/index_1.html"/>
</assert_contents>
</output>
</test>
</tests>
<help format="markdown"><![CDATA[
# Datavzrd

Datavzrd creates visual and interactive HTML reports from collections of tabular data (CSV, TSV, JSON or Parquet). Reports include automatically generated plots per column and can be fully customized via a YAML configuration file.

## Configuration modes

1. **Auto-generate the configuration from the input tables**: datavzrd inspects each input table and builds the configuration on the fly, with one table view per input file. Each view provides per-column plots, searching, sorting and filtering. This is the easiest way to get an interactive report - no configuration file is needed.

2. **Provide a datavzrd configuration file**: supply your own YAML configuration file for full control over report layout, column rendering (ticks, bars, heatmaps, pills, custom plots), linkouts and inter-table linking. See the datavzrd documentation at https://datavzrd.github.io/docs/configuration.html for a full description of the configuration format.

When a configuration file is provided, the `path` of each dataset in its `datasets` section is rewritten to the input tables of this job: datasets are matched to the input tables **in the order in which they appear in the configuration file**, so provide exactly one input table per dataset, in that order.

In addition, the **name of every dataset and view in the configuration file must exactly match the (sanitized) file name of the corresponding input table, and must be unique across the whole configuration**, because datavzrd uses these names as directory names inside the rendered report. The tool names each input table after its file name (with non-alphanumeric characters replaced by underscores, and the extension changed to the actual format, e.g. `.csv` or `.tsv`).

Example: for a configuration file with

```yaml
views:
oscars:
dataset: oscars
render-table:
...
datasets:
oscars:
path: anything.csv
```

select the corresponding input tables such that they are named `oscars.csv` etc., e.g. by renaming the datasets before running the tool. Note that the `path` values themselves are ignored (they are rewritten to the input tables), only the dataset and view names must match the input table names.

Relative references to external files in the configuration (such as `spec-path`, `script-path`, `custom-path` or `render-img` paths) are not supported by this wrapper - use the inline `spec` or `custom` fields instead.

## Output

The interactive HTML report can be displayed directly in Galaxy (choose *Interactive HTML report*). All report files (tables, plots and data) are stored alongside the displayed `index.html`. Alternatively, the whole report directory can be returned as a zip archive, e.g. for sharing or archiving of large reports.

]]></help>
<expand macro="citations"/>
</tool>
26 changes: 26 additions & 0 deletions tools/datavzrd/macros.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
<macros>
<token name="@TOOL_VERSION@">2.72.0</token>
<token name="@VERSION_SUFFIX@">0</token>
<token name="@PROFILE@">25.0</token>
<xml name="render_requirements">
<requirements>
<requirement type="package" version="@TOOL_VERSION@">datavzrd</requirement>
<requirement type="package" version="3.13">python</requirement>
<requirement type="package" version="6.0.3">pyyaml</requirement>
<requirement type="package" version="3.0">zip</requirement>
</requirements>
</xml>
<xml name="version_command">
<version_command>datavzrd --version</version_command>
</xml>
<xml name="report_name">
<param name="report_name" type="text" value="Datavzrd Report" label="Report name" help="Name of the generated report, shown in the report header">
<validator type="regex" message="Report name must not contain single quotes, backslashes or newline characters">^[^'\\\n\r]+$</validator>
</param>
</xml>
<xml name="citations">
<citations>
<citation type="doi">10.1371/journal.pone.0323079</citation>
</citations>
</xml>
</macros>
Loading
Loading