Skip to content

Commit 30395b4

Browse files
erazocarerazocar
authored
Merge pull request #4 from uihilab/dev-branch
Updates on HydroCompute from dev branch and HydroBlox
2 parents 4bbaf89 + 876c8d8 commit 30395b4

39 files changed

+7331
-644
lines changed

‎README.md‎

Lines changed: 124 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
[**Link to documentation**](https://uihilab.github.io/HydroCompute/)
66

77
* [Introduction](https://github.com/uihilab/HydroCompute#Introduction)
8+
* [Database Integration](https://github.com/uihilab/HydroCompute#Database-Integration)
89
* [How to Use](https://github.com/uihilab/HydroCompute#How-to-Use)
910
* [Expansions and Test Cases](https://github.com/uihilab/HydroCompute#Expansions-and-Test-Cases)
1011
* [Community](https://github.com/uihilab/HydroCompute#Community)
@@ -15,7 +16,14 @@
1516
* [References](#references)
1617

1718
## Introduction
18-
This work introduces HydroCompute, a computational library for hydrology and environmental sciences that runs on the client side. It employes 4 different engines, 3 main computations and 1 for peer-to-peer connection. The library has been developed using ES6 standards and the most recent available APIs for WebAssembly, WebGPU, WebRTC, and the Web Workers specifications.
19+
This work introduces hydroCompute, a computational library for hydrology and environmental sciences that runs on the client side. It employs distributed computing environments including **JavaScript** (Native & HydroLang), **Python** (Pyodide), **R** (WebR), and **WebGPU**, along with **WebRTC** for peer-to-peer connectivity. The library has been developed using ES6 standards and the most recent available APIs for WebAssembly, WebGPU, WebRTC, and the Web Workers specifications.
20+
21+
## Database Integration
22+
HydroCompute now integrates **HydroComputeDB**, a robust wrapper around IndexedDB, to manage data persistence across sessions and workers. This ensures that large datasets and simulation results are stored efficiently in the browser without blocking the main thread. The database manages several stores:
23+
* `settings`: Stores configuration and code snippets for dynamic execution.
24+
* `workflowStates`: Manages the state of complex, multi-step workflows.
25+
* `results`: Stores the output of computations, linked by execution IDs.
26+
* `wasmModules`: Caches compiled WebAssembly modules for faster loading.
1927

2028
## How to Use
2129
Please download the library and run `index.html`. If a new html file should be created, the library must be onloaded onto the file as a script
@@ -30,21 +38,18 @@ Please download the library and run `index.html`. If a new html file should be c
3038
The library is loaded into an HTML web app by declaring either it as a window object when loading, or as a single instance run as follows:
3139

3240
```javascript
33-
const compute = new hydrocompute('engineName');
41+
const compute = new hydroCompute('engineName');
3442
```
3543

36-
When instantiated if no specific engines are passed into the constructor, the library will default to run using the functions within the JavaScript engine. The available engines along with existing code and how to develop more functions for usage in the library can be found in the following links:
37-
38-
* [WebAssembly](https://github.com/uihilab/HydroCompute/tree/master/src/wasm): Available C and AssemblyScript bindings.
39-
* [JavaScript](https://github.com/uihilab/HydroCompute/tree/master/src/javascript): Available as native JavaScript object.
40-
* [WebGPU](https://github.com/uihilab/HydroCompute/tree/master/src/webgpu): GLSL code onloaded as strings in a JavaScript object.
44+
When instantiated if no specific engines are passed into the constructor, the library will default to run using the functions within the JavaScript engine.
4145

42-
### Running a simulation
46+
### Running a Simulation
4347

4448
By default, the hydrocompute library runs need 3 specific instructions settings: data, steps, and functions. The data submitted to the library is saved using the following instruction:
4549

4650
```javascript
47-
compute.data({ id: 'itemName', data: some2Dor1Darray })
51+
// Save data to the internal database
52+
await compute.data({ id: 'itemName', data: someNDArray })
4853
```
4954

5055
If no id is passed, the library will save a random name generated for the data. To revise the available data, then pass the command
@@ -53,25 +58,125 @@ If no id is passed, the library will save a random name generated for the data.
5358
compute.availableData()
5459
```
5560

56-
Steps are inferred from the configuration for each
61+
Steps are inferred from the configuration for each run.
62+
63+
<details>
64+
<summary><b>JavaScript (Native & HydroLang)</b></summary>
65+
<br>
66+
67+
Run native JavaScript functions or harness **HydroLang** capabilities directly.
68+
69+
```javascript
70+
// Native JS execution
71+
compute.run({
72+
dataIds: [['dataId']],
73+
functions: [['Math.max']]
74+
});
75+
76+
// Using HydroLang
77+
compute.run({
78+
engine: 'javascript',
79+
functions: [['str']], // Module
80+
funcArgs: [[{ func: 'aridity', args: { /* params */ } }]]
81+
});
82+
```
83+
</details>
84+
<br>
85+
86+
<details>
87+
<summary><b>Python (Pyodide)</b></summary>
88+
<br>
89+
90+
Execute Python code directly in the browser using Pyodide.
91+
92+
```javascript
93+
compute.run({
94+
type: 'python',
95+
engine: 'python',
96+
dataIds: [['my_data_id']],
97+
functions: [['python_script_id']] // ID of code saved in 'settings' store
98+
});
99+
```
100+
</details>
101+
<br>
102+
103+
<details>
104+
<summary><b>R (WebR)</b></summary>
105+
<br>
57106

58-
If there is only 1 data souce available inside the available data namespace, then just calling run would suffice.
107+
Run R scripts and hydrological packages using WebR.
59108

60109
```javascript
61-
compute.run()
110+
compute.run({
111+
type: 'webr',
112+
engine: 'webr',
113+
dataIds: [['my_data_id']],
114+
functions: [['r_script_id']] // ID of code saved in 'settings' store
115+
});
62116
```
117+
</details>
118+
<br>
119+
120+
<details>
121+
<summary><b>WebAssembly (WASM)</b></summary>
122+
<br>
63123

64-
To run a batch work, do:
124+
Execute high-performance compiled modules.
65125

66126
```javascript
127+
// Set engine to WASM
128+
await compute.setEngine('wasm');
129+
67130
compute.run({
68-
linked: Boolean stating linkeage between steps
69-
functions: [Array of functions per step],
70-
dataId: [Array of names of saved data],
71-
dependencies: [Array of dependencies as numbers, if applicable],
72-
funcArgs: [Array of additional configurations per function, if applicable]
73-
})
131+
engine: 'wasm',
132+
functions: [['accumulate_flow']],
133+
dependencies: [[[ 'dem_data_id' ]]],
134+
dataIds: [[[ 'dem_data_id' ]]]
135+
});
74136
```
137+
</details>
138+
<br>
139+
140+
<details>
141+
<summary><b>WebGPU</b></summary>
142+
<br>
143+
144+
Leverage GPU acceleration for massive parallel datasets.
145+
146+
```javascript
147+
// Set engine to WebGPU
148+
await compute.setEngine('webgpu');
149+
150+
compute.run({
151+
engine: 'webgpu',
152+
functions: [['matrix_multiply']],
153+
dataIds: [['matrixA_id', 'matrixB_id']]
154+
});
155+
```
156+
</details>
157+
<br>
158+
159+
<details>
160+
<summary><b>Combined Workflow (Multi-Engine)</b></summary>
161+
<br>
162+
163+
Chain multiple engines together in a single workflow. For example, process data in Python, analyze in R, and visualize in JavaScript.
164+
165+
```javascript
166+
compute.run({
167+
linked: true, // Pass results from step 0 to step 1
168+
functions: [
169+
['python_script_id'], // Step 0: Python preprocessing
170+
['r_stats_script_id'], // Step 1: R statistical analysis
171+
['vis_func_id'] // Step 2: JS Visualization preparation
172+
],
173+
engine: ['python', 'webr', 'javascript'], // Define engine per step if supported (or set individually)
174+
// Note: Actual mixed-engine runs rely on the 'type' param per step or switching engines between runs.
175+
// A common pattern is to run them sequentially and link via Data IDs.
176+
});
177+
```
178+
</details>
179+
75180
The console of the browser will show the number of executions done by the engine once the results are finished. To retrieve the results, prompt the following command.
76181

77182
```javascript

‎examples/integrated/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Integrated Catchment Workflow
2+
3+
## Overview
4+
This "Big Example" demonstrates the power of HydroCompute by chaining multiple engines into a single, cohesive hydrological workflow.
5+
6+
## The Workflow
7+
1. **Input**: Raw rainfall time series and digital elevation model (DEM).
8+
2. **Step 1 (JavaScript)**: Calculate precipitation statistics (Mean/Max) to characterize the storm event.
9+
3. **Step 2 (Python)**: Smooth the resulting hydrograph to remove sensor noise and identify peak flow timing.
10+
4. **Step 3 (R)**: Perform Flood Frequency Analysis to determine the return period of this specific event (e.g., "Is this a 50-year flood?").
11+
5. **Step 4 (WASM)**: Map the flow accumulation paths to visualize where the water goes.
12+
13+
## Implementation
14+
This example uses a `main.js` that orchestrates calls to different engines sequentially, passing data between steps (simulated or real). It serves as a dashboard for a complete hydrological analysis.

‎examples/integrated/index.html‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
7+
<title>Integrated Catchment Analysis - HydroCompute</title>
8+
<style>
9+
body {
10+
font-family: 'Segoe UI', sans-serif;
11+
margin: 0;
12+
padding: 0;
13+
color: #333;
14+
background: #f5f7fa;
15+
}
16+
17+
header {
18+
background: #2c3e50;
19+
color: white;
20+
padding: 1rem 2rem;
21+
}
22+
23+
header h1 {
24+
margin: 0;
25+
font-size: 1.5rem;
26+
}
27+
28+
.nav {
29+
margin-top: 0.5rem;
30+
font-size: 0.9rem;
31+
}
32+
33+
.nav a {
34+
color: #bdc3c7;
35+
text-decoration: none;
36+
}
37+
38+
.nav a:hover {
39+
color: white;
40+
}
41+
42+
.dashboard {
43+
display: grid;
44+
grid-template-columns: 1fr 1fr;
45+
gap: 20px;
46+
padding: 20px;
47+
max-width: 1200px;
48+
margin: 0 auto;
49+
}
50+
51+
.card {
52+
background: white;
53+
padding: 1.5rem;
54+
border-radius: 8px;
55+
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.05);
56+
}
57+
58+
.card h2 {
59+
margin-top: 0;
60+
font-size: 1.2rem;
61+
border-bottom: 2px solid #ecf0f1;
62+
padding-bottom: 0.5rem;
63+
color: #7f8c8d;
64+
}
65+
66+
.card.js {
67+
border-top: 4px solid #f1c40f;
68+
}
69+
70+
.card.python {
71+
border-top: 4px solid #3498db;
72+
}
73+
74+
.card.r {
75+
border-top: 4px solid #9b59b6;
76+
}
77+
78+
.card.wasm {
79+
border-top: 4px solid #e67e22;
80+
}
81+
82+
.status {
83+
font-weight: bold;
84+
color: #bdc3c7;
85+
margin-bottom: 1rem;
86+
font-size: 0.9rem;
87+
}
88+
89+
.status.running {
90+
color: #e67e22;
91+
}
92+
93+
.status.done {
94+
color: #27ae60;
95+
}
96+
97+
.result-box {
98+
background: #f8f9fa;
99+
padding: 0.8rem;
100+
border-radius: 4px;
101+
font-family: monospace;
102+
font-size: 0.9rem;
103+
min-height: 100px;
104+
overflow: hidden;
105+
}
106+
107+
#pipeline-control {
108+
grid-column: 1 / -1;
109+
text-align: center;
110+
padding: 2rem;
111+
background: white;
112+
border-radius: 8px;
113+
}
114+
115+
button {
116+
background: #27ae60;
117+
color: white;
118+
border: none;
119+
padding: 10px 20px;
120+
font-size: 1rem;
121+
border-radius: 4px;
122+
cursor: pointer;
123+
transition: background 0.2s;
124+
}
125+
126+
button:hover {
127+
background: #219150;
128+
}
129+
130+
button:disabled {
131+
background: #95a5a6;
132+
cursor: not-allowed;
133+
}
134+
135+
#workflow-log {
136+
grid-column: 1 / -1;
137+
background: #2c3e50;
138+
color: #ecf0f1;
139+
padding: 1rem;
140+
border-radius: 8px;
141+
font-family: monospace;
142+
height: 150px;
143+
overflow-y: auto;
144+
}
145+
</style>
146+
</head>
147+
148+
<body>
149+
<header>
150+
<h1>Integrated Catchment Analysis</h1>
151+
<div class="nav"><a href="../index.html">← Back to Examples</a></div>
152+
</header>
153+
154+
<div class="dashboard">
155+
<div id="pipeline-control">
156+
<p>Runs a multi-step hydrological workflow across JavaScript, Python, R, and WASM engines.</p>
157+
<button id="runBtn">Run Full Workflow</button>
158+
</div>
159+
160+
<div class="card js">
161+
<h2>1. Precipitation Stats (JS)</h2>
162+
<div class="status" id="status-js">Waiting...</div>
163+
<div class="result-box" id="result-js">-</div>
164+
</div>
165+
166+
<div class="card python">
167+
<h2>2. Hydrograph Smoothing (Python)</h2>
168+
<div class="status" id="status-py">Waiting...</div>
169+
<div class="result-box" id="result-py">-</div>
170+
</div>
171+
172+
<div class="card r">
173+
<h2>3. Flood Frequency (R)</h2>
174+
<div class="status" id="status-r">Waiting...</div>
175+
<div class="result-box" id="result-r">-</div>
176+
</div>
177+
178+
<div class="card wasm">
179+
<h2>4. Flow Mapping (WASM)</h2>
180+
<div class="status" id="status-wasm">Waiting...</div>
181+
<div class="result-box" id="result-wasm">-</div>
182+
</div>
183+
184+
<div id="workflow-log">Ready to start...</div>
185+
</div>
186+
187+
<script type="module" src="./main.js"></script>
188+
</body>
189+
190+
</html>

0 commit comments

Comments
 (0)