This repository was archived by the owner on Aug 6, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathgenerator_explained.html
More file actions
290 lines (288 loc) · 13.8 KB
/
Copy pathgenerator_explained.html
File metadata and controls
290 lines (288 loc) · 13.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
<!DOCTYPE html>
<html lang="en">
<head>
<title>Newt -- generator_explained</title>
<link rel="stylesheet" href="https://caltechlibrary.github.io/css/site.css">
<base href="./">
</head>
<body>
<header>
<a href="https://library.caltech.edu"><img src="https://caltechlibrary.github.io/assets/liblogo.gif" alt="Caltech Library logo"></a>
</header>
<nav>
<ul>
<li><a href="/">Home</a></li>
<li><a href="./">README</a></li>
<li><a href="user_manual.html">User Manual</a></li>
<li><a href="LICENSE">LICENSE</a></li>
<li><a href="INSTALL.html">INSTALL</a></li>
<li><a href="about.html">About</a></li>
<li><a href="search.html">Search</a></li>
<li><a href="https://github.com/caltechlibrary/newt">GitHub</a></li>
</ul>
</nav>
<section>
<h1 id="newts-newt-generate-explained">Newt’s <code>newt generate</code>
Explained</h1>
<p>// FIXME: REWRITE NEEEDED based on direction of current prototype</p>
<p>Newt comes with a code generator. It does the heavy lift of knitting
your application together from off the shelf parts. The code generator
is used to create a new Newt YAML file as well as SQL, TypeScript,
Handlebar templates and other miscelanios code bits. You run the code
generator via the Newt development tools, <code>newt</code> (or
<code>newt.exe</code> on Windows). The tool is truction around the
concept of an actions on an object.</p>
<p>The basic worflow of code generation is as follows</p>
<ol type="1">
<li>create/update your application’s Newt YAML file</li>
<li>create/update your data models</li>
<li>generate the application</li>
<li>run, test and repeat</li>
</ol>
<p>Your application at this stage should be able to create modeled
objects, retrieve modeled objects, update modeled objects, delete
modeled objects and list modeled objects. When youare happy with the
base functionality you can then turn to refining the application by
customizing the generated code, adding custom routes and templates as
well as browser side functionality delivered via Newt’s data router
static file service.</p>
<h2 id="generated-files-for-postgrespostgrest-based-projects">Generated
files for Postgres+PostgREST based projects</h2>
<p>The Newt code generator delivers the following code assets for your
Postgres+PostgREST based Newt Application</p>
<dl>
<dt><code>postgrest.conf</code></dt>
<dd>
A basic PostgREST configuration file for your project
</dd>
<dt><code>setup.sql</code></dt>
<dd>
This file is used to configure Postgres to work with PostgREST. This is
code you can execute via the psql shell. Usually you don’t check this
file into your code (e.g. GitHub) repository as it will need to contain
DB credentials for the application’s DB account.
</dd>
<dt><code>models.sql</code></dt>
<dd>
This file setups the database schema as well as creates Pg/SQL functions
for use with PostgREST in your CRUD-L application operations
</dd>
</dl>
<p>Depending on what the setting of the base directory and model name
it’ll create the follow templates. In this example
<code><base_dir></code> would be replaced by the “base directory”
setting your your Newt YAML file and <code><model></code> would be
replaced by the model name.</p>
<dl>
<dt><code><base_dir>/<model>_create_form.hbs</code></dt>
<dd>
This implements the HTML data entry form create a new object based on
<code><model></code>.
</dd>
<dt><code><base_dir>/<model>_create_response.hbs</code></dt>
<dd>
This implements the HTML response from the create object data entry
form.
</dd>
<dt><code><base_dir>/<model>_read.hbs</code></dt>
<dd>
This implements the HTML read only view a object previously created or
updated.
</dd>
<dt><code><base_dir>/<model>_update_form.hbs</code></dt>
<dd>
This implements the HTML data entry form to update an objected previouly
created.
</dd>
<dt><code><base_dir>/<model>_update_response.hbs</code></dt>
<dd>
This implements the HTML response from the update data entry form.
</dd>
<dt><code><base_dir>/<model>_delete_form.hbs</code></dt>
<dd>
This implemented the HTML form to delete an object previously created.
</dd>
<dt><code><base_dir>/<model>_delete_response.hbs</code></dt>
<dd>
This implemented the HTML response from deleting an object.
</dd>
<dt><code><base_dir>/<model>_list.hbs</code></dt>
<dd>
This lists objects previously created or updated.
</dd>
</dl>
<p>Once your code is generated you’ll first need to setup the Postgres
database to work with PostgREST. That is done by using the Postgres
<code>psql</code> shell to run the SQL code. You first run
<code>setup.sql</code> and then run <code>models.sql</code>.</p>
<p>After that you’re ready to test your basic application. The
<code>newt</code> (or <code>newt.exe</code> on Windows) tool can also
“run” your generated code. You would use the “run” action and the name
of your Newt Project file.</p>
<h2 id="minimal-functionality">Minimal functionality</h2>
<p>The generated code, after settting up Postgres+PostgREST has a
minimum level of functionality. It implements the five basic operations
to manage metadata - Create, Read, Update, Delete and List (abbr:
CRUD-L) for each object type modeled in your Newt YAML file. Newt
assumes each model is independent but multiple objects may be defined in
your application. If you need to combine models (not unusual) then you
will need to enhance the generated SQL, routes and/or templates to
better support that type of integration. For now let us focus on the
basics.</p>
<h2 id="newts-approach">Newt’s Approach</h2>
<p>The Newt philosophy is to model data in YAML and then mangage the
data via data management system. It presumes the data management system
provides a JSON API for interacting with it. In Newt this is referred to
as a “JSON data source”. What is a data management system? Typical this
will be a JSON friendly database combined with a JSON API. Newt provides
code generation for Postgres (the database) plus PostgREST (the JSON
API). The code generation includes setting up a simple configuration
file, generating SQL to define tables, schema as well as SQL functions.
The result can be used to bring up an instance of Postgres+PostgREST for
your project.</p>
<p>Newt also supports using Dataset+datasetd as a primary JSON data
source. Dataset can store JSON documents in a pairtree, in MySQL,
Postgres or SQLite3 database. Newt’s Dataset+datasetd support assumes
you’re using SQLite3 as your storage format.</p>
<p>Newt code generation can accomplish the following tasks</p>
<ol type="1">
<li>generate the basic configuration files needed for your JSON data
source</li>
<li>Provide any additioan SQL or data langauge files needed to implement
CRUD-L opperations for your models</li>
<li>Configure (wire up) the routes and templates mapping in your Newt
YAML file</li>
<li>Generate Handlebar templates for the created routes in your Newt
YAML file</li>
<li>Generate TypeScript middleware suitable for Deno to validate you
pipeline inputs</li>
</ol>
<h2
id="bringing-up-postgrespostgrest-and-the-newt-template-engine">Bringing
up Postgres+PostgREST and the Newt Template Engine</h2>
<p>Postgres and PostgREST provide us with a JSON API that can be used in
a Newt router pipeline. Following the Newt philosophy data modeling and
management happen in the database. That is done via the SQL language
which the Newt generator can produce. After the database is setup in
Postgres with the scheme, tables, functions and views needed for basic
CRUD-L operations you can generate a compatible PostgREST configuration
file. Here’s the steps what the generated code does based on the models
defined in a Newt YAML file.</p>
<ol start="2" type="1">
<li>For each Newt model create a table in the database/scheme</li>
<li>For each Newt model create a function for each of the CRUD[^71]
options[^72]</li>
<li>For each Newt model create a SQL function to handle the list view of
the CRUD-L</li>
<li>Generate a PostgREST configuration file.</li>
</ol>
<p>When the “postgrest.conf”, “setup.sql”, and “models.sql” are
generated then do the following.</p>
<ol type="1">
<li>Create the Postgres database and schema[^70] if they don’t
exist</li>
<li>Run the “setup.sql” and “models.sql” via the <code>psql</code>
client.</li>
</ol>
<pre class="shell"><code>newt generate app.yaml
dropdb --if-exists myapp
createdb app
psql app -c "\\i setup.sql"
psql app -c "\\i models.sql"</code></pre>
<p>NOTE: The generator always generates the files using predefined
names. If generated files already exist it will rename the old version
with a “.bak” file extension before writing the newly generated code. If
you’ve customized the generated files then the new version will NOT have
those customizations. If you are customizing the generated files you
have a couple options. First is to rename them or put them in a sub
directory. Another would be to develop a workflow where you diff your
modifications and try to apply the resulting patch file after
regenerating the files.</p>
<p>NOTE: The Newt generator will alter (add or update) routes associated
with the generated SQL functions and the Mustache templates. It will
also alter the templates defined your Newt application YAML file.</p>
<p>When we ran <code>newt generate app.yaml</code> Mustache templates
were also created in the current directory along side the “setup.sql”,
“models.sql” and “postgrest.conf” files. The Mustache templates are
named according to a model name and their role. When we create, update
or delete objects we’ve modeled there are two round trips needed in the
request and response cycle with our browser. That also means there needs
to be two templates, one presenting the web form and the other showing
the processed result. The model name and role are concatenated with an
underscore, “_”, form the base of the filename followed by the “.tmpl”
extension. The read and list operations only require a single template
each. If I have a model with the id of “bird” then the resulting
template files would be as follows.</p>
<ul>
<li><code>views/bird_create_form.hbs</code> (contains the web form)</li>
<li><code>views/bird_create_response.hbs</code> (response for the
submitted form)</li>
<li><code>views/bird_read.hbs</code> (display a single object we’ve
modeled)</li>
<li><code>views/bird_update_form.hbs</code> (contains the web form)</li>
<li><code>views/bird_update_response.hbs</code> (response for the
submitted form)</li>
<li><code>views/bird_delete_form.hbs</code> (contains the web form)</li>
<li><code>views/bird_delete_response.hbs</code> (response for the
submitted form)</li>
<li><code>views/bird_list.hbs</code> (list objects we’ve modeled)</li>
</ul>
<p>When we generated the Mustache templates the <code>templates</code>
properties were updated to include these templates for the Newt Mustache
template engine. Also the related routes were updated with a pipeline
that interactive with the Postgres+PostgREST JSON API sending that
response through the Mustache Template engine.</p>
<h2 id="testing-our-generated-application">Testing our generated
application</h2>
<p>The <code>newt</code> command can run Newt Router, Newt Mustache and
PostgREST using the “run” option. We can test our generated application
by “running” them and then pointing our web browser at the Newt Router
localhost and port to test.</p>
<ol type="1">
<li>Newt Run</li>
<li>Point our web browser at the router URL</li>
</ol>
<p>In a shell/terminal session type the following.</p>
<pre class="shell"><code>newt run app.yaml</code></pre>
<p>Now launch your web browser and point it at the URL indicated
(e.g. “http://localhost:8010”) to test your application.</p>
<h2 id="where-to-go-next">Where to go next?</h2>
<p>There are three “places” to customize a Newt application. The first
would be to customize data processing inside Postgres. Postgres is a
full feature object relational database system. Most of the SQL:2023
Core. That provides allow of data management capability. Postgres
supports a wide range of data types, more than Newt uses. Postgres also
supports many popular programming languages embedded inside Postgres.
This includes the <a
href="https://www.postgresql.org/docs/current/plpython.html">Python
programming language</a>. When you are using a general purpose language
like Python inside your database system you have synergy between actions
on your data, via SQL triggers, and the functions you write in Python
reacting to the data. PL/Python may also be used to reach outside of
Postgres to perform other tasks, e.g. feed indexing updates to
Solr/OpenSearch/ElasticSearch or send an email.</p>
<p>If you can integrate supports for other services by adding additional
“routes” to your Newt YAML file. You can also add steps to your
pipeline. Let’s say you have an object modeled in Postgres but you also
want to include additional information from another service
(e.g. ORCID). You can write a simple web service that takes the object
as generated by the PostgREST request and processes it further before
handing the result back to Mustache for rendering.</p>
<p>Finally the third options is to enhance your templates with CSS,
JavaScript and other static web assets. Newt’s router functions both as
a data request router but also as a static file server. If you wanted to
integrate external content with a web form (e.g. ORCID or ROR
information) you could do this via JavaScript and have their web browser
connect to the external service retrieving additional metadata before
the form is submitted.</p>
</section>
<footer>
<span><h1><A href="http://caltech.edu">Caltech</a></h1></span>
<span>© 2024 <a href="https://www.library.caltech.edu/copyright">Caltech library</a></span>
<address>1200 E California Blvd, Mail Code 1-32, Pasadena, CA 91125-3200</address>
<span>Phone: <a href="tel:+1-626-395-3405">(626)395-3405</a></span>
<span><a href="mailto:library@caltech.edu">Email Us</a></span>
<a class="cl-hide" href="sitemap.xml">Site Map</a>
</footer>
</body>
</html>