Skip to content

Update preface and quick start guide #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jdecarolis wants to merge 1 commit into
base: unstable
Choose a base branch
from
+116 −102
Open

Update preface and quick start guide #241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 64 additions & 49 deletions docs/source/preface.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,61 +16,76 @@ soon as possible.

What is Temoa?
--------------
Temoa is an energy system optimization model (ESOM) developed over many years
to support transparent, data-driven analysis of energy systems. ESOMs
serve as an important planning tool because they allow users to examine energy
futures using a self-consistent framework for evaluation. Temoa is implemented
as a linear program that minimizes the total cost of energy supply by optimizing
the installation and operation of energy technologies over a user-defined
planning horizon. Energy supply must meet end-use demands subject to physical
and technical constraints governing system operation, along with user-defined
policy scenarios.

The energy system on which Temoa and other ESOMs operate can be visualized as
a directed graph. Primary energy sources represent the points of origin, which
are transformed by a network of energy conversion and delivery technologies, and
ultimately produce consumable energy commodities that satisfy end-use demands.
[#esom_definition]_. Temoa provides tools to explicitly represent this network,
visualize system structure, and trace energy flows through time.

A defining strength of Temoa is its flexible treatment of time. Users may
define arbitrary model periods of varying length and represent intra-period
operations using seasonal and time-of-day slices, full chronological hours, or
representative days. Capacity expansion can be solved under perfect foresight
or using a rolling-horizon. In addition, Temoa supports technology
vintaging, separate loan periods and physical lifetimes, and both global and
technology-specific discount rates. Beyond deterministic optimization, Temoa
supports stochastic optimization as well as modeling-to-generate alternatives
(MGA) to explore near-optimal solution spaces. All of Temoa's features were
driven by specific analytic needs over a decade of model development and
policy-focused application.

Temoa is implemented within an open-source software stack and is released under
the MIT license, with source code available on GitHub [#open_source_realities]_.
The model is written in Python and seamlessly integrates with the broader Python
ecosystem. Input data are stored in a relational SQLite database, enabling
transparency, reproducibility, and easy modification. The model maintains a
strict distinction between source code and the input data on which it operates.
The model can be executed on single machines, multi-core systems, or
high-performance computing environments.

The name Temoa (Tools for Energy Model Optimization and Analysis) reflects the
project's broader scope. The platform comprises four interrelated components:
the underlying mathematical formulation, its software implementation, a suite
of supporting tools for data management, analysis, and visualization, and an
online presence that supports documentation, dissemination, and community
engagement. Together, these elements are designed to foster collaboration,
extensibility, and trust in energy system modeling results.

Temoa is an energy system optimization model (ESOM). Briefly, ESOMs optimize the
installation and utilization of energy technology capacity over a user-defined
time horizon. Optimal decisions are driven by an objective function that minimizes
the cost of energy supply. Conceptually, one may think of an ESOM as a "left-to-right"
network graph, with a set of energy sources on the lefthand side of the graph that
are transformed into consumable energy commodities by a set of energy technologies,
which are ultimately used to meet demands on the righthand side of the network graph.
[#esom_definition]_

Key features of the core Temoa model include:

* Flexible time slicing by season and time-of-day
* Variable length model time periods
* Technology vintaging
* Separate technology loan periods and lifetimes
* Global and technology-specific discount rates
* Capability to perform stochastic optimization
* Capability to perform modeling-to-generate alternatives (MGA)


Temoa design features include:

* Source code licensed under MIT, available through Github [#open_source_realities]_
* Open source software stack
* Part of a rich Python ecosystem
* Data stored in a relational database system (sqlite)
* Ability to utilize multi-core and compute cluster environments

The word 'Temoa' is actually an acronym for "Tools for Energy Model Optimization
and Analysis," currently composed of four (major) pieces of infrastructure:

* The mathematical model
* The implemented model (code)
* Surrounding tools
* An online presence

Each of these pieces is fundamental to creating a transparent and usable model
with a community oriented around collaboration.


Why Temoa?
----------
In 2009, when the idea for Temoa was born, most options for energy systems
modeling were geared towards government institutions that could afford the
expensive commercial software licenses. Closed source code and data also
meant that it was impossible for third parties to verify published model
results, even though those results were being used to inform public policy
decisions involving significant transfers of wealth and direct consequences for
people's lives. In addition, models were typically used to run a limited number
of scenarios that did not address the true underlying uncertainty about the
future.

Today's vibrant open source energy modeling community did not exist at
that time. We were motivated to build Temoa around three high-level objectives:
(1) make the model code and data open source to enable third party replication
of results, (2) use an open source software stack to minimize the barriers to
entry in energy modeling, and (3) build a toolkit to evaluate future uncertainty
in different ways, depending on the question at hand.

Temoa remains one of the most fully-featured, open source energy system models
focused on projecting changes across the whole energy system.

In short, because we believe that ESOM-based analyses should be repeatable by
independent third parties. The only way to make this happen is to
have a freely available model, and to create an ecosystem of freely shared data
and model inputs.

For a longer explanation, please see :cite:`Hunter_etal_2013`. In summary,
ESOM-based analyses are (1) impossible to validate, (2) complex enough as to be
non-repeatable without electronic access to **exact** versions of code *and* data
input, and (3) often do a poor job addressing uncertainty. We believe that
ESOM-based analyses should be completely open, independently reproducible,
electronically available, and address uncertainty about the future.


Temoa Origin and Pronunciation
Expand Down
105 changes: 52 additions & 53 deletions docs/source/quick_start.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,13 @@ solver. HiGHS is automatically available when you install Temoa and requires no
Running Temoa
-------------

To run the model, a configuration (‘config’) file is needed. An
To run the model, a configuration (``config``) file is needed. An
example config file called :code:`config_sample.toml` resides within the
:code:`data_files/my_configs` folder. Running the model with a config file allows
the user to (1) use a sqlite
database for storing input and output data, (2) create a formatted Excel output
file, (2) specify the solver to use, (3) return the log file produced during
model execution, (4) return the lp file utilized by the solver, and (5) to
execute several of the modeling extensions.
the user to (1) use a sqlite database for storing input and output data, (2)
create a formatted Excel output file, (2) specify the solver to use, (3) return
the log file produced during model execution, (4) return the lp file utilized by
the solver, and (5) to execute several of the modeling extensions.

.. parsed-literal::
$ temoa run config.toml
Expand Down Expand Up @@ -127,53 +126,53 @@ execute several of the modeling extensions.
Database Construction
====================================

Input datasets in Temoa can be constructed either as text files or relational
databases. Input text files are referred to as 'DAT' files and follow a specific
format. Take a look at the example DAT files in the :code:`temoa/data_files`
directory.

While DAT files work fine for small datasets, relational databases are preferred
for larger datasets. To first order, you can think of a database as a collection
of tables, where a 'primary key' within each table defines a unique entry (i.e.,
row) within the table. In addition, a 'foreign key' defines a table element drawn
from another table. Foreign keys enforce the defined relationships between
different sets and parameters.
Input datasets in Temoa are stored in a relational database management system.
For those unfamiliar with databases, you can think of them as collections of
tables. Within each table, a 'primary key' uniquely identifies each row. A
'foreign key' is a column in one table that references the primary key of
another table, thereby establishing relationships between tables and ensuring
data consistency across the database.

Temoa uses `sqlite`_, a widely used, self-contained database
system. Building a database first requires constructing a sql file, which is
simply a text file that defines the structure of different database tables and
includes the input data. The snippet below is from the technology table used to
define the 'temoa_utopia' dataset:

**NOTE**: *As of Version 3, the below table format is dated, but still shows the general structure of
the SQL files used to create the database.*
includes the input data. The snippet below is from the ``time_period`` table
used to define the ``test_system`` dataset:

.. parsed-literal::
CREATE TABLE technologies (
tech text primary key,
flag text,
sector text,
tech_desc text,
tech_category text,
FOREIGN KEY(flag) REFERENCES technology_labels(tech_labels),
FOREIGN KEY(sector) REFERENCES sector_labels(sector));
INSERT INTO "technologies" VALUES('IMPDSL1','r','supply',' imported diesel','petroleum');
INSERT INTO "technologies" VALUES('IMPGSL1','r','supply',' imported gasoline','petroleum');
INSERT INTO "technologies" VALUES('IMPHCO1','r','supply',' imported coal','coal');

The first line creates the table. **Lines 2-6** define the columns within this table.
Note that the the technology ('tech') name defines the primary key. Therefore, the
same technology name cannot be entered twice; each technology name must be unique.
**Lines 7-8** define foreign keys within the table. For example, each technology
should be specified with a label (e.g., 'r' for 'resource'). Those labels must
come from the 'technology_labels' table. Likewise, the sector name must be defined
in the 'sector_labels' table. This enforcement of names across tables using
foreign keys helps immediately catch typos. (As you can imagine, typos happen in
plain text files and Excel when defining thousands of rows of data.) Another big
advantage of using databases is that the model run outputs are stored in
separate database output tables. The outputs by model run are indexed by a scenario name,
which makes it possible to perform thousands of runs, programatically store all
the results, and execute arbitrary queries that instantaneously return the requested
1 CREATE TABLE time_period
2 (
3 sequence INTEGER UNIQUE,
4 period INTEGER
5 PRIMARY KEY,
6 flag TEXT
7 REFERENCES time_period_type (label)
8 );
9 INSERT INTO "time_period" VALUES(1,2015,'e');
10 INSERT INTO "time_period" VALUES(2,2020,'f');
11 INSERT INTO "time_period" VALUES(3,2025,'f');
12 INSERT INTO "time_period" VALUES(4,2030,'f');
13 INSERT INTO "time_period" VALUES(5,2035,'f');

The first line creates the table. **Lines 3-7** define the columns within this
table. Note that ``period`` is the primary key. Therefore, the same time period
cannot be entered twice; each name must be unique. **Line 7**, which helps
define the ``flag`` column, declares a foreign key reference to the ``label``
column of the ``time_period_type`` table. As a result, if the user tries to
enter a label in this table that does not exist in the ``time_period_type``
table, it will fail with an error. This foreign key reference ensures that the
modeler doesn't accidently type the wrong label in this table. For context,
there are two basic types of time periods in Temoa, ``e``, which defines
pre-existing periods, and ``f``, which defines future time periods that are to
be optimized.

This enforcement of names across tables using foreign keys helps immediately
catch typos. As you can imagine, typos happen in plain text files and Excel when
defining thousands of rows of data. Another big advantage of using databases is
that the model run outputs are stored in separate database output tables. The
outputs by model run are indexed by a scenario name, which makes it possible to
perform thousands of runs, programatically store all the results, and execute
arbitrary queries that instantaneously return the requested
data.

Because some database table elements serve as foreign keys in other tables, we
Expand Down Expand Up @@ -224,12 +223,12 @@ recommend that you populate input tables in the following order:
* limit_tech_input_split
* limit_tech_output_split

For help getting started, take a look at how :code:`data_files/temoa_utopia.sql` is
constructed. Use :code:`data_files/temoa_schema.sql` (a database file with the requisite
structure but no data added) to begin building your own database file. We recommend
leaving the database structure intact, and simply adding data to the schema file, or
constructing an empty database from the schema file and then using a database editor
to import data.
For help getting started, take a look at how ``data_files/example_dbs/utopia.sql``
is constructed. To begin building your own database file, use
``data_files/temoa_schema_v4.sql``, which is a database file with the requisite
structure but no data added. We recommend leaving the database structure intact,
and simply adding data to the schema file, or constructing an empty database
from the schema file and then using a script or database editor to import data.
Once the sql file is complete, you can convert it into a binary sqlite file by
installing sqlite3 and executing the following command:

Expand Down