From 786a8b3605168c2bd5b2fae9f56f5671df4ebea5 Mon Sep 17 00:00:00 2001 From: Nicolas Biri Date: Tue, 2 Jan 2024 11:35:28 +0100 Subject: [PATCH 1/5] Add content to READMEs --- README.adoc | 24 ++++++++------ ...mparison-other-chain-indexing-projects.rst | 33 +++++++++---------- marconi-cardano-core/README.md | 11 +++++++ marconi-cardano-indexers/README.md | 28 ++++++++++++++++ .../src/Marconi/Cardano/Indexers/BlockInfo.hs | 5 +++ marconi-core-json-rpc/README.md | 3 ++ 6 files changed, 76 insertions(+), 28 deletions(-) create mode 100644 marconi-cardano-core/README.md create mode 100644 marconi-cardano-indexers/README.md diff --git a/README.adoc b/README.adoc index bf8829a57a..55cd352d34 100644 --- a/README.adoc +++ b/README.adoc @@ -1,6 +1,6 @@ = https://github.com/input-output-hk/marconi[Cardano Blockchain Indexer Framework] :email: plutus@iohk.io -:author: Input Output HK Limited +:author: Input Output Global :toc: left :reproducible: @@ -13,26 +13,30 @@ Its original purpose is to provide a library that eases the tailored indexing of information. However, this repository also contains a set of indexers that are ready to use and some executables that group them. +We've also bundle some of them in executables that you can reuse if you want, as +the noe we use for link:./marconi-sidechain/README.md[sidechain] While we'd recommend defining indexers that fits your need, you can reuse the existing ones to test Marconi. == Key features * Customizable backend. The core library comes with - file:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend], + link:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend], a file:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend] or a file:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend]. -* file:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers] +* link:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers] to query them, -* file:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing] +* link:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing] (new indexers don't reset existing ones). -* file:./marconi-core/src/Marconi/Core/Transformer/WithCatchup.hs[Batch insert] to +* link:./marconi-core/src/Marconi/Core/Transformer/WithCatchup.hs[Batch insert] to speed up synchronisation with the node. * Logging using https://github.com/input-output-hk/iohk-monitoring-framework[iohk-monitoring-framework]. -* Indexers can be file:./marconi-core/src/Marconi/Core/Coordinator.hs[grouped]. +* Indexers can be link:./marconi-core/src/Marconi/Core/Coordinator.hs[grouped]. +* Differentiate easily volatile data (subject to rollback) from stable one. -A comparison with other chain-indexing projects is also https://example.com/TODO[available]. +A comparison with other chain-indexing projects is also +link:./doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst[available]. == Quick start @@ -49,16 +53,16 @@ The more in-depth user documentation is available in our http://example.com/TODO The Haskell API documentation (Haddock) for the Marconi libraries is hosted here: * https://input-output-hk.github.io/marconi/main/marconi-core[marconi-core] +* https://input-output-hk.github.io/marconi/main/marconi-core-json-rpc[marconi-core-json-rpc] * https://input-output-hk.github.io/marconi/main/marconi-cardano-core[marconi-cardano-core] * https://input-output-hk.github.io/marconi/main/marconi-cardano-indexers[marconi-cardano-indexers] -* https://input-output-hk.github.io/marconi/main/marconi-core-json-rpc[marconi-core-json-rpc] You may generate them directly with `Cabal` for each component: -``` cabal haddock marconi-core cabal haddock marconi-core-json-rpc -cabal haddock marconi-cardano-chain-index +cabal haddock marconi-cardano-core +cabal haddock marconi-cardano-indexers ``` [[Architecture]] diff --git a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst index dcf0c85f05..2ab0b1c793 100644 --- a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst +++ b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst @@ -10,20 +10,18 @@ Marconi provides few out-of-the-box indexing solutions and the existing executables were designed either as an example or to suit specific needs, not to address a general indexing need. -cardano-node ------------- +`cardano-node https://github.com/input-output-hk/cardano-node` +-------------------------------------------------------------- -https://github.com/input-output-hk/cardano-node[``cardano-node``] -doesn't offer any kind of indexing out-of-the-box. +cardano-node doesn't offer any kind of indexing out-of-the-box. However, through the mini-protocols, one can get access to a stream of blocks that can be processed. The mini-protocol is what Marconi uses internally to synchronise with the node. -cardano-db-sync ---------------- +`cardano-db-sync https://github.com/input-output-hk/cardano-db-sync` +-------------------------------------------------------------------- -https://github.com/input-output-hk/cardano-db-sync[``cardano-db-sync``] -is the reference indexing solution for Cardano. +cardano-db-sync is the reference indexing solution for Cardano. It's exhaustive and is ready to use out of the box. Compared to Marconi, ``cardano-db-sync`` doesn't offer any way to customise the indexing (aside filtering out tables that you don't want to index) and @@ -36,10 +34,10 @@ indepedently), while Marconi use a SQLite database embedded in the solution ``cardano-db-sync`` doesn't provide any kind of API on top of it, and many people uses ``blockfrost`` on top of it to access it through an API. -Kupo ----- +`Kupo https://cardanosolutions.github.io/kupo/` +----------------------------------------------- -https://cardanosolutions.github.io/kupo/[Kupo] is also a predefined indexing solution. +Kupo is also a predefined indexing solution. As Marconi, it uses ``SQLite`` as a backend. It also provides facilities to monitor addresses or policy ids for a short amount of time, but no further customisation. @@ -47,11 +45,10 @@ It doesn't have the modularity of Marconi, but it's an out of the box solutions that works. -Scrolls -------- +`Scrolls https://github.com/txpipe/scrolls` +------------------------------------------- -https://github.com/txpipe/scrolls[Scrolls] is an aggregator of on-chain data, -with a no-sql storage. +Scrolls is an aggregator of on-chain data, with a no-sql storage. Use `node-to-node` protocol to synchronised, which is at the same time more defensive (thus slower) and more resilient than the node-to-client protocol of Marconi. @@ -61,9 +58,9 @@ Scrolls is one step further than Marconi in the "one aggregator to answer one specific question" direction. -Carp ----- +`Carp https://dcspark.github.io/carp/` +-------------------------------------- -https://dcspark.github.io/carp/[Carp] is another modular indexing solution. +Carp is another modular indexing solution. It uses postgres database for storage and provides a set of indexers that you can choose from for indexing. They also provide a REST API to access the data. diff --git a/marconi-cardano-core/README.md b/marconi-cardano-core/README.md new file mode 100644 index 0000000000..3223080f11 --- /dev/null +++ b/marconi-cardano-core/README.md @@ -0,0 +1,11 @@ +# marconi-cardano-core + +`marconi-cardano-core` is a Haskell library that ease the use of `marconi` on +Cardano. + +It provides facilities to get the block streams from a node and a few helpers +to connect indexers to this stream. + +It also contains a [test library](test-lib) that provides a mockchain that you +can use to test your indexers. + diff --git a/marconi-cardano-indexers/README.md b/marconi-cardano-indexers/README.md new file mode 100644 index 0000000000..701ed367a9 --- /dev/null +++ b/marconi-cardano-indexers/README.md @@ -0,0 +1,28 @@ +# marconi-cardano-indexers + +This package contains a set of indexers for Cardano that can be reused in any +marconi-indexing project. + +## Available indexers + +- [`BlockInfo`](src/Marconi/Cardano/Indexers/BlockInfo.hs): get the starting + slot, the creation time and the epoch number of a block. +- [`ChainTip`](src/Marconi/Cardano/Indexers/BlockInfo.hs) get the slot and block + header hash of the chain tip of the node your synchronising against. +- [`Datum`](src/Marconi/Cardano/Indexers/Datum.hs): get all the datum on the + chain with their hash. +- [`EpochNonce`](src/Marconi/Cardano/Indexers/EpochNonce.hs) get the epoch Nonce used + at each epoch. +- [`EpochSDD`](src/Marconi/Cardano/Indexers/EpochSDD.hs) get the stake pool + delegation at each epoch. +- [`MintTokenEvent`](src/Marconi/Cardano/Indexers/MintTokenEvent.hs) get all the + minting and/or burning events. +- [`Spent`](src/Marconi/Cardano/Indexers/Spent.hs) get all the spent Tx outputs + with the transaction they were spent in. +- [`Utxo`](src/Marconi/Cardano/Indexers/Spent.hs) get all the utxos that were + created. + +Note that both `EpochNonce` and `EpochSDD` require that you maintain a ledger +state (that can be obtained using an +[`ExtLedgerStateCoordinator`](src/Marconi/Cardano/Indexers/Spent.hs)), which is +quite memory and CPU expensive, especially on mainnet. diff --git a/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs b/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs index fbd261daed..bdbd3eb914 100644 --- a/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs +++ b/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs @@ -13,6 +13,7 @@ module Marconi.Cardano.Indexers.BlockInfo ( BlockInfo (BlockInfo), blockNo, timestamp, + timestampUTCTime, epochNo, -- * Indexer and worker @@ -49,6 +50,7 @@ import Data.Text (Text) import Data.Text qualified as Text import Data.Time qualified as Time import Data.Time.Clock.POSIX (POSIXTime) +import Data.Time.Clock.POSIX qualified as Time import Data.Word (Word64) import Database.SQLite.Simple (NamedParam ((:=))) import Database.SQLite.Simple qualified as SQL @@ -80,6 +82,9 @@ Aeson.deriveJSON Aeson.defaultOptions{Aeson.fieldLabelModifier = tail} ''BlockIn Lens.makeLenses ''BlockInfo +timestampUTCTime :: Lens.Getter BlockInfo Time.UTCTime +timestampUTCTime = timestamp . Lens.to (Time.posixSecondsToUTCTime . fromIntegral) + type instance Core.Point BlockInfo = C.ChainPoint -- | A raw SQLite indexer for BlackInfo diff --git a/marconi-core-json-rpc/README.md b/marconi-core-json-rpc/README.md index f37b95144e..9134672169 100644 --- a/marconi-core-json-rpc/README.md +++ b/marconi-core-json-rpc/README.md @@ -1 +1,4 @@ # marconi-core-json-rpc + +Provides types and function to ease the setup of JSON-RPC queries for an +indexer. From 2ecc6269c8bb929795d406e8d3e5f52c8bf2dd5a Mon Sep 17 00:00:00 2001 From: Nicolas Biri Date: Tue, 2 Jan 2024 12:31:43 +0100 Subject: [PATCH 2/5] Fix hyperlink --- .../comparison-other-chain-indexing-projects.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst index 2ab0b1c793..3ed6080d7a 100644 --- a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst +++ b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst @@ -10,7 +10,7 @@ Marconi provides few out-of-the-box indexing solutions and the existing executables were designed either as an example or to suit specific needs, not to address a general indexing need. -`cardano-node https://github.com/input-output-hk/cardano-node` +`cardano-node `_ -------------------------------------------------------------- cardano-node doesn't offer any kind of indexing out-of-the-box. @@ -18,7 +18,7 @@ However, through the mini-protocols, one can get access to a stream of blocks that can be processed. The mini-protocol is what Marconi uses internally to synchronise with the node. -`cardano-db-sync https://github.com/input-output-hk/cardano-db-sync` +`cardano-db-sync `_ -------------------------------------------------------------------- cardano-db-sync is the reference indexing solution for Cardano. @@ -34,7 +34,7 @@ indepedently), while Marconi use a SQLite database embedded in the solution ``cardano-db-sync`` doesn't provide any kind of API on top of it, and many people uses ``blockfrost`` on top of it to access it through an API. -`Kupo https://cardanosolutions.github.io/kupo/` +`Kupo `_ ----------------------------------------------- Kupo is also a predefined indexing solution. @@ -45,7 +45,7 @@ It doesn't have the modularity of Marconi, but it's an out of the box solutions that works. -`Scrolls https://github.com/txpipe/scrolls` +`Scrolls `_ ------------------------------------------- Scrolls is an aggregator of on-chain data, with a no-sql storage. @@ -58,7 +58,7 @@ Scrolls is one step further than Marconi in the "one aggregator to answer one specific question" direction. -`Carp https://dcspark.github.io/carp/` +`Carp `_ -------------------------------------- Carp is another modular indexing solution. From 500a67179023307cf144cca070a80b7ff014629b Mon Sep 17 00:00:00 2001 From: Nicolas Biri Date: Tue, 2 Jan 2024 12:38:40 +0100 Subject: [PATCH 3/5] Fix markup --- README.adoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.adoc b/README.adoc index 55cd352d34..45bb18c117 100644 --- a/README.adoc +++ b/README.adoc @@ -1,4 +1,4 @@ -= https://github.com/input-output-hk/marconi[Cardano Blockchain Indexer Framework] += https://github.com/input-output-hk/marconi[Marconi] :email: plutus@iohk.io :author: Input Output Global :toc: left @@ -22,8 +22,8 @@ existing ones to test Marconi. * Customizable backend. The core library comes with link:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend], - a file:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend] - or a file:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend]. + a link:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend] + or a link:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend]. * link:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers] to query them, * link:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing] @@ -59,6 +59,7 @@ The Haskell API documentation (Haddock) for the Marconi libraries is hosted here You may generate them directly with `Cabal` for each component: +``` cabal haddock marconi-core cabal haddock marconi-core-json-rpc cabal haddock marconi-cardano-core From 94ce7fdef2e5bd6a39c6304afecb37618f108ce8 Mon Sep 17 00:00:00 2001 From: Nicolas Biri Date: Tue, 9 Jan 2024 09:38:32 +0100 Subject: [PATCH 4/5] Fix markup --- .../comparison-other-chain-indexing-projects.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst index 3ed6080d7a..83429fe8cb 100644 --- a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst +++ b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst @@ -11,7 +11,7 @@ executables were designed either as an example or to suit specific needs, not to address a general indexing need. `cardano-node `_ --------------------------------------------------------------- +----------------------------------------------------------------- cardano-node doesn't offer any kind of indexing out-of-the-box. However, through the mini-protocols, one can get access to a stream of blocks @@ -19,7 +19,7 @@ that can be processed. The mini-protocol is what Marconi uses internally to synchronise with the node. `cardano-db-sync `_ --------------------------------------------------------------------- +----------------------------------------------------------------------- cardano-db-sync is the reference indexing solution for Cardano. It's exhaustive and is ready to use out of the box. @@ -35,7 +35,7 @@ indepedently), while Marconi use a SQLite database embedded in the solution people uses ``blockfrost`` on top of it to access it through an API. `Kupo `_ ------------------------------------------------ +-------------------------------------------------- Kupo is also a predefined indexing solution. As Marconi, it uses ``SQLite`` as a backend. @@ -46,7 +46,7 @@ that works. `Scrolls `_ -------------------------------------------- +---------------------------------------------- Scrolls is an aggregator of on-chain data, with a no-sql storage. Use `node-to-node` protocol to synchronised, which is at the same time @@ -59,7 +59,7 @@ answer one specific question" direction. `Carp `_ --------------------------------------- +----------------------------------------- Carp is another modular indexing solution. It uses postgres database for storage and provides a set of indexers that you From 00fefb749024b8a852fc8cdd6bce8d3945fb5c3a Mon Sep 17 00:00:00 2001 From: Nicolas Biri Date: Mon, 15 Jan 2024 11:19:13 +0100 Subject: [PATCH 5/5] remove dead link from the main readme --- README.adoc | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.adoc b/README.adoc index 45bb18c117..f62e06d07a 100644 --- a/README.adoc +++ b/README.adoc @@ -38,15 +38,17 @@ existing ones to test Marconi. A comparison with other chain-indexing projects is also link:./doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst[available]. +//// == Quick start -To be completed... +To be completed +//// == Documentation === User documentation -The more in-depth user documentation is available in our http://example.com/TODO[website]. +A more complete user manual should be available soon. === Haddock documentation @@ -69,7 +71,7 @@ cabal haddock marconi-cardano-indexers [[Architecture]] == Architecture overview -See the http://example.com/TODO[Design and Architecture] section the website for more in-depth explainations. +See the link:./doc/read-the-docs-site/architecture/index.rst[Design and Architecture] section the website for more in-depth explainations. == CONTRIBUTING @@ -82,9 +84,11 @@ See link:CONTRIBUTING{outfilesuffix}[CONTRIBUTING] for comprehensive documentati Issues can be filed in the https://github.com/input-output-hk/marconi/issues[GitHub Issue tracker]. +//// == Roadmap To be completed... +//// == Licensing