diff --git a/README.adoc b/README.adoc index bf8829a57a..f62e06d07a 100644 --- a/README.adoc +++ b/README.adoc @@ -1,6 +1,6 @@ -= https://github.com/input-output-hk/marconi[Cardano Blockchain Indexer Framework] += https://github.com/input-output-hk/marconi[Marconi] :email: plutus@iohk.io -:author: Input Output HK Limited +:author: Input Output Global :toc: left :reproducible: @@ -13,58 +13,65 @@ Its original purpose is to provide a library that eases the tailored indexing of information. However, this repository also contains a set of indexers that are ready to use and some executables that group them. +We've also bundle some of them in executables that you can reuse if you want, as +the noe we use for link:./marconi-sidechain/README.md[sidechain] While we'd recommend defining indexers that fits your need, you can reuse the existing ones to test Marconi. == Key features * Customizable backend. The core library comes with - file:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend], - a file:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend] - or a file:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend]. -* file:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers] + link:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend], + a link:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend] + or a link:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend]. +* link:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers] to query them, -* file:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing] +* link:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing] (new indexers don't reset existing ones). -* file:./marconi-core/src/Marconi/Core/Transformer/WithCatchup.hs[Batch insert] to +* link:./marconi-core/src/Marconi/Core/Transformer/WithCatchup.hs[Batch insert] to speed up synchronisation with the node. * Logging using https://github.com/input-output-hk/iohk-monitoring-framework[iohk-monitoring-framework]. -* Indexers can be file:./marconi-core/src/Marconi/Core/Coordinator.hs[grouped]. +* Indexers can be link:./marconi-core/src/Marconi/Core/Coordinator.hs[grouped]. +* Differentiate easily volatile data (subject to rollback) from stable one. -A comparison with other chain-indexing projects is also https://example.com/TODO[available]. +A comparison with other chain-indexing projects is also +link:./doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst[available]. +//// == Quick start -To be completed... +To be completed +//// == Documentation === User documentation -The more in-depth user documentation is available in our http://example.com/TODO[website]. +A more complete user manual should be available soon. === Haddock documentation The Haskell API documentation (Haddock) for the Marconi libraries is hosted here: * https://input-output-hk.github.io/marconi/main/marconi-core[marconi-core] +* https://input-output-hk.github.io/marconi/main/marconi-core-json-rpc[marconi-core-json-rpc] * https://input-output-hk.github.io/marconi/main/marconi-cardano-core[marconi-cardano-core] * https://input-output-hk.github.io/marconi/main/marconi-cardano-indexers[marconi-cardano-indexers] -* https://input-output-hk.github.io/marconi/main/marconi-core-json-rpc[marconi-core-json-rpc] You may generate them directly with `Cabal` for each component: ``` cabal haddock marconi-core cabal haddock marconi-core-json-rpc -cabal haddock marconi-cardano-chain-index +cabal haddock marconi-cardano-core +cabal haddock marconi-cardano-indexers ``` [[Architecture]] == Architecture overview -See the http://example.com/TODO[Design and Architecture] section the website for more in-depth explainations. +See the link:./doc/read-the-docs-site/architecture/index.rst[Design and Architecture] section the website for more in-depth explainations. == CONTRIBUTING @@ -77,9 +84,11 @@ See link:CONTRIBUTING{outfilesuffix}[CONTRIBUTING] for comprehensive documentati Issues can be filed in the https://github.com/input-output-hk/marconi/issues[GitHub Issue tracker]. +//// == Roadmap To be completed... +//// == Licensing diff --git a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst index dcf0c85f05..83429fe8cb 100644 --- a/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst +++ b/doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst @@ -10,20 +10,18 @@ Marconi provides few out-of-the-box indexing solutions and the existing executables were designed either as an example or to suit specific needs, not to address a general indexing need. -cardano-node ------------- +`cardano-node `_ +----------------------------------------------------------------- -https://github.com/input-output-hk/cardano-node[``cardano-node``] -doesn't offer any kind of indexing out-of-the-box. +cardano-node doesn't offer any kind of indexing out-of-the-box. However, through the mini-protocols, one can get access to a stream of blocks that can be processed. The mini-protocol is what Marconi uses internally to synchronise with the node. -cardano-db-sync ---------------- +`cardano-db-sync `_ +----------------------------------------------------------------------- -https://github.com/input-output-hk/cardano-db-sync[``cardano-db-sync``] -is the reference indexing solution for Cardano. +cardano-db-sync is the reference indexing solution for Cardano. It's exhaustive and is ready to use out of the box. Compared to Marconi, ``cardano-db-sync`` doesn't offer any way to customise the indexing (aside filtering out tables that you don't want to index) and @@ -36,10 +34,10 @@ indepedently), while Marconi use a SQLite database embedded in the solution ``cardano-db-sync`` doesn't provide any kind of API on top of it, and many people uses ``blockfrost`` on top of it to access it through an API. -Kupo ----- +`Kupo `_ +-------------------------------------------------- -https://cardanosolutions.github.io/kupo/[Kupo] is also a predefined indexing solution. +Kupo is also a predefined indexing solution. As Marconi, it uses ``SQLite`` as a backend. It also provides facilities to monitor addresses or policy ids for a short amount of time, but no further customisation. @@ -47,11 +45,10 @@ It doesn't have the modularity of Marconi, but it's an out of the box solutions that works. -Scrolls -------- +`Scrolls `_ +---------------------------------------------- -https://github.com/txpipe/scrolls[Scrolls] is an aggregator of on-chain data, -with a no-sql storage. +Scrolls is an aggregator of on-chain data, with a no-sql storage. Use `node-to-node` protocol to synchronised, which is at the same time more defensive (thus slower) and more resilient than the node-to-client protocol of Marconi. @@ -61,9 +58,9 @@ Scrolls is one step further than Marconi in the "one aggregator to answer one specific question" direction. -Carp ----- +`Carp `_ +----------------------------------------- -https://dcspark.github.io/carp/[Carp] is another modular indexing solution. +Carp is another modular indexing solution. It uses postgres database for storage and provides a set of indexers that you can choose from for indexing. They also provide a REST API to access the data. diff --git a/marconi-cardano-core/README.md b/marconi-cardano-core/README.md new file mode 100644 index 0000000000..3223080f11 --- /dev/null +++ b/marconi-cardano-core/README.md @@ -0,0 +1,11 @@ +# marconi-cardano-core + +`marconi-cardano-core` is a Haskell library that ease the use of `marconi` on +Cardano. + +It provides facilities to get the block streams from a node and a few helpers +to connect indexers to this stream. + +It also contains a [test library](test-lib) that provides a mockchain that you +can use to test your indexers. + diff --git a/marconi-cardano-indexers/README.md b/marconi-cardano-indexers/README.md new file mode 100644 index 0000000000..701ed367a9 --- /dev/null +++ b/marconi-cardano-indexers/README.md @@ -0,0 +1,28 @@ +# marconi-cardano-indexers + +This package contains a set of indexers for Cardano that can be reused in any +marconi-indexing project. + +## Available indexers + +- [`BlockInfo`](src/Marconi/Cardano/Indexers/BlockInfo.hs): get the starting + slot, the creation time and the epoch number of a block. +- [`ChainTip`](src/Marconi/Cardano/Indexers/BlockInfo.hs) get the slot and block + header hash of the chain tip of the node your synchronising against. +- [`Datum`](src/Marconi/Cardano/Indexers/Datum.hs): get all the datum on the + chain with their hash. +- [`EpochNonce`](src/Marconi/Cardano/Indexers/EpochNonce.hs) get the epoch Nonce used + at each epoch. +- [`EpochSDD`](src/Marconi/Cardano/Indexers/EpochSDD.hs) get the stake pool + delegation at each epoch. +- [`MintTokenEvent`](src/Marconi/Cardano/Indexers/MintTokenEvent.hs) get all the + minting and/or burning events. +- [`Spent`](src/Marconi/Cardano/Indexers/Spent.hs) get all the spent Tx outputs + with the transaction they were spent in. +- [`Utxo`](src/Marconi/Cardano/Indexers/Spent.hs) get all the utxos that were + created. + +Note that both `EpochNonce` and `EpochSDD` require that you maintain a ledger +state (that can be obtained using an +[`ExtLedgerStateCoordinator`](src/Marconi/Cardano/Indexers/Spent.hs)), which is +quite memory and CPU expensive, especially on mainnet. diff --git a/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs b/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs index fbd261daed..bdbd3eb914 100644 --- a/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs +++ b/marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs @@ -13,6 +13,7 @@ module Marconi.Cardano.Indexers.BlockInfo ( BlockInfo (BlockInfo), blockNo, timestamp, + timestampUTCTime, epochNo, -- * Indexer and worker @@ -49,6 +50,7 @@ import Data.Text (Text) import Data.Text qualified as Text import Data.Time qualified as Time import Data.Time.Clock.POSIX (POSIXTime) +import Data.Time.Clock.POSIX qualified as Time import Data.Word (Word64) import Database.SQLite.Simple (NamedParam ((:=))) import Database.SQLite.Simple qualified as SQL @@ -80,6 +82,9 @@ Aeson.deriveJSON Aeson.defaultOptions{Aeson.fieldLabelModifier = tail} ''BlockIn Lens.makeLenses ''BlockInfo +timestampUTCTime :: Lens.Getter BlockInfo Time.UTCTime +timestampUTCTime = timestamp . Lens.to (Time.posixSecondsToUTCTime . fromIntegral) + type instance Core.Point BlockInfo = C.ChainPoint -- | A raw SQLite indexer for BlackInfo diff --git a/marconi-core-json-rpc/README.md b/marconi-core-json-rpc/README.md index f37b95144e..9134672169 100644 --- a/marconi-core-json-rpc/README.md +++ b/marconi-core-json-rpc/README.md @@ -1 +1,4 @@ # marconi-core-json-rpc + +Provides types and function to ease the setup of JSON-RPC queries for an +indexer.