Rc/supernova

README.md

@@ -2,7 +2,7 @@
 
 The notifier service is a component that receives block events synchronously
 from multiversx observer nodes, and it forwards them to a subscribing component
-(via message queuing service or websockets)
+(via message queuing service or websockets).
 
 ## Prerequisites
 
@@ -11,15 +11,12 @@ has to setup one or multiple observers. For running an observing squad,
 these [docs](https://docs.multiversx.com/integrators/observing-squad/) 
 cover the whole process. 
 
-The observer node can be connected using WebSocket integration. Please check observer
-node config for setting up the connector. Enable `HostDriversConfig` in order to use
-WebSocket integration.
+There are two ways the observer nodes can push data to the notifier service:
 
-The required configs for launching an observer/s with a driver attached,
-can be found [here](https://github.com/multiversx/mx-chain-go/blob/master/cmd/node/config/external.toml).
+1. **WebSocket Integration (Recommended)**: The observer node pushes events to the notifier over a persistent WebSocket connection. This is the newer, more efficient method. To use this, check the observer node config and enable `HostDriversConfig`. 
+   The required configs for launching an observer/s with a WS driver attached can be found [here](https://github.com/multiversx/mx-chain-go/blob/master/cmd/node/config/external.toml).
 
-The HTTP integration is still available for backwards compatibility, but it will be
-deprecated in the future.
+2. **HTTP POST Integration (Deprecated)**: The observer node makes separate HTTP POST requests to the notifier's API endpoints (`/events/push`, etc.) for every event. This integration is still available for backwards compatibility, but it will be deprecated in the future.
 
 ## How to run
 
@@ -64,6 +61,8 @@ The supported config variables are:
   as the one specified in the `ProxyUrl` described above.
 - `Username`: the username used to authorize an observer. Can be left empty for `UseAuthorization = false` on observer connector.
 - `Password`: the password used to authorize an observer. Can be left empty for `UseAuthorization = false` on observer connector.
+- `CheckDuplicates`: signals if the events received from observers have already been pushed to clients. Requires a Redis instance/cluster.
+- `WithReadStateChanges`: signals if read state changes operations will be handled by the notifier. Note: this requires read state changes to also be enabled on the observer nodes.
 
 If observer connector is set to use BasicAuth with `UseAuthorization = true`, `Username` and `Password` has to be
 set here on events notifier, and `Auth` flag has to be enabled in
@@ -131,16 +130,22 @@ make docker-new publisher_type=rabbitmq
 
 ### API Endpoints
 
-Notifier service will expose several events routes, the observer nodes will
-push events to these routes:
+Depending on the observer integration method chosen, the notifier service will expose the following routes.
+
+**For WebSocket Integration (Observer -> Notifier):**
+When using the WebSocket integration, the observer pushes all events through a single persistent WebSocket connection rather than relying on the HTTP routes below.
+
+**For HTTP POST Integration (Observer -> Notifier):**
+The observer nodes will push events to these routes via consecutive HTTP requests:
 - `/events/push` (POST) -> it will handle all events for each round
 - `/events/revert` (POST) -> if there is a reverted block, the event will be
   pushed on this route
 - `/events/finalized` (POST) -> when the block has been finalized, the events
   will be pushed on this route
 
-If the service will be in "notifier" mode, it will expose a additional route:
-- `/hub/ws` (GET) - this route can be used to manage the websocket connection (check [websocket subscribing](#websockets) section for more details on this)
+**For Client Subscriptions (Notifier -> Consumer):**
+If the service will be in "notifier" mode (using the `ws` publisher), it will expose an additional route for end consumers to receive data:
+- `/hub/ws` (GET) - this route can be used to manage the WebSocket connection (check [websocket subscribing](#websockets) section for more details on this)
 
 ## Redis
 
@@ -166,10 +171,110 @@ in code in `data/outport.go` file.
 Once the proxy is launched together with the observer/s, the driver's methods
 will be called. 
 
+### Event types
+
+There are multiple event types available, they can be found as constants in common package,
+[constants](https://github.com/multiversx/mx-chain-notifier-go/blob/main/common/constants.go).
+Below is the list of available event types together with their associated JSON payload structures.
+
+- `all_events`: Pushes all logs and events for a block.
+```json
+{
+  "hash": "blockHash1",
+  "events": [
+    {
+      "address": "addr1",
+      "identifier": "identifier",
+      "topics": ["<< base64 encoded topic1 >>", "<< base64 encoded topic2 >>"],
+      "data": "<< base64 encoded data >> ",
+      ...
+    }
+  ]
+}
+```
+
+- `block_events`: Pushes block info alongside its logs and events.
+```json
+{
+  "hash": "blockHash1",
+  "shardId": 1,
+  "timestamp": 12345678,
+  "timestampMs": 12345678000,
+  "events": [
+    {
+      "address": "addr1",
+      "identifier": "identifier",
+      ...
+    }
+  ]
+}
+```
+
+- `revert_events`: Pushes information relating to a reverted block.
+```json
+{
+    "hash": "blockHash1",
+    "nonce": 11,
+    "round": 2,
+    "epoch": 1,
+    "shardId": 1,
+    "timestamp": 12345678,
+    "timestampMs": 12345678000
+}
+```
+
+- `finalized_events`: Pushes the hash of a finalized block.
+```json
+{
+    "hash": "blockHash"
+}
+```
+
+- `block_txs`: Pushes all transactions contained within a block.
+```json
+{
+  "hash": "blockHash1",
+  "txs": {
+    "txHash1": {      
+        "Nonce": 123,
+        "Round": 1,
+        "Epoch": 1,
+        ...
+    }
+  }
+}
+```
+
+- `block_scrs`: Pushes all smart contract results contained within a block.
+```json
+{
+  "hash": "blockHash1",
+  "scrs": {
+    "scrHash1": {      
+        "Nonce": 123,
+        ...
+    }
+  }
+}
+```
+
+- `block_state_accesses`: Pushes information regarding the state accesses (reads/writes) occurring within a block. *(Requires `WithReadStateChanges` config enabled)*
+```json
+{
+  "hash": "blockHash1",
+  "shardID": 1,
+  "timestampMs": 12345678000,
+  "nonce": 123,
+  "stateAccessesPerAccounts": {
+       ...
+  }
+}
+```
+
 ### RabbitMQ
 
 When using a setup with `RabbitMQ` you have to subscribe to each exchange
-separately.
+separately. This can be handled via RabbitMQ Management UI platform.
 
 ### WebSockets
 
@@ -295,63 +400,3 @@ inner marshalled data like:
 }
 ```
 
-There are multiple event types available, they can be found as constants in common package,
-[constants](https://github.com/multiversx/mx-chain-notifier-go/blob/main/common/constants.go). Below there is the event type together with the associated marshalled data type.
-- `all_events`
-```json
-{
-  "hash": "blockHash1",
-  "events": [
-    {
-      "address": "addr1",
-      "identifier": "identifier",
-      ...
-    }
-  ]
-}
-```
-
-- `revert_events`
-```json
-{
-    "hash": "blockHash1",
-    "nonce": 11,
-    "round": 2,
-    "epoch": 1,
-}
-```
-
-- `finalized_events`
-```json
-{
-    "hash": "blockHash"
-}
-```
-
-- `block_txs`:
-```json
-{
-  "hash": "blockHash1",
-  "txs": {
-    "txHash1": {      
-        "Nonce": 123,
-        "Round": 1,
-        "Epoch": 1,
-        ...
-    }
-  }
-}
-```
-
-- `block_scrs`
-```json
-{
-  "hash": "blockHash1",
-  "scrs": {
-    "scrHash1": {      
-        "Nonce": 123,
-        ...
-    }
-  }
-}
-```

api/groups/eventsGroup.go

@@ -98,6 +98,7 @@ func getPayloadVersion(c *gin.Context) uint32 {
 func (h *eventsGroup) pushEvents(c *gin.Context) {
 	pushEventsRawData, err := c.GetRawData()
 	if err != nil {
+		log.Error("pushEvents: failed to get raw data", "error", err)
 		shared.JSONResponse(c, http.StatusBadRequest, nil, err.Error())
 		return
 	}
@@ -106,6 +107,7 @@ func (h *eventsGroup) pushEvents(c *gin.Context) {
 
 	err = h.payloadHandler.ProcessPayload(pushEventsRawData, outport.TopicSaveBlock, payloadVersion)
 	if err != nil {
+		log.Error("pushEvents: failed to process payload", "error", err)
 		shared.JSONResponse(c, http.StatusBadRequest, nil, err.Error())
 		return
 	}
@@ -116,6 +118,7 @@ func (h *eventsGroup) pushEvents(c *gin.Context) {
 func (h *eventsGroup) revertEvents(c *gin.Context) {
 	revertEventsRawData, err := c.GetRawData()
 	if err != nil {
+		log.Error("revertEvents: failed to get raw data", "error", err)
 		shared.JSONResponse(c, http.StatusBadRequest, nil, err.Error())
 		return
 	}
@@ -124,6 +127,7 @@ func (h *eventsGroup) revertEvents(c *gin.Context) {
 
 	err = h.payloadHandler.ProcessPayload(revertEventsRawData, outport.TopicRevertIndexedBlock, payloadVersion)
 	if err != nil {
+		log.Error("revertEvents: failed to process payload", "error", err)
 		shared.JSONResponse(c, http.StatusBadRequest, nil, err.Error())
 		return
 	}
@@ -134,6 +138,7 @@ func (h *eventsGroup) revertEvents(c *gin.Context) {
 func (h *eventsGroup) finalizedEvents(c *gin.Context) {
 	finalizedRawData, err := c.GetRawData()
 	if err != nil {
+		log.Error("finalizedEvents: failed to get raw data", "error", err)
 		shared.JSONResponse(c, http.StatusBadRequest, nil, err.Error())
 		return
 	}
@@ -142,6 +147,7 @@ func (h *eventsGroup) finalizedEvents(c *gin.Context) {
 
 	err = h.payloadHandler.ProcessPayload(finalizedRawData, outport.TopicFinalizedBlock, payloadVersion)
 	if err != nil {
+		log.Error("finalizedEvents: failed to process payload", "error", err)
 		shared.JSONResponse(c, http.StatusBadRequest, nil, err.Error())
 		return
 	}

api/groups/eventsGroup_test.go

@@ -190,7 +190,7 @@ func TestEventsGroup_PushEvents(t *testing.T) {
 						ExecutionOrder: 1,
 					},
 				},
-				Logs: []*outport.LogData{
+				Logs: []*transaction.LogData{
 					{
 						Log: &transaction.Log{
 							Address: []byte("logaddr1"),

common/common.go

@@ -0,0 +1,6 @@
+package common
+
+// ConvertTimeStampMsToSec will convert unix timestamp from milliseconds to seconds
+func ConvertTimeStampMsToSec(timeStamp uint64) uint64 {
+	return timeStamp / 1000
+}

data/block.go

@@ -32,6 +32,8 @@ type InterceptorBlockData struct {
 	ScrsWithOrder            map[string]*outport.SCRInfo
 	LogEvents                []Event
 	StateAccessesPerAccounts map[string]*stateChange.StateAccesses
+	Nonce                    uint64
+	TimeStampMs              uint64
 }
 
 // ArgsSaveBlockData holds the block data that will be received on push events
@@ -47,6 +49,8 @@ type ArgsSaveBlockData struct {
 	NumberOfShards         uint32
 	HeaderTimeStampMs      uint64
 	StateAccesses          map[string]*stateChange.StateAccesses
+	StateAccessesForBlock  map[string]*outport.StateAccessesForBlock
+	Results                map[string]*outport.ExecutionResultData
 }
 
 // OutportBlockDataOld holds the block data that will be received on push events
@@ -62,6 +66,7 @@ type OutportBlockDataOld struct {
 	AlteredAccounts        map[string]*alteredAccount.AlteredAccount
 	NumberOfShards         uint32
 	IsImportDB             bool
+	StateAccesses          map[string]*stateChange.StateAccesses
 }
 
 // ArgsSaveBlock holds block data with header type

go.mod

@@ -8,8 +8,8 @@ require (
 	github.com/go-redis/redis/v8 v8.11.3
 	github.com/google/uuid v1.6.0
 	github.com/gorilla/websocket v1.5.3
-	github.com/multiversx/mx-chain-communication-go v1.2.0
-	github.com/multiversx/mx-chain-core-go v1.4.1
+	github.com/multiversx/mx-chain-communication-go v1.3.0
+	github.com/multiversx/mx-chain-core-go v1.4.2-0.20260219122727-014ae9f9311f
 	github.com/multiversx/mx-chain-logger-go v1.1.0
 	github.com/pelletier/go-toml v1.9.3
 	github.com/prometheus/client_model v0.6.1
@@ -48,7 +48,7 @@ require (
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/mr-tron/base58 v1.2.0 // indirect
-	github.com/multiversx/mx-chain-crypto-go v1.2.13-0.20250218161752-9482d9a22234 // indirect
+	github.com/multiversx/mx-chain-crypto-go v1.3.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/pelletier/go-toml/v2 v2.1.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect

go.sum

@@ -128,12 +128,12 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
 github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
 github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
 github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
-github.com/multiversx/mx-chain-communication-go v1.2.0 h1:0wOoLldiRbvaOPxwICbnRCqCpLqPewg8M/FMbC/0OXY=
-github.com/multiversx/mx-chain-communication-go v1.2.0/go.mod h1:wS3aAwkmHbC9mlzQdvL6p7l8Rqw3vmzhj7WZW1dTveA=
-github.com/multiversx/mx-chain-core-go v1.4.1 h1:ljs53jpdjtCohpaqm2n/dvTGrFlSgIpoZYH8RVt5cWo=
-github.com/multiversx/mx-chain-core-go v1.4.1/go.mod h1:IO+vspNan+gT0WOHnJ95uvWygiziHZvfXpff6KnxV7g=
-github.com/multiversx/mx-chain-crypto-go v1.2.13-0.20250218161752-9482d9a22234 h1:NNI7kYxzsq+4mTPSUJo0cK1+iPxjUX+gRJDaBRwEQ7M=
-github.com/multiversx/mx-chain-crypto-go v1.2.13-0.20250218161752-9482d9a22234/go.mod h1:QZAw2bZcOxGQRgYACTrmP8pfTa3NyxENIL+00G6nM5E=
+github.com/multiversx/mx-chain-communication-go v1.3.0 h1:ziNM1dRuiR/7al2L/jGEA/a/hjurtJ/HEqgazHNt9P8=
+github.com/multiversx/mx-chain-communication-go v1.3.0/go.mod h1:gDVWn6zUW6aCN1YOm/FbbT5MUmhgn/L1Rmpl8EoH3Yg=
+github.com/multiversx/mx-chain-core-go v1.4.2-0.20260219122727-014ae9f9311f h1:kngckbX3TbZpU0LQpetUM8xNvUu/FtmZQtM66yoGcz0=
+github.com/multiversx/mx-chain-core-go v1.4.2-0.20260219122727-014ae9f9311f/go.mod h1:IO+vspNan+gT0WOHnJ95uvWygiziHZvfXpff6KnxV7g=
+github.com/multiversx/mx-chain-crypto-go v1.3.0 h1:0eK2bkDOMi8VbSPrB1/vGJSYT81IBtfL4zw+C4sWe/k=
+github.com/multiversx/mx-chain-crypto-go v1.3.0/go.mod h1:nPIkxxzyTP8IquWKds+22Q2OJ9W7LtusC7cAosz7ojM=
 github.com/multiversx/mx-chain-logger-go v1.1.0 h1:97x84A6L4RfCa6YOx1HpAFxZp1cf/WI0Qh112whgZNM=
 github.com/multiversx/mx-chain-logger-go v1.1.0/go.mod h1:K9XgiohLwOsNACETMNL0LItJMREuEvTH6NsoXWXWg7g=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=