feat: azure resources sub types management and first customization

docs/how-to/030_azure-source.md

@@ -82,6 +82,94 @@ For a more in-depth explanation refer to [Naming rules and restrictions for Azur
 Use `.name` wherever available to display casing matters, its availability is dependant on the specific resource APIs.
 The spelling Azure reported, if needed, is written to the source logs at the `Debug` level whenever it differs from the normalised value.
 
+## Resource sub-types
+
+Some Azure resource types describe more than one thing: a `Microsoft.Web/sites` resource is an App
+Service site, but when its `kind` carries the `functionapp` token it is also a Function App.
+
+For these types the Azure source produces, out of one Azure resource, both the item of the resource
+itself and one or more **sub-type** items, each described by its own mapping file and related to the
+item of the resource it was derived from.
+A sub-type is always additive: the item of the Azure resource is produced exactly as it was before,
+and the sub-type item is created next to it together with a relationship pointing at it.
+
+### How a sub-type mapping is dispatched
+
+The `type` of a sub-type mapping file is an **internal dispatch key**, not an Azure provider type:
+
+- it is never sent to Azure and never used to build a resource graph query
+- it is never matched against the resource type of an event, because only the source can decide to
+	emit it
+- its `extra.apiVersion` is never read, since the resource is always retrieved with the `apiVersion`
+	of its parent type. It is kept in the file only for symmetry with every other Azure mapping
+
+Which Azure type produces which sub-type is hardcoded in the source, together with the check deciding
+whether a retrieved resource must produce it. Declaring a sub-type therefore takes both a new mapping
+file and a change to that hardcoded dictionary: a sub-type can never be introduced by configuration
+alone.
+
+### The sub-types shipped with ibdm
+
+| Azure type | Sub-type mapping | Produced when |
+| --- | --- | --- |
+| `Microsoft.Web/sites` | `docs/mappings/azure/websites_functionapps.yaml`, `type: functionapps` | the `kind` of the site carries the `functionapp` token |
+
+`kind` is a comma separated list of tokens, such as `app`, `app,linux` or `functionapp,linux`, and
+the tokens are compared one by one: a site whose kind is `myfunctionapp` is not a Function App.
+A site without a usable `kind` produces no sub-type and nothing fails.
+
+The mapping creates an `functionapps` item and, through its `extra` section, a `dependency`
+relationship from that item to the `websites` item of the same site.
+
+Both mapping files must be loaded for the sub-type to be produced. Loading
+`docs/mappings/azure/websites.yaml` alone reproduces exactly the behaviour the source had before
+sub-types existed, deletion included. Loading `websites_functionapps.yaml` alone can instead never
+produce anything, so the source logs a warning when it starts and carries on.
+
+`ibdm sync azure` and `ibdm run azure` behave identically, because the check runs on the payload the
+Azure APIs returned and is indifferent to which of them retrieved it. To adopt a sub-type on an
+already imported subscription load both mapping files and run `ibdm sync azure` once: every site that
+already exists gets its sub-type item and its relationship.
+
+### Deleting a resource that has sub-types
+
+`Microsoft.Resources.ResourceDeleteSuccess` carries only the id of the deleted resource. Its `kind`
+is gone and no API can return it any more, so at deletion time the check cannot run: the source
+deletes the item of the resource **and the item of every sub-type its type can produce**, whether or
+not that resource ever produced it.
+
+For a `Microsoft.Web/sites` resource with both mapping files loaded, three deletions reach the
+catalog:
+
+| deleted | why |
+| --- | --- |
+| the `websites` item | the resource itself |
+| the `functionapps` item | the only sub-type configured for its type |
+| the relationship of the `functionapps` item | its `deletePolicy` is `cascade` |
+
+A deletion addressed to a sub-type item the resource never produced is inert: the catalog publish
+reports no per item outcome, so nothing fails and nothing is left behind. The identifier of a
+sub-type item also lives in its own namespace, `functionapps-<resource id>` for the Function Apps,
+so such a deletion can only ever name the sub-type item of that very resource.
+
+Removing a sub-type mapping file is not the reverse operation: the items it already published stop
+being updated and stop being deleted together with their resource, so they have to be removed by
+hand.
+
+### Authoring a sub-type mapping
+
+- Declare `syncable: true`. Keeping a sub-type key out of the resource graph queries is the job of
+	the hardcoded dictionary, not of `syncable`, and `syncable: false` would only risk confining the
+	sub-type to `ibdm run azure`
+- Build the identifier of the item, and the identifier of every `deletePolicy: "cascade"` extra, out
+	of `.id` alone. A deletion payload carries only `id` and `type`, so a template reading any other
+	field fails to render and that deletion is lost
+- Give the sub-type item its own item family, so that its identifiers can never collide with the ones
+	of another mapping, and make sure the item type definition for that family exists in the catalog
+- Treat the payload as read only. A sub-type receives a shallow copy of the payload of its parent, so
+	writing into a nested value, such as `properties` or `tags`, would be seen by every other item
+	produced out of the same resource
+
 ## Authentication
 
 The source is using the [`DefaultAzureCredential` chain of authentication] so you can setup

docs/mappings/azure/websites_functionapps.yaml

@@ -0,0 +1,43 @@
+apiVersion: azure.mia-platform.eu/v1
+itemFamily: functionapps
+type: functionapps
+extra:
+  apiVersion: "2025-03-01"
+syncable: true
+mappings:
+  identifier: |-
+    {{ printf "functionapps-%s" .id | sha256sum }}
+  metadata:
+    title: |-
+      {{ printf "Function App - %s" .name }}
+  spec:
+    name: "{{ .name }}"
+    id: "{{ .id }}"
+    location: "{{ .location | lower }}"
+    state: "{{ .properties.state | lower }}"
+    kind: "{{ .kind }}"
+    enabled: "{{ .properties.enabled }}"
+    availabilityState: "{{ .properties.availabilityState | lower }}"
+    defaultHostname: "{{ .properties.defaultHostName }}"
+    serverFarmId: "{{ .properties.serverFarmId }}"
+    tags: |-
+      {{ $tags := (get "tags" . "") -}}
+      {{- if $tags -}}
+        {{- $tags | toJSON  -}}
+      {{- else -}}
+        {{- object | toJSON -}}
+      {{- end }}
+  extra:
+  - apiVersion: mia-platform.eu/v1
+    itemFamily: relationships
+    deletePolicy: "cascade"
+    identifier: |-
+      {{ $src := printf "urn:mia-platform-catalog:azure.mia-platform.eu:v1:FunctionApp:%s" (printf "functionapps-%s" .id | sha256sum) -}}
+      {{- $type := "urn:mia-platform-catalog:mia-platform.eu:v1:RelationshipType:dependency.mia-platform.eu" -}}
+      {{- $tgt := printf "urn:mia-platform-catalog:azure.mia-platform.eu:v1:WebSite:%s" (printf "%s" .id | sha256sum) -}}
+      {{- printf "%s-%s-%s" $src $type $tgt | sha256sum }}
+    sourceRef: |-
+      urn:mia-platform-catalog:azure.mia-platform.eu:v1:FunctionApp:{{ printf "functionapps-%s" .id | sha256sum }}
+    targetRef: |-
+      urn:mia-platform-catalog:azure.mia-platform.eu:v1:WebSite:{{ (printf "%s" .id | sha256sum) }}
+    typeRef: "urn:mia-platform-catalog:mia-platform.eu:v1:RelationshipType:dependency.mia-platform.eu"

internal/source/azure/azure.go

@@ -98,6 +98,7 @@ func (s *Source) StartSyncProcess(ctx context.Context, typesToFilter map[string]
 	if err := s.validateForSync(); err != nil {
 		return handleError(err)
 	}
+	warnOrphanSubTypes(logger, typesToFilter)
 
 	client, err := s.azureGraphClient()
 	if err != nil {
@@ -111,74 +112,95 @@ func (s *Source) StartSyncProcess(ctx context.Context, typesToFilter map[string]
 	})
 
 	for resType := range typesToFilter {
-		var query *string
-		switch resType {
-		case arm.ResourceGroupResourceType.String():
-			graphResourceType := arm.SubscriptionResourceType.String() + "/resourceGroups"
-			query = to.Ptr(fmt.Sprintf(resourceContainerGraphQueryTemplate, graphResourceType))
-		case arm.SubscriptionResourceType.String():
-			query = to.Ptr(fmt.Sprintf(resourceContainerGraphQueryTemplate, resType))
-		default:
-			query = to.Ptr(fmt.Sprintf(resourceGraphQueryTemplate, resType))
+		// a sub-type is emitted while handling its parent resource, and its type key is an internal
+		// dispatch key: querying Azure for it would only ask for a type that does not exist.
+		if isSubTypeKey(resType) {
+			logger.Debug("skipping sub-type mapping, it is emitted with its parent type", "type", resType)
+			continue
 		}
 
-		queryRequest := armresourcegraph.QueryRequest{
-			Subscriptions: []*string{to.Ptr(s.SubscriptionID)},
-			Query:         query,
+		if err := s.syncResourceType(ctx, client, resType, typesToFilter, dataChannel); err != nil {
+			// handleError swallows the cancellation, so a stopped sync process is not a failure
+			return handleError(err)
 		}
+	}
 
-		for {
-			timestamp := timeProvider()
-			response, err := client.Resources(ctx, queryRequest, nil)
+	s.syncContext.Swap(nil)
+	return nil
+}
 
-			switch {
-			case errors.Is(err, context.Canceled):
-				logger.Debug("stopping sync process due to context cancellation")
-				return nil
-			case err != nil:
-				return handleError(err)
-			}
+// syncResourceType pages through every resource of resType the Resource Graph returns and emits
+// the item of each one of them, together with the ones of the sub-types they additionally produce.
+func (s *Source) syncResourceType(ctx context.Context, client *armresourcegraph.Client, resType string, typesToFilter map[string]source.Extra, dataChannel chan<- source.Data) error {
+	logger := logger.FromContext(ctx).WithName(logName)
+	queryRequest := armresourcegraph.QueryRequest{
+		Subscriptions: []*string{to.Ptr(s.SubscriptionID)},
+		Query:         resourceGraphQuery(resType),
+	}
+
+	for {
+		timestamp := timeProvider()
+		response, err := client.Resources(ctx, queryRequest, nil)
+
+		switch {
+		case errors.Is(err, context.Canceled):
+			logger.Debug("stopping sync process due to context cancellation")
+			return nil
+		case err != nil:
+			return err
+		}
 
-			if data, ok := response.Data.([]any); ok {
-				for _, item := range data {
-					if values, ok := item.(map[string]any); ok {
-						normalizeResourceValues(logger, values, resType)
-						dataChannel <- source.Data{
-							Type:      resType,
-							Operation: source.DataOperationUpsert,
-							Time:      timestamp,
-							Values:    values,
-						}
-					} else {
-						// something very wrong is going on, print an error and continue
-						logger.Debug("retrieve data item is not a valid map")
+		if data, ok := response.Data.([]any); ok {
+			for _, item := range data {
+				if values, ok := item.(map[string]any); ok {
+					normalizeResourceValues(logger, values, resType)
+					for _, resourceData := range resourceDataToEmit(resType, values, typesToFilter, source.DataOperationUpsert, timestamp) {
+						dataChannel <- resourceData
 					}
+				} else {
+					// something very wrong is going on, print an error and continue
+					logger.Debug("retrieve data item is not a valid map")
 				}
-			} else {
-				// something very wrong is going on, print an error and continue
-				logger.Debug("response data is not a valid type")
 			}
+		} else {
+			// something very wrong is going on, print an error and continue
+			logger.Debug("response data is not a valid type")
+		}
 
-			if response.ResultTruncated == nil || *response.ResultTruncated == armresourcegraph.ResultTruncatedFalse {
-				break
-			}
+		if response.ResultTruncated == nil || *response.ResultTruncated == armresourcegraph.ResultTruncatedFalse {
+			break
+		}
 
-			queryRequest.Options = &armresourcegraph.QueryRequestOptions{
-				SkipToken: response.SkipToken,
-			}
+		queryRequest.Options = &armresourcegraph.QueryRequestOptions{
+			SkipToken: response.SkipToken,
 		}
 	}
 
 	s.syncContext.Swap(nil)
 	return nil
 }
 
+// resourceGraphQuery returns the Resource Graph query retrieving every resource of resType, taken
+// from the container table for the types that live in it.
+func resourceGraphQuery(resType string) *string {
+	switch resType {
+	case arm.ResourceGroupResourceType.String():
+		graphResourceType := arm.SubscriptionResourceType.String() + "/resourceGroups"
+		return to.Ptr(fmt.Sprintf(resourceContainerGraphQueryTemplate, graphResourceType))
+	case arm.SubscriptionResourceType.String():
+		return to.Ptr(fmt.Sprintf(resourceContainerGraphQueryTemplate, resType))
+	default:
+		return to.Ptr(fmt.Sprintf(resourceGraphQueryTemplate, resType))
+	}
+}
+
 // StartEventStream implement source.EventSource.
 func (s *Source) StartEventStream(ctx context.Context, typesToFilter map[string]source.Extra, dataChannel chan<- source.Data) error {
 	logger := logger.FromContext(ctx).WithName(logName)
 	if err := s.validateForEventStream(); err != nil {
 		return handleError(err)
 	}
+	warnOrphanSubTypes(logger, typesToFilter)
 
 	client, err := s.azureClient()
 	if err != nil {
@@ -206,7 +228,9 @@ func (s *Source) StartEventStream(ctx context.Context, typesToFilter map[string]
 }
 
 func partitionEventHandler(client *armresources.Client, typesToFilter map[string]source.Extra, dataChannel chan<- source.Data) eventHandler {
-	typesSlice := slices.Sorted(maps.Keys(typesToFilter))
+	// a sub-type type key is an internal dispatch key and can never be the type of an event
+	// subject, so it is left out of the set the subject type is resolved against.
+	typesSlice := slices.DeleteFunc(slices.Sorted(maps.Keys(typesToFilter)), isSubTypeKey)
 
 	return func(ctx context.Context, receivedData *azeventhubs.ReceivedEventData) {
 		logger := logger.FromContext(ctx).WithName(logName)
@@ -258,21 +282,17 @@ func partitionEventHandler(client *armresources.Client, typesToFilter map[string
 				}
 
 				normalizeResourceValues(logger, values, resourceType)
-				dataChannel <- source.Data{
-					Type:      resourceType,
-					Operation: source.DataOperationUpsert,
-					Time:      *envelope.Time,
-					Values:    values,
+				for _, resourceData := range resourceDataToEmit(resourceType, values, typesToFilter, source.DataOperationUpsert, *envelope.Time) {
+					dataChannel <- resourceData
 				}
 			case azsystemevents.TypeResourceDeleteSuccess:
 				logger.Trace("deleting resource", "resourceType", resourceType)
+				// the event carries only the resource id, so no sub-type check can run here and a
+				// delete is emitted for every configured sub-type of the resource type.
 				values := map[string]any{idKey: resID.String()}
 				normalizeResourceValues(logger, values, resourceType)
-				dataChannel <- source.Data{
-					Type:      resourceType,
-					Operation: source.DataOperationDelete,
-					Time:      *envelope.Time,
-					Values:    values,
+				for _, resourceData := range resourceDataToEmit(resourceType, values, typesToFilter, source.DataOperationDelete, *envelope.Time) {
+					dataChannel <- resourceData
 				}
 			default:
 				logger.Trace("skipping resource", "resourceType", resourceType, "eventType", envelope.Type, "apiVersion", apiVersion)