Update documentation to reflect transition from Elasticsearch to OpenSearch

.ruby-version

@@ -1 +1 @@
-3.3.9
+3.4.1

config/tech-docs.yml

@@ -155,7 +155,12 @@ redirects:
   /manual/data-gov-uk-supporting-ckan.html: /manual/data-gov-uk-2nd-line.html
   /manual/data-sources-for-transition.html: /manual/transition-architecture.html
   /manual/deploying.html: /manual/development-pipeline.html
-  /manual/elasticsearch.html: /manual/elasticsearch-dumps.html
+  /manual/add-a-new-field-to-an-opensearch-document-type.html: /manual/add-a-new-field-to-an-opensearch-document-type.html
+  /manual/alerts/opensearch-cluster-health.html: /manual/alerts/opensearch-cluster-health.html
+  /manual/debug-issues-in-opensearch.html: /manual/debug-issues-in-opensearch.html
+  /manual/opensearch-dumps.html: /manual/opensearch-dumps.html
+  /manual/opensearch.html: /manual/opensearch-dumps.html
+  /manual/reindex-opensearch.html: /manual/reindex-opensearch.html
   /manual/emergency-publishing-redis.html: /manual/emergency-publishing.html
   /manual/error-reporting.html: /manual/sentry.html
   /manual/fastly-error-rate.html: /manual/alerts/fastly-error-rate.html

source/manual/add-a-new-field-to-an-opensearch-document-type.html.md

@@ -0,0 +1,41 @@
+---
+owner_slack: "#govuk-search"
+title: Add a new field to an OpenSearch document type
+parent: "/manual.html"
+layout: manual_layout
+section: Search on GOV.UK
+related_repos: [search-api]
+---
+
+## The OpenSearch schema
+
+`config/schema` contains JSON files that together define a schema for documents in Search API. This is described in more detail in the
+[schema documentation][schema-documentation].
+
+When adding a new field, first you need to decide which field type to use.
+`field_types.json` defines common OpenSearch configuration that we reuse for multiple fields having the same type.
+
+The type you use affects whether the field is [analysed][] by OpenSearch and whether you can use it in [filters][] and [aggregates][].
+
+Add your new field to `field_definitions.json`.
+
+If your field should be valid for any kind of document, you can add it to `base_opensearch_type.json`. Otherwise, add it to the appropriate JSON file under `elasticsearch_types`.
+
+## Integration testing
+
+The easiest way to test the new fields is to write an integration test for it. These tests run against a development OpenSearch cluster, and create new search indices each test run.
+
+## Updating Search API schema indexes on all environments
+
+In order for the new field to work as expected, you will need to reindex OpenSearch. Follow the guidance on [how to reindex OpenSearch][reindex-guide].
+
+If you consider the change low risk and are only adding new fields for which content doesn't yet exist, you can run the `search:update_schema` [task][update-schema-task].
+This task will attempt to update the OpenSearch index schema in place without requiring a re-index.
+If you have made any changes which affect existing fields, OpenSearch will reject the change and a full re-index will be required.
+
+[schema-documentation]: https://docs.publishing.service.gov.uk/repos/search-api/schemas.html
+[analysed]: https://docs.opensearch.org/latest/analyzers/
+[filters]: https://docs.opensearch.org/latest/query-dsl/
+[aggregates]: https://docs.opensearch.org/latest/aggregations/
+[reindex-guide]: https://docs.publishing.service.gov.uk/manual/reindex-opensearch.html
+[update-schema-task]: https://github.com/alphagov/search-api/blob/main/lib/tasks/indices.rake#L131-L149

source/manual/alerts/elasticsearch-cluster-health.html.md → source/manual/alerts/opensearch-cluster-health.html.md

@@ -1,12 +1,12 @@
 ---
 owner_slack: "#govuk-searchandnav"
-title: Elasticsearch cluster health
+title: OpenSearch cluster health
 parent: "/manual.html"
 layout: manual_layout
 section: Icinga alerts
 ---
 
-Elasticsearch reports cluster health as one of three possible states, based on
+OpenSearch reports cluster health as one of three possible states, based on
 the state of its [primary and replica shards][primary-and-replica-shards].
 
 - `green` - all primary and secondary (replica) shards are allocated. There are
@@ -18,17 +18,17 @@ the state of its [primary and replica shards][primary-and-replica-shards].
   entire cluster is cold started, before it's initially allocated the primary
   shards. If it happens at other times this may be a sign of data loss.
 
-[primary-and-replica-shards]: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/_basic_concepts.html#_shards_amp_replicas
+[primary-and-replica-shards]: https://docs.opensearch.org/latest/api-reference/cluster-api/cluster-health/
 
 More [comprehensive documentation on cluster health][cluster-health-endpoint]
-can be found in the Elasticsearch documentation.
+can be found in the OpenSearch documentation.
 
 Make sure you understand the consequences of the problem before jumping to a
 solution.
 
 Icinga uses the `check_elasticsearch_aws` check from [nagios-plugins][] to
-monitor the health of the AWS managed Elasticsearch cluster. This plugin uses
-various endpoints of the Elasticsearch API, but also extrapolates additional
+monitor the health of the AWS managed OpenSearch cluster. This plugin uses
+various endpoints of the OpenSearch API, but also extrapolates additional
 information to help you diagnose any problems.
 
 [nagios-plugins]: https://github.com/alphagov/nagios-plugins/
@@ -43,13 +43,13 @@ There are tabs for 'Cluster health' and 'Instance health'.  The graphs in the
 console link to AWS Cloudwatch, where historic metrics can be viewed over custom
 time periods.
 
-#### Use the Elasticsearch API
+#### Use the OpenSearch API
 
-An alternative to using the dashboard is accessing the Elasticsearch health API
+An alternative to using the dashboard is accessing the OpenSearch health API
 yourself. Start with the [`/_cluster/health` endpoint][cluster-health-endpoint]
 and go from there.
 
-[cluster-health-endpoint]: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html
+[cluster-health-endpoint]: https://docs.opensearch.org/latest/api-reference/cluster-api/cluster-health/
 
 Response JSON from the `/_cluster/health` endpoint looks like:
 
@@ -68,14 +68,14 @@ Response JSON from the `/_cluster/health` endpoint looks like:
 }
 ```
 
-A tunnel to Elasticsearch in a specific environment (e.g staging) can be created
+A tunnel to OpenSearch in a specific environment (e.g staging) can be created
 using the following:
 
 ```
-gds govuk connect ssh --environment staging search -- -N -L 9200:elasticsearch6:80
+gds govuk connect ssh --environment staging search -- -N -L 9200:selasticsearch6:80
 ```
 
-Elasticsearch will then be available at <http://localhost:9200>.
+OpenSearch will then be available at <http://localhost:9200>.
 
 #### Logging
 
@@ -85,7 +85,7 @@ Access to logs is detailed in the [logging documentation](/manual/logging.html#e
 
 GOV.UK have a Enterprise level support plan with AWS for staging and
 production. Since we are using a managed service, AWS should be the first point
-of contact for fixing issues with the Elasticsearch cluster.  They can be
+of contact for fixing issues with the OpenSearch cluster.  They can be
 contacted by telephone, live chat or support request.
 
 Response times are:

source/manual/backups.html.md

@@ -12,7 +12,7 @@ This document describes the backup policies and strategies for GOV.UK's producti
 > For restore playbooks, see:
 >
 > - [Restore a database in Amazon RDS](howto-backup-and-restore-in-aws-rds.html)
-> - [Restore Elasticsearch indices from snapshots](elasticsearch-dumps.html)
+> - [Restore OpenSearch indices from snapshots](opensearch-dumps.html)
 
 ## Databases in Amazon RDS and DocumentDB
 
@@ -74,14 +74,14 @@ These buckets have:
 - versioning, so that we can restore previous or deleted object versions
 - 120-day timelock on the replica (in governance mode, so that we can still delete an object if, for example, a department publishes private information by mistake)
 
-## Elasticsearch indices in Amazon OpenSearch
+## OpenSearch indices in Amazon OpenSearch
 
 | Strategy | Retention | Recovery point objective (RPO) | Recovery time objective (RTO) |
 | --- | --- | --- | --- |
 | Incremental snapshots | 14 days | 1 hour | 1 working day (best effort) |
 
-GOV.UK does not store any original data in Elasticsearch, but regenerating the full index takes a long time.
+GOV.UK does not store any original data in OpenSearch, but regenerating the full index takes a long time.
 
-The Search feature on GOV.UK no longer uses Elasticsearch, but some other features of the website still rely on Elasticsearch indices for legacy reasons.
+The Search feature on GOV.UK no longer uses OpenSearch, but some other features of the website still rely on OpenSearch indices for legacy reasons.
 
-Amazon OpenSearch Service [takes hourly snapshots](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore) of Elasticsearch indices as standard. We can [restore Elasticsearch indices from snapshots](elasticsearch-dumps.html) relatively quickly compared to a full reindex.
+Amazon OpenSearch Service [takes hourly snapshots](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore) of OpenSearch indices as standard. We can [restore OpenSearch indices from snapshots](opensearch-dumps.html) relatively quickly compared to a full reindex.

source/manual/databases.html.md

@@ -12,7 +12,7 @@ GOV.UK uses mostly PostgreSQL and some MySQL for apps that require a relational
 
 GOV.UK uses MongoDB or DocumentDB for apps that require a document database. Whilst there are some [big differences between the two](https://www.mongodb.com/atlas-vs-amazon-documentdb), they're broadly compatible, and we tend to use the term "Mongo" to apply to both.
 
-Finally, GOV.UK uses an ElasticSearch database for search.
+Finally, GOV.UK uses an OpenSearch database for search.
 
 ## Hosting
 
@@ -22,7 +22,7 @@ Each Postgres and MySQL database runs in its own RDS instance. Whilst RDS instan
 
 Mongo databases are hosted either in DocumentDB clusters (managed by AWS) or MongoDB clusters (managed by us on self-hosted EC2 instances). On production, there is currently one DocumentDB cluster for Licensify and one 'shared' DocumentDB cluster, each with three instances. There is also one Mongo cluster of three EC2s. We have agreed that [we should move apps from MongoDB to DocumentDB](/repos/govuk-aws/architecture/decisions/0038-mongo_replacement_by_documentdb.html).
 
-ElasticSearch is hosted in [AWS's OpenSearch service](https://eu-west-1.console.aws.amazon.com/esv3/home?region=eu-west-1#opensearch/domains). It has [two types of node](https://github.com/alphagov/govuk-aws/blob/6b5f78824bb14f5f6aaa7f7d269915b7831a13c3/terraform/projects/app-elasticsearch6/main.tf#L186-L193) - "master" and "data" (instance).
+OpenSearch is hosted in [AWS's OpenSearch service](https://eu-west-1.console.aws.amazon.com/aos/home?region=eu-west-1#opensearch/dashboard). It has [two types of node](https://github.com/alphagov/govuk-infrastructure/blob/a4e63cec8ebf07116c2cf4d49165c66601ed37f8/terraform/variables/production/search-opensearch.tfvars#L8-L14) - "master" and "data" (instance).
 
 ## DB admin
 

source/manual/debug-issues-in-elasticsearch.html.md → source/manual/debug-issues-in-opensearch.html.md

@@ -1,6 +1,6 @@
 ---
 owner_slack: "#govuk-search"
-title: Debug issues in elasticsearch
+title: Debug issues in OpenSearch
 parent: "/manual.html"
 layout: manual_layout
 section: Search on GOV.UK
@@ -15,14 +15,14 @@ switch between the two using the [GOV.UK browser extension](https://github.com/a
 You can compare the data returned with the relevant publishing app or content store to check if it's up
 to date. An empty response means no documents match the given query.
 
-## Check the document is in elasticsearch
+## Check the document is in OpenSearch
 
 If the document is missing from the search API, you can check the search index itself to
-see if it is present and has the expected fields, by sending a request to elasticsearch
+see if it is present and has the expected fields, by sending a request to OpenSearch
 [from a shell][open-a-shell] in `search-api`:
 
 ```
-curl -X GET "$ELASTICSEARCH_URI/govuk/_search?pretty" -H "content-type: application/json" -d "
+curl -X GET "$OPENSEARCH_URI/govuk/_search?pretty" -H "content-type: application/json" -d "
 {
   \"query\": {
     \"match\": {

source/manual/docs-style-guide.html.md

@@ -54,13 +54,13 @@ Good example:
 
 > Title: Reboot a machine
 >
-> Subheadings: Redis, MongoDB, Elasticsearch
+> Subheadings: Redis, MongoDB, OpenSearch
 
 Bad example:
 
 > Title: Rebooting machines
 >
-> Subheadings: Rebooting Redis machines, Rebooting MongoDB machines, Rebooting Elasticsearch machines.
+> Subheadings: Rebooting Redis machines, Rebooting MongoDB machines, Rebooting OpenSearch machines.
 
 ## Writing style
 

source/manual/fix-out-of-date-search-indices.html.md

@@ -9,7 +9,7 @@ related_repos: [search-api]
 
 ## `govuk` index
 
-GOV.UK content in [Search API](/repos/search-api.html) is stored in the `govuk` index in Elasticsearch.
+GOV.UK content in [Search API](/repos/search-api.html) is stored in the `govuk` index in OpenSearch.
 This is populated from the [Publishing API message queue][queue]. This can get out of sync with publishing API,
 which affects any part of the site using it, including navigation pages and related links.
 This can happen after [restoring a backup][restore-backups] or [reindexing search][reindex-search].
@@ -36,7 +36,7 @@ updates this index. To update it manually, you can run the [associated rake task
 
 ## Environment syncs
 
-There are cron jobs that [synchronise elasticsearch][sync-job] environments. Staging is synced with production every night, and integration is synced with staging on a Monday morning.
+There are cron jobs that [synchronise OpenSearch][sync-job] environments. Staging is synced with production every night, and integration is synced with staging on a Monday morning.
 These jobs work in sequence to take a snapshot of a higher level environment (e.g. production) and then restore the snapshot in the lower level environment (e.g. staging), before the snapshot for that environment is taken.
 
 If documents that are published or unpublished in production are not reflected in staging after 1 day or in integration after 1 week, that
@@ -55,12 +55,12 @@ may wipe out changes that other developers are testing.
 
 For more information on environment syncs across GOV.UK see https://docs.publishing.service.gov.uk/manual/govuk-env-sync.html
 
-[restore-backups]: /manual/elasticsearch-dumps.html
-[reindex-search]: /manual/reindex-elasticsearch.html
+[restore-backups]: /manual/opensearch-dumps.html
+[reindex-search]: /manual/reindex-opensearch.html
 [queue]: https://github.com/alphagov/search-api/blob/main/docs/new-indexing-process.md
 [popularity-docs]: https://docs.publishing.service.gov.uk/repos/search-api/updating_popularity.html
 [popularity-job]: https://github.com/alphagov/govuk-helm-charts/blob/main/charts/app-config/values-production.yaml#L2972
 [popularity-rake-task]: https://github.com/alphagov/search-api/blob/main/lib/tasks/page_traffic.rake
 [sync-job]: https://github.com/alphagov/govuk-helm-charts/blob/main/charts/search-index-env-sync/values.yaml
 [argo]: https://argo.eks.production.govuk.digital/
-[snapshot-docs]: https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore
+[snapshot-docs]: https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/

source/manual/govuk-env-sync.html.md

@@ -36,7 +36,7 @@ $ kubectl logs -n apps db-backup-link-checker-api-postgres-29220583-abcde -c 0-r
 
 ## How it works
 
-The env sync process is made up of a lot of small Kubernetes cronjobs - one per app and environment - [configured in the 'db-backup' chart](https://github.com/alphagov/govuk-helm-charts/blob/main/charts/db-backup/values.yaml). There is also a [search-index-env-sync](https://github.com/alphagov/govuk-helm-charts/tree/main/charts/search-index-env-sync) chart for copying search API's elasticsearch index data.
+The env sync process is made up of a lot of small Kubernetes cronjobs - one per app and environment - [configured in the 'db-backup' chart](https://github.com/alphagov/govuk-helm-charts/blob/main/charts/db-backup/values.yaml). There is also a [search-index-env-sync](https://github.com/alphagov/govuk-helm-charts/tree/main/charts/search-index-env-sync) chart for copying search API's OpenSearch index data.
 
 The 'production' cronjobs back up their respective application's database data to a production S3 bucket called `govuk-production-database-backups` (using a 'backup' operation - see [example](https://github.com/alphagov/govuk-helm-charts/blob/4b922fc7eb79757080570d33b1ae668c4d9dbb4f/charts/db-backup/values.yaml#L107-L111)). The bucket is [configured to automatically replicate](https://github.com/alphagov/govuk-infrastructure/blob/4f451dd56d43042e3fe0477235e9f2126618c957/terraform/deployments/govuk-publishing-infrastructure/db_backup_s3.tf) to S3 buckets [in the other environments](https://github.com/alphagov/govuk-infrastructure/blob/4f451dd56d43042e3fe0477235e9f2126618c957/terraform/deployments/govuk-publishing-infrastructure/db_backup_iam.tf#L1-L8). You can read more about [how GOV.UK data backups are configured in AWS](/manual/backups.html).
 

source/manual/govuk-search.html.md

@@ -44,7 +44,7 @@ Documentation on how finder frontend queries the two search api applications is
 
 [search-api-v2][] was built to improve the quality of search results for the majority of GOV.UK users (when compared with search-api) and retains a "minimally compatible" API with search-api. It uses Google Cloud Platform (GCP)'s Agent Search ("Discovery Engine") product as its underlying search engine.
 
-[search-api][] uses an old version of Elasticsearch as its underlying search engine. There is currently no clear roadmap for retiring search-api.
+[search-api][] uses OpenSearch as its underlying search engine. There is currently no clear roadmap for retiring search-api.
 
 ## How content gets into search