From 81a36212a807fff1435d1d76ef38d8fbd7bdb383 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 17:42:46 +0200 Subject: [PATCH 01/53] =?UTF-8?q?Upgrade=20Elasticsearch=20from=202.6=20to?= =?UTF-8?q?=205.6.=20Test=20de=20cr=C3=A9ation=20de=20la=20branch=20remote?= =?UTF-8?q?=20sur=20un=20texte=20sans=20impact.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/README.md | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/scripts/README.md b/scripts/README.md index 9aa080106..88a29f035 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,4 +1,31 @@ -# ElasticSearch 2.4 update for search improvement +# ElasticSearch Upgrade from v2.6 to V5.6 for docker env. like dev one. + +The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. +The ElasticSearch 2.6 is still build and run but would not be the main ES for the service. It could be decommissioned after migration. +docker-compose file contains as well : + +- api building with integration of the requirements.txt for python pip installation (component updates) +- legacy ES 2.6 listening on port 9201 instead of 9200 : > elasticsearch26 +- redis no change +- postgresql no change +- ES 5.6 listening on port 9200 and 9300 (elasticsearch5.Dockerfile) > elasticsearch +- Logstash 5.6 for data migration purpose. (logstash5.Dockerfile) + +## migration steps + + 1. **build the environment :** ` docker-compose build ` + 2. **Launch the environment :** ` docker-compose up -d ` + 3. **Create the index and its mapping :** `./scripts/esUpdateMappings25.sh` + 4. **Data Migration :** `docker-compose run logstash5 /usr/share/logstash/bin/logstash -f /root/logstash.conf` + 5. **postgresql shema up to date:** `docker-compose exec api .build/venv/bin/alembic upgrade head` + +## post migration + +if all the process has been passed without issues. you can comment or remove in the docker-compose.yml services : +- logstash +- elasticsearch26 + +# ElasticSearch 2.6 update for search improvement Script has to be execute one time in order to be relevant with the code update. @@ -6,18 +33,16 @@ Script has to be execute one time in order to be relevant with the code update. ### script requirement : - it requires curl from the execution machine - - it will update elasticsearch on localhost:9200, if elasticsearch is on other server, please update script changing localhost by the correct ES url. - an elasticsearch plugin has to be install, in case of elasticsearch cluster - the plugin has to be install on each node - -To install manually the the plugin analysis-icu from elasticsearch server : + +## To install manually the the plugin analysis-icu from elasticsearch server : ``` ./bin/plugin install analysis-icu analysis-icu ``` - The installation process is : - Install the plugin, - restart ElasticSearch instance, @@ -25,7 +50,8 @@ The installation process is : - update document mapping (_mappings /espon) - start a full indexation. -To launch the script from the host side : +## Automatic update via shell script from the host server + ``` cd scripts/ ./esUpdateMapSettings.sh From 1fbe5031b9760606b08444278b3b83181d156ad0 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 17:53:07 +0200 Subject: [PATCH 02/53] Upgrade Elasticsearch from 2.6 to 5.6 - Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts --- .github/workflows/ci.yml | 3 +- c2corg_api/search/__init__.py | 2 +- c2corg_api/search/mapping.py | 56 +- c2corg_api/search/search.py | 1 - c2corg_api/search/search_filters.py | 19 +- .../tests/search/test_search_filters.py | 26 +- dataMigration/README.md | 6 + dataMigration/v5/logstash.conf | 51 + dataMigration/v6/logstash/logstash.conf | 45 + docker-compose.yml | 33 +- elasticsearch5.Dockerfile | 2 + logstash5.Dockerfile | 2 + requirements.txt | 4 +- scripts/esUpdateMapSettings25.sh | 52 + scripts/esjson2-5/a.json | 242 ++ scripts/esjson2-5/b.json | 246 ++ scripts/esjson2-5/c.json | 250 ++ scripts/esjson2-5/i.json | 257 ++ scripts/esjson2-5/m.json | 238 ++ scripts/esjson2-5/mappings.json | 2823 +++++++++++++++++ scripts/esjson2-5/o.json | 338 ++ scripts/esjson2-5/r.json | 369 +++ scripts/esjson2-5/settings.json | 378 +++ scripts/esjson2-5/u.json | 238 ++ scripts/esjson2-5/w.json | 349 ++ scripts/esjson2-5/x.json | 268 ++ 26 files changed, 6237 insertions(+), 61 deletions(-) create mode 100644 dataMigration/README.md create mode 100644 dataMigration/v5/logstash.conf create mode 100644 dataMigration/v6/logstash/logstash.conf create mode 100644 elasticsearch5.Dockerfile create mode 100644 logstash5.Dockerfile create mode 100755 scripts/esUpdateMapSettings25.sh create mode 100644 scripts/esjson2-5/a.json create mode 100644 scripts/esjson2-5/b.json create mode 100644 scripts/esjson2-5/c.json create mode 100644 scripts/esjson2-5/i.json create mode 100644 scripts/esjson2-5/m.json create mode 100644 scripts/esjson2-5/mappings.json create mode 100644 scripts/esjson2-5/o.json create mode 100644 scripts/esjson2-5/r.json create mode 100644 scripts/esjson2-5/settings.json create mode 100644 scripts/esjson2-5/u.json create mode 100644 scripts/esjson2-5/w.json create mode 100644 scripts/esjson2-5/x.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 72a72ad90..c1abaf517 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,8 +79,7 @@ jobs: - name: Run (old) elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.6/elasticsearch-2.4.6.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - /tmp/elasticsearch/bin/plugin install analysis-icu + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch --daemonize --path.data /tmp sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - run: curl -v http://localhost:9200 diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index d5ff092b4..d0307f735 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -118,10 +118,10 @@ def get_text_query_on_title(search_term, search_lang=None): query=search_term, fields=fields, type='phrase', - fuzziness=2, max_expansions=3, zero_terms_query="none", slop=4, + operator="and" ) diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index dd9b5bd23..c10b339ff 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -6,7 +6,7 @@ from c2corg_api.models.common.attributes import default_langs from c2corg_api.models.common.sortable_search_attributes import \ sortable_quality_types -from elasticsearch_dsl import DocType, String, MetaField, Long, GeoPoint +from elasticsearch_dsl import DocType, MetaField, Long, GeoPoint, Text, Keyword class BaseMeta: @@ -21,25 +21,25 @@ class BaseMeta: # https://github.com/komoot/photon/blob/master/es/index_settings.json def default_title_field(lang: None): if lang is None: - return String( + return Text( index='not_analyzed', - similarity='c2corgsimilarity', + #BM25 use by default by lucene not required : similarity='c2corgsimilarity', fields={ - 'ngram': String( + 'ngram': Text( analyzer='index_ngram', search_analyzer='search_ngram'), - 'raw': String( + 'raw': Text( analyzer='index_raw', search_analyzer='search_raw') }) else: - return String( + return Text( index='not_analyzed', - similarity='c2corgsimilarity', + #BM25 use by default by lucene not required : similarity='c2corgsimilarity', fields={ - 'ngram': String( + 'ngram': Text( analyzer='index_ngram', search_analyzer='search_ngram'), - 'raw': String( + 'raw': Text( analyzer='index_raw', search_analyzer='search_raw'), - 'contentheavy': String( + 'contentheavy': Text( analyzer='{0}_heavy'.format(lang)) }) @@ -74,65 +74,65 @@ class Meta(BaseMeta): # fr title_fr = default_title_field("french") - summary_fr = String( + summary_fr = Text( analyzer='index_french', search_analyzer='search_french') - description_fr = String( + description_fr = Text( analyzer='index_french', search_analyzer='search_french') # it title_it = default_title_field("italian") - summary_it = String( + summary_it = Text( analyzer='index_italian', search_analyzer='search_italian') - description_it = String( + description_it = Text( analyzer='index_italian', search_analyzer='search_italian') # de title_de = default_title_field("german") - summary_de = String( + summary_de = Text( analyzer='index_german', search_analyzer='search_german') - description_de = String( + description_de = Text( analyzer='index_german', search_analyzer='search_german') # en title_en = default_title_field("english") - summary_en = String( + summary_en = Text( analyzer='index_english', search_analyzer='search_english') - description_en = String( + description_en = Text( analyzer='index_english', search_analyzer='search_english') # es title_es = default_title_field("spanish") - summary_es = String( + summary_es = Text( analyzer='index_spanish', search_analyzer='search_spanish') - description_es = String( + description_es = Text( analyzer='index_spanish', search_analyzer='search_spanish') # ca title_ca = default_title_field("catalan") - summary_ca = String( + summary_ca = Text( analyzer='index_catalan', search_analyzer='search_catalan') - description_ca = String( + description_ca = Text( analyzer='index_catalan', search_analyzer='search_catalan') # eu title_eu = default_title_field("basque") - summary_eu = String( + summary_eu = Text( analyzer='index_basque', search_analyzer='search_basque') - description_eu = String( + description_eu = Text( analyzer='index_basque', search_analyzer='search_basque') # sl title_sl = default_title_field("slovene") - summary_sl = String( + summary_sl = Text( analyzer='index_slovene', search_analyzer='search_slovene') - description_sl = String( + description_sl = Text( analyzer='index_slovene', search_analyzer='search_slovene') # zh title_zh = default_title_field("chinois") - summary_zh = String( + summary_zh = Text( analyzer='index_chinois', search_analyzer='search_chinois') - description_zh = String( + description_zh = Text( analyzer='index_chinois', search_analyzer='search_chinois') @staticmethod diff --git a/c2corg_api/search/search.py b/c2corg_api/search/search.py index 7929caf1f..b5149391b 100644 --- a/c2corg_api/search/search.py +++ b/c2corg_api/search/search.py @@ -46,7 +46,6 @@ def do_multi_search_for_types(search_types, search_term, limit, lang): (_, get_documents_config) = search_type search = create_search(get_documents_config.document_type).\ query(get_text_query_on_title(search_term, lang)).\ - fields([]).\ extra(from_=0, size=limit) multi_search = multi_search.add(search) diff --git a/c2corg_api/search/search_filters.py b/c2corg_api/search/search_filters.py index 9fbd347b3..c66796697 100644 --- a/c2corg_api/search/search_filters.py +++ b/c2corg_api/search/search_filters.py @@ -8,8 +8,8 @@ from c2corg_api.search import (create_search, get_text_query_on_title, search_documents) from c2corg_api.search.mapping_types import reserved_query_fields -from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Missing, Range, - Script, Term, Terms) +from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Range, + Script, Term, Terms, Q) # Missing not more exist from pyproj import Transformer log = logging.getLogger(__name__) @@ -35,8 +35,7 @@ def build_query(url_params, meta_params, doc_type): if filter: search = search.filter(filter) - search = search.\ - fields([]).\ + search = search. \ extra(from_=offset, size=limit) if url_params.get('bbox'): @@ -171,9 +170,9 @@ def create_enum_range_min_max_filter(field, query_term): return Bool(must_not=Bool(should=[ Range(**kwargs_start), Range(**kwargs_end), - Bool(must=[ - Missing(field=field.field_min), - Missing(field=field.field_max) + Bool(must_not=[ + Q("exists", field=field.field_min), + Q("exists", field=field.field_max) ]) ])) @@ -351,9 +350,9 @@ def create_number_range_filter(field, query_term): return Bool(must_not=Bool(should=[ Range(**kwargs_start), Range(**kwargs_end), - Bool(must=[ - Missing(field=field.field_min), - Missing(field=field.field_max) + Bool(must_not=[ + Q('exists',field=field.field_min ), + Q('exists',field=field.field_max ) ]) ])) diff --git a/c2corg_api/tests/search/test_search_filters.py b/c2corg_api/tests/search/test_search_filters.py index 61308938d..707a0693b 100644 --- a/c2corg_api/tests/search/test_search_filters.py +++ b/c2corg_api/tests/search/test_search_filters.py @@ -6,8 +6,8 @@ from c2corg_api.search.mappings.outing_mapping import SearchOuting from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint from c2corg_api.tests import BaseTestCase -from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Missing, Range, - Script, Term, Terms) +from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Range, + Script, Term, Terms, Q) class AdvancedSearchTest(BaseTestCase): @@ -29,7 +29,6 @@ def test_build_query(self): filter(Term(available_locales='fr')).\ filter(Terms(areas=[1234, 4567])). \ filter(Range(elevation={'gte': 1500})). \ - fields([]).\ extra(from_=0, size=10) self.assertQueryEqual(query, expected_query) @@ -52,7 +51,6 @@ def test_build_query_bbox(self): 'left': 6.28279913, 'bottom': 46.03129072, 'right': 6.28369744, 'top': 46.03191439}, type='indexed')). \ - fields([]). \ extra(from_=0, size=10) self.assertQueryEqual(query, expected_query) @@ -67,7 +65,6 @@ def test_build_query_limit_offset(self): query = build_query(params, meta_params, 'w') expected_query = create_search('w'). \ query(get_text_query_on_title('search word')). \ - fields([]).\ extra(from_=40, size=20) self.assertQueryEqual(query, expected_query) @@ -82,7 +79,6 @@ def test_build_query_sort_outing(self): query = build_query(params, meta_params, 'o') expected_query = create_search('o'). \ filter(Term(activities='skitouring')). \ - fields([]). \ sort({'date_end': {'order': 'desc'}}, {'id': {'order': 'desc'}}). \ extra(from_=40, size=20) self.assertQueryEqual(query, expected_query) @@ -212,9 +208,9 @@ def test_create_filter_enum_range_min_max(self): Bool(must_not=Bool(should=[ Range(climbing_rating_min={'gt': 17}), Range(climbing_rating_max={'lt': 5}), - Bool(must=[ - Missing(field='climbing_rating_min'), - Missing(field='climbing_rating_max') + Bool(must_not=[ + Q("exists", field='climbing_rating_min'), + Q("exists", field='climbing_rating_max') ]) ]))) @@ -243,9 +239,9 @@ def test_create_filter_integer_range(self): Bool(must_not=Bool(should=[ Range(elevation_min={'gt': 2400}), Range(elevation_max={'lt': 1200}), - Bool(must=[ - Missing(field='elevation_min'), - Missing(field='elevation_max') + Bool(must_not=[ + Q("exists", field='elevation_min'), + Q("exists", field='elevation_max') ]) ]))) self.assertEqual( @@ -253,9 +249,9 @@ def test_create_filter_integer_range(self): Bool(must_not=Bool(should=[ Range(height_min={'gt': 2400}), Range(height_max={'lt': 1200}), - Bool(must=[ - Missing(field='height_min'), - Missing(field='height_max') + Bool(must_not=[ + Q("exists", field='height_min'), + Q("exists", field='height_max') ]) ]))) diff --git a/dataMigration/README.md b/dataMigration/README.md new file mode 100644 index 000000000..dc45c7a62 --- /dev/null +++ b/dataMigration/README.md @@ -0,0 +1,6 @@ +## dataMigraiton folder content logstash pipeline script per target release ES number. + +- v5: to move ES data from 2.6 original to v5.6 +- v6: to move ES data from v5.6 to v6.x +- v7: to move ES data from v6.X to 7.X + diff --git a/dataMigration/v5/logstash.conf b/dataMigration/v5/logstash.conf new file mode 100644 index 000000000..ea42dae95 --- /dev/null +++ b/dataMigration/v5/logstash.conf @@ -0,0 +1,51 @@ +#pre requisite : docker-compose run -it logstash5 /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune +input +{ + elasticsearch + { + hosts => ["elasticsearch26:9200"] + index => "c2corg" + docinfo => true + size => 1000 + scroll => "5m" + docinfo_target => "[@metadata]" + } +} + +filter { + if (![geom]) { + mutate { + add_field => { "geom" => [0,0] } + } + } + + mutate { + add_field => { "doc_id" => "%{[@metadata][_id]}" } + } + + #prune { + # whitelist_names => [ "doc_id" ] + #} + +} + +output +{ + elasticsearch + { + hosts => ["elasticsearch:9200"] + index => "c2corg" + document_type => "%{doc_type}" + document_id => "%{doc_id}" + } + + stdout { + codec => "dots" + } + +} + +#stdout { codec => "dots"} +#stdout { codec => rubydebug } +#stdout { codec => "json" } +#docker-compose run logstash5 /usr/share/logstash/bin/logstash -f /root/logstash.conf \ No newline at end of file diff --git a/dataMigration/v6/logstash/logstash.conf b/dataMigration/v6/logstash/logstash.conf new file mode 100644 index 000000000..9f23710ee --- /dev/null +++ b/dataMigration/v6/logstash/logstash.conf @@ -0,0 +1,45 @@ +input +{ + elasticsearch + { + hosts => ["elasticsearch56:9200"] + index => "c2corg" + docinfo => true + size => 1000 + scroll => "5m" + docinfo_target => "[@metadata]" + } +} + +filter { + if (![geom]) { + mutate { + add_field => { "geom" => [0,0] } + } + } + + mutate { + add_field => { "doc_id" => "%{[@metadata][_id]}" } + } +} + +output +{ + elasticsearch + { + hosts => ["elasticsearch6:9200"] + index => "c2corg_%{[@metadata][_type]}" + document_type => "%{doc_type}" + document_id => "%{doc_id}" + } + + stdout { + codec => "dots" + } + +} + +#stdout { codec => "dots"} +#stdout { codec => rubydebug } +#stdout { codec => "json" } +#docker-compose run logstash5 /usr/share/logstash/bin/logstash -f /root/logstash.conf \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index f1acb1cd1..1848772f6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -4,6 +4,7 @@ services: build: context: . dockerfile: dev_api.Dockerfile + network: host depends_on: - postgresql - elasticsearch @@ -25,6 +26,7 @@ services: - ./development.ini.in:/var/www/development.ini.in - ./test.ini.in:/var/www/test.ini.in - ./pytest.ini:/var/www/pytest.ini + - ./requirements.txt:/var/www/requirements.txt command: make -f config/docker-dev serve links: - postgresql @@ -39,12 +41,39 @@ services: - ./docker-compose/pgsql-settings.d/:/c2corg_anon/pgsql-settings.d/ - .:/v6_api - elasticsearch: + elasticsearch26: image: 'docker.io/c2corg/c2corg_es:anon-2018-11-02' + container_name: elasticsearch26 ports: - - 9200:9200 + - 9201:9200 command: -Des.index.number_of_replicas=0 -Des.path.data=/c2corg_anon -Des.script.inline=true + elasticsearch: + platform: linux/x86_64 + container_name: elasticsearch + build: + context: . + dockerfile: elasticsearch5.Dockerfile + network: host + environment: + - discovery.type=single-node + - cluster.initial_master_nodes="docker-cluster" + ports: + - 9200:9200 + - 9300:9300 + + logstash5: + platform: linux/x86_64 + name: logstash5 + build: + context: . + dockerfile: logstash5.Dockerfile + network: host + ports: + - 5044:5044 + volumes: + - ./dataMigration/v5/logstash.conf:/root/logstash.conf + redis: image: 'docker.io/redis:3.2' ports: diff --git a/elasticsearch5.Dockerfile b/elasticsearch5.Dockerfile new file mode 100644 index 000000000..f7cfc2fdf --- /dev/null +++ b/elasticsearch5.Dockerfile @@ -0,0 +1,2 @@ +FROM elasticsearch:5.6.10 +RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu diff --git a/logstash5.Dockerfile b/logstash5.Dockerfile new file mode 100644 index 000000000..73e24bb58 --- /dev/null +++ b/logstash5.Dockerfile @@ -0,0 +1,2 @@ +FROM --platform=linux/X86_64 logstash:5.6.16 +#RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/requirements.txt b/requirements.txt index 3ecd31164..cd0fde4c4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,8 +4,8 @@ bcrypt==4.1.2 bleach[css]==6.1.0 colander==2.0 dogpile.cache==1.3.2 -elasticsearch==2.4.1 -elasticsearch_dsl==2.2.0 +elasticsearch==6.0.0 #elasticsearch==2.4.1 +elasticsearch-dsl==6.0.1 #elasticsearch_dsl==2.2.0 geoalchemy2==0.4.2 geojson==3.1.0 geomet==1.1.0 diff --git a/scripts/esUpdateMapSettings25.sh b/scripts/esUpdateMapSettings25.sh new file mode 100755 index 000000000..3be5fbeeb --- /dev/null +++ b/scripts/esUpdateMapSettings25.sh @@ -0,0 +1,52 @@ +#!/bin/bash +#change localhost to the correct url if elasticsearch is not hosted on your computer, or in docker +#operate the plugin installation from the server side manually if not dockerized. +#installation du plugin es et reboot +#Install elastic plugin analysis-icu +#echo 'Installing es plugin to elasticsearch docker' +#docker-compose exec -it elasticsearch bin/plugin install analysis-icu +#echo 'Installing done - restarting es docker' +#docker-compose restart elasticsearch +#sleep 10 #in sec to let the docker restart +#1. index Creation +echo -e '\n index Creation' +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg +sleep 2 +#2. closing the index to upgrade settings +echo -e '\n closing index' +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg/_close +sleep 2 +#3. Updating analyser settings +echo -e '\n updating Settings' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/settings.json http://localhost:9200/c2corg/_settings +sleep 2 +#4. restarting the index +echo -e '\n restarting index' +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg/_open +sleep 2 +#5. Docs Mappings update +echo 'updating doc type mappings' +echo 'doc type : a' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/a.json http://localhost:9200/c2corg/_mapping/a +echo -e '\n doc type : b' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/b.json http://localhost:9200/c2corg/_mapping/b +echo -e '\n doc type : c' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/c.json http://localhost:9200/c2corg/_mapping/c +echo -e '\n doc type : i' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/i.json http://localhost:9200/c2corg/_mapping/i +echo -e '\n doc type : m' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/m.json http://localhost:9200/c2corg/_mapping/m +echo -e '\n doc type : o' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/o.json http://localhost:9200/c2corg/_mapping/o +echo -e '\n doc type : r' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/r.json http://localhost:9200/c2corg/_mapping/r +echo -e '\n doc type : u' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/u.json http://localhost:9200/c2corg/_mapping/u +echo -e '\n doc type : w' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/w.json http://localhost:9200/c2corg/_mapping/w +echo -e '\n doc type : x' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson2-5/x.json http://localhost:9200/c2corg/_mapping/x +sleep 2 +#6. RE-index the indice to apply mappings +echo -e '\n Indexing with new parameters - should take sometimes - pls wait server acknowlegment' +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg/_update_by_query?conflicts=proceed \ No newline at end of file diff --git a/scripts/esjson2-5/a.json b/scripts/esjson2-5/a.json new file mode 100644 index 000000000..86eac914d --- /dev/null +++ b/scripts/esjson2-5/a.json @@ -0,0 +1,242 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "area_type": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/b.json b/scripts/esjson2-5/b.json new file mode 100644 index 000000000..e7f8048e6 --- /dev/null +++ b/scripts/esjson2-5/b.json @@ -0,0 +1,246 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "book_types": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/c.json b/scripts/esjson2-5/c.json new file mode 100644 index 000000000..e80d94edb --- /dev/null +++ b/scripts/esjson2-5/c.json @@ -0,0 +1,250 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "article_categories": { + "type": "text", + "index": false + }, + "article_type": { + "type": "text", + "index": false + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/i.json b/scripts/esjson2-5/i.json new file mode 100644 index 000000000..6e18fd32d --- /dev/null +++ b/scripts/esjson2-5/i.json @@ -0,0 +1,257 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "categories": { + "type": "text", + "index": false + }, + "date_time": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "image_type": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/m.json b/scripts/esjson2-5/m.json new file mode 100644 index 000000000..4f8c89415 --- /dev/null +++ b/scripts/esjson2-5/m.json @@ -0,0 +1,238 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/mappings.json b/scripts/esjson2-5/mappings.json new file mode 100644 index 000000000..67c9fff5b --- /dev/null +++ b/scripts/esjson2-5/mappings.json @@ -0,0 +1,2823 @@ +{ + "c2corg": { + "mappings": { + "x": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "avalanche_level": { + "type": "integer" + }, + "avalanche_slope": { + "type": "integer" + }, + "date": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "event_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "nb_impacted": { + "type": "integer" + }, + "nb_participants": { + "type": "integer" + }, + "quality": { + "type": "integer" + }, + "severity": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "c": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "article_categories": { + "type": "text", + "index": "not_analyzed" + }, + "article_type": { + "type": "text", + "index": "not_analyzed" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "u": { + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "o": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "avalanche_signs": { + "type": "text", + "index": "not_analyzed" + }, + "condition_rating": { + "type": "integer" + }, + "date_end": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "date_start": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation_access": { + "type": "integer" + }, + "elevation_down_snow": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_up_snow": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "frequentation": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "glacier_rating": { + "type": "integer" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "length_total": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "public_transport": { + "type": "boolean" + }, + "quality": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "routes": { + "type": "long" + }, + "ski_rating": { + "type": "integer" + }, + "snow_quality": { + "type": "integer" + }, + "snow_quantity": { + "type": "integer" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "users": { + "type": "long" + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + }, + "m": { + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "b": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "book_types": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "w": { + "_all": { + "enabled": false + }, + "properties": { + "access_time": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "best_periods": { + "type": "text", + "index": "not_analyzed" + }, + "capacity": { + "type": "integer" + }, + "capacity_staffed": { + "type": "integer" + }, + "children_proof": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_indoor_types": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_outdoor_types": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_rating_max": { + "type": "integer" + }, + "climbing_rating_median": { + "type": "integer" + }, + "climbing_rating_min": { + "type": "integer" + }, + "climbing_styles": { + "type": "text", + "index": "not_analyzed" + }, + "custodianship": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "equipment_ratings": { + "type": "integer" + }, + "exposition_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "height_max": { + "type": "integer" + }, + "height_median": { + "type": "integer" + }, + "height_min": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "length": { + "type": "integer" + }, + "lift_access": { + "type": "boolean" + }, + "orientations": { + "type": "text", + "index": "not_analyzed" + }, + "paragliding_rating": { + "type": "integer" + }, + "product_types": { + "type": "text", + "index": "not_analyzed" + }, + "prominence": { + "type": "integer" + }, + "public_transportation_rating": { + "type": "text", + "index": "not_analyzed" + }, + "public_transportation_types": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "rain_proof": { + "type": "text", + "index": "not_analyzed" + }, + "rock_types": { + "type": "text", + "index": "not_analyzed" + }, + "routes_quantity": { + "type": "integer" + }, + "snow_clearance_rating": { + "type": "text", + "index": "not_analyzed" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "waypoint_type": { + "type": "text", + "index": "not_analyzed" + }, + "weather_station_types": { + "type": "text", + "index": "not_analyzed" + } + } + }, + "a": { + "_all": { + "enabled": false + }, + "properties": { + "area_type": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + } + } + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "i": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "categories": { + "type": "text", + "index": "not_analyzed" + }, + "date_time": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "image_type": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + } + } + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "r": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "aid_rating": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_outdoor_type": { + "type": "text", + "index": "not_analyzed" + }, + "configuration": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "difficulties_height": { + "type": "integer" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "durations": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_min": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "exposition_rock_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "glacier_gear": { + "type": "text", + "index": "not_analyzed" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_access": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_down": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_mtb_exposition": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "labande_ski_rating": { + "type": "integer" + }, + "mixed_rating": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_height_diff_portages": { + "type": "integer" + }, + "mtb_length_asphalt": { + "type": "integer" + }, + "mtb_length_trail": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "orientations": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "risk_rating": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "rock_required_rating": { + "type": "integer" + }, + "rock_types": { + "type": "text", + "index": "not_analyzed" + }, + "route_length": { + "type": "integer" + }, + "route_types": { + "type": "text", + "index": "not_analyzed" + }, + "ski_exposition": { + "type": "integer" + }, + "ski_rating": { + "type": "integer" + }, + "slackline_type": { + "type": "text", + "index": "not_analyzed" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } + } + } +} diff --git a/scripts/esjson2-5/o.json b/scripts/esjson2-5/o.json new file mode 100644 index 000000000..2bab0d646 --- /dev/null +++ b/scripts/esjson2-5/o.json @@ -0,0 +1,338 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "avalanche_signs": { + "type": "text", + "index": false + }, + "condition_rating": { + "type": "integer" + }, + "date_end": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "date_start": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation_access": { + "type": "integer" + }, + "elevation_down_snow": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_up_snow": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "frequentation": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "glacier_rating": { + "type": "integer" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "length_total": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "public_transport": { + "type": "boolean" + }, + "quality": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "routes": { + "type": "long" + }, + "ski_rating": { + "type": "integer" + }, + "snow_quality": { + "type": "integer" + }, + "snow_quantity": { + "type": "integer" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "users": { + "type": "long" + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/r.json b/scripts/esjson2-5/r.json new file mode 100644 index 000000000..6548b4bcf --- /dev/null +++ b/scripts/esjson2-5/r.json @@ -0,0 +1,369 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": false + }, + "aid_rating": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "climbing_outdoor_type": { + "type": "text", + "index": false + }, + "configuration": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "difficulties_height": { + "type": "integer" + }, + "doc_type": { + "type": "text", + "index": false + }, + "durations": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_min": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "exposition_rock_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "glacier_gear": { + "type": "text", + "index": false + }, + "global_rating": { + "type": "integer" + }, + "height_diff_access": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_down": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_mtb_exposition": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "labande_ski_rating": { + "type": "integer" + }, + "mixed_rating": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_height_diff_portages": { + "type": "integer" + }, + "mtb_length_asphalt": { + "type": "integer" + }, + "mtb_length_trail": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "orientations": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "risk_rating": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "rock_required_rating": { + "type": "integer" + }, + "rock_types": { + "type": "text", + "index": false + }, + "route_length": { + "type": "integer" + }, + "route_types": { + "type": "text", + "index": false + }, + "ski_exposition": { + "type": "integer" + }, + "ski_rating": { + "type": "integer" + }, + "slackline_type": { + "type": "text", + "index": false + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/settings.json b/scripts/esjson2-5/settings.json new file mode 100644 index 000000000..ff95e3ef8 --- /dev/null +++ b/scripts/esjson2-5/settings.json @@ -0,0 +1,378 @@ +{ + "analysis": { + "filter": { + "autocomplete_filter": { + "type": "edge_ngram", + "min_gram": "2", + "max_gram": "15", + "token_chars": [ + "letter", + "digit" + ] + }, + "english_stop": { + "type": "stop", + "stopwords": "_english_" + }, + "english_stemmer": { + "type": "stemmer", + "language": "english" + }, + "english_possessive_stemmer": { + "type": "stemmer", + "language": "possessive_english" + }, + "french_elision": { + "type": "elision", + "articles_case": true, + "articles": [ + "l", "m", "t", "qu", "n", "s", + "j", "d", "c", "jusqu", "quoiqu", + "lorsqu", "puisqu" + ] + }, + "french_stop": { + "type": "stop", + "stopwords": "_french_" + }, + "french_stemmer": { + "type": "stemmer", + "language": "light_french" + }, + "german_stop": { + "type": "stop", + "stopwords": "_german_" + }, + "german_stemmer": { + "type": "stemmer", + "language": "light_german" + }, + "italian_elision": { + "type": "elision", + "articles_case": true, + "articles": [ + "c", "l", "all", "dall", "dell", + "nell", "sull", "coll", "pell", + "gl", "agl", "dagl", "degl", "negl", + "sugl", "un", "m", "t", "s", "v", "d" + ] + }, + "italian_stop": { + "type": "stop", + "stopwords": "_italian_" + }, + "italian_stemmer": { + "type": "stemmer", + "language": "light_italian" + }, + "spanish_stop": { + "type": "stop", + "stopwords": "_spanish_" + }, + "spanish_stemmer": { + "type": "stemmer", + "language": "light_spanish" + }, + "catalan_elision": { + "type": "elision", + "articles_case": true, + "articles": ["d", "l", "m", "n", "s", "t"] + }, + "catalan_stop": { + "type": "stop", + "stopwords": "_catalan_" + }, + "catalan_stemmer": { + "type": "stemmer", + "language": "catalan" + }, + "basque_stop": { + "type": "stop", + "stopwords": "_basque_" + }, + "basque_stemmer": { + "type": "stemmer", + "language": "basque" + } + }, + "similarity": { + "c2corgsimilarity": { + "type": "BM25" + } + }, + "char_filter": { + "punctuationgreedy": { + "type": "pattern_replace", + "pattern": "[\\.,]" + } + }, + "analyzer": { + "index_ngram": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "icu_folding", + "autocomplete_filter"], + "tokenizer": "icu_tokenizer" + }, + "search_ngram": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "index_raw": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "search_raw": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "index_english": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "english_possessive_stemmer", + "lowercase", + "english_stop", + "english_stemmer", + "autocomplete_filter" + ] + }, + "search_english": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "english_possessive_stemmer", + "lowercase", + "english_stop", + "english_stemmer" + ] + }, + "index_french": { + "tokenizer": "standard", + "filter": [ + "french_elision", + "lowercase", + "french_stop", + "french_stemmer", + "autocomplete_filter" + ] + }, + "search_french": { + "tokenizer": "standard", + "filter": [ + "french_elision", + "lowercase", + "french_stop", + "french_stemmer", + "autocomplete_filter" + ] + }, + "index_german": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "german_stop", + "german_normalization", + "german_stemmer", + "autocomplete_filter" + ] + }, + "search_german": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "german_stop", + "german_normalization", + "german_stemmer" + ] + }, + "index_italian": { + "tokenizer": "standard", + "filter": [ + "italian_elision", + "lowercase", + "italian_stop", + "italian_stemmer", + "autocomplete_filter" + ] + }, + "search_italian": { + "tokenizer": "standard", + "filter": [ + "italian_elision", + "lowercase", + "italian_stop", + "italian_stemmer" + ] + }, + "index_spanish": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer", + "autocomplete_filter" + ] + }, + "search_spanish": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer" + ] + }, + "index_catalan": { + "tokenizer": "standard", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer", + "autocomplete_filter" + ] + }, + "search_catalan": { + "tokenizer": "standard", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer" + ] + }, + "index_basque": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer", + "autocomplete_filter" + ] + }, + "search_basque": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer" + ] + }, + "french_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "french_elision", + "french_stop", + "icu_folding", + "lowercase", + "french_stemmer" + ] + }, + "german_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "german_stop", + "german_stemmer", + "lowercase", + "icu_folding" + ] + }, + "english_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "english_possessive_stemmer", + "english_stop", + "lowercase", + "icu_folding" + ] + }, + "italian_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "italian_elision", + "italian_stop", + "lowercase", + "icu_folding", + "italian_stemmer" + ] + }, + "spanish_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer", + "icu_folding" + ] + }, + "catalan_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer", + "icu_folding" + ] + }, + "basque_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer", + "icu_folding" + ] + }, + "index_slovene": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "autocomplete_filter", + "lowercase" + ] + }, + "search_slovene": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase" + ] + }, + "slovene_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "icu_folding" + ] + }, + "index_chinois": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "autocomplete_filter" + ] + }, + "search_chinois": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase" + ] + }, + "chinois_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "icu_folding" + ] + } + } + } +} \ No newline at end of file diff --git a/scripts/esjson2-5/u.json b/scripts/esjson2-5/u.json new file mode 100644 index 000000000..4f8c89415 --- /dev/null +++ b/scripts/esjson2-5/u.json @@ -0,0 +1,238 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/w.json b/scripts/esjson2-5/w.json new file mode 100644 index 000000000..503b9a018 --- /dev/null +++ b/scripts/esjson2-5/w.json @@ -0,0 +1,349 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "access_time": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "best_periods": { + "type": "text", + "index": false + }, + "capacity": { + "type": "integer" + }, + "capacity_staffed": { + "type": "integer" + }, + "children_proof": { + "type": "text", + "index": false + }, + "climbing_indoor_types": { + "type": "text", + "index": false + }, + "climbing_outdoor_types": { + "type": "text", + "index": false + }, + "climbing_rating_max": { + "type": "integer" + }, + "climbing_rating_median": { + "type": "integer" + }, + "climbing_rating_min": { + "type": "integer" + }, + "climbing_styles": { + "type": "text", + "index": false + }, + "custodianship": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "equipment_ratings": { + "type": "integer" + }, + "exposition_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "height_max": { + "type": "integer" + }, + "height_median": { + "type": "integer" + }, + "height_min": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "length": { + "type": "integer" + }, + "lift_access": { + "type": "boolean" + }, + "orientations": { + "type": "text", + "index": false + }, + "paragliding_rating": { + "type": "integer" + }, + "product_types": { + "type": "text", + "index": false + }, + "prominence": { + "type": "integer" + }, + "public_transportation_rating": { + "type": "text", + "index": false + }, + "public_transportation_types": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "rain_proof": { + "type": "text", + "index": false + }, + "rock_types": { + "type": "text", + "index": false + }, + "routes_quantity": { + "type": "integer" + }, + "snow_clearance_rating": { + "type": "text", + "index": false + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "waypoint_type": { + "type": "text", + "index": false + }, + "weather_station_types": { + "type": "text", + "index": false + } + } + } \ No newline at end of file diff --git a/scripts/esjson2-5/x.json b/scripts/esjson2-5/x.json new file mode 100644 index 000000000..dd419c007 --- /dev/null +++ b/scripts/esjson2-5/x.json @@ -0,0 +1,268 @@ +{ + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "avalanche_level": { + "type": "integer" + }, + "avalanche_slope": { + "type": "integer" + }, + "date": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "event_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "nb_impacted": { + "type": "integer" + }, + "nb_participants": { + "type": "integer" + }, + "quality": { + "type": "integer" + }, + "severity": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file From 5a77c75a4d8d52db5cbd8d3ce9de089f873cc487 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 18:00:23 +0200 Subject: [PATCH 03/53] Upgrade Elasticsearch from 2.6 to 5.6 - Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts --- c2corg_api/search/mapping.py | 4 +--- c2corg_api/search/search_filters.py | 7 +++---- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index c10b339ff..29dda3a23 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -6,7 +6,7 @@ from c2corg_api.models.common.attributes import default_langs from c2corg_api.models.common.sortable_search_attributes import \ sortable_quality_types -from elasticsearch_dsl import DocType, MetaField, Long, GeoPoint, Text, Keyword +from elasticsearch_dsl import DocType, MetaField, Long, GeoPoint, Text class BaseMeta: @@ -23,7 +23,6 @@ def default_title_field(lang: None): if lang is None: return Text( index='not_analyzed', - #BM25 use by default by lucene not required : similarity='c2corgsimilarity', fields={ 'ngram': Text( analyzer='index_ngram', search_analyzer='search_ngram'), @@ -33,7 +32,6 @@ def default_title_field(lang: None): else: return Text( index='not_analyzed', - #BM25 use by default by lucene not required : similarity='c2corgsimilarity', fields={ 'ngram': Text( analyzer='index_ngram', search_analyzer='search_ngram'), diff --git a/c2corg_api/search/search_filters.py b/c2corg_api/search/search_filters.py index c66796697..b4dfdb2b6 100644 --- a/c2corg_api/search/search_filters.py +++ b/c2corg_api/search/search_filters.py @@ -8,8 +8,7 @@ from c2corg_api.search import (create_search, get_text_query_on_title, search_documents) from c2corg_api.search.mapping_types import reserved_query_fields -from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Range, - Script, Term, Terms, Q) # Missing not more exist +from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Range, Script, Term, Terms, Q) from pyproj import Transformer log = logging.getLogger(__name__) @@ -351,8 +350,8 @@ def create_number_range_filter(field, query_term): Range(**kwargs_start), Range(**kwargs_end), Bool(must_not=[ - Q('exists',field=field.field_min ), - Q('exists',field=field.field_max ) + Q('exists', field=field.field_min), + Q('exists', field=field.field_max) ]) ])) From 73e2075d14d1fc55b3114a07453a499d78f377d1 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 18:03:08 +0200 Subject: [PATCH 04/53] Upgrade Elasticsearch from 2.6 to 5.6 - Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts --- c2corg_api/search/search_filters.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/c2corg_api/search/search_filters.py b/c2corg_api/search/search_filters.py index b4dfdb2b6..d435f979f 100644 --- a/c2corg_api/search/search_filters.py +++ b/c2corg_api/search/search_filters.py @@ -8,7 +8,8 @@ from c2corg_api.search import (create_search, get_text_query_on_title, search_documents) from c2corg_api.search.mapping_types import reserved_query_fields -from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Range, Script, Term, Terms, Q) +from elasticsearch_dsl.query import (Bool, GeoBoundingBox, Range, + Script, Term, Terms, Q) from pyproj import Transformer log = logging.getLogger(__name__) From 600afaddeb3d0ead9d3d8ca1f1e23232b1fa0c81 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 18:18:42 +0200 Subject: [PATCH 05/53] Upgrade Elasticsearch from 2.6 to 5.6 - Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c1abaf517..0ec21e17e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,7 +80,7 @@ jobs: run: | mkdir /tmp/elasticsearch wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - /tmp/elasticsearch/bin/elasticsearch --daemonize --path.data /tmp + /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - run: curl -v http://localhost:9200 - name: Run tests From 359cb30686c1b575f6c68fea3034c6c466909b20 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 18:23:53 +0200 Subject: [PATCH 06/53] Upgrade Elasticsearch from 2.6 to 5.6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI: - retirer dans l'installation de ES 5.6 les paramètres inconsistant + install plugin analysis-icu sur ES --- .github/workflows/ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0ec21e17e..3b33f9e08 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,6 +80,7 @@ jobs: run: | mkdir /tmp/elasticsearch wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + /tmp/elasticsearch/bin/plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - run: curl -v http://localhost:9200 From d2f90c72045156aeba286d07b07b46cc6293bc2e Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 22 May 2024 18:26:31 +0200 Subject: [PATCH 07/53] Upgrade Elasticsearch from 2.6 to 5.6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI: - retirer dans l'installation de ES 5.6 les paramètres inconsistant + install plugin analysis-icu sur ES --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3b33f9e08..5feddc541 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,7 +80,7 @@ jobs: run: | mkdir /tmp/elasticsearch wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - /tmp/elasticsearch/bin/plugin install analysis-icu + /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - run: curl -v http://localhost:9200 From 5d7ef8eb04672eb51c525df782e9802ea64304ee Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 5 Jun 2024 14:33:40 +0200 Subject: [PATCH 08/53] Upgrade Elasticsearch from 5.6 to 6.8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Script de création / mapping / settings des indices ES --- scripts/esUpdateMapSettings56.sh | 92 + scripts/esjson5-6/a.json | 239 +++ scripts/esjson5-6/b.json | 243 +++ scripts/esjson5-6/c.json | 247 +++ scripts/esjson5-6/cluster.json | 10 + scripts/esjson5-6/i.json | 254 +++ scripts/esjson5-6/m.json | 235 +++ scripts/esjson5-6/mappings.json | 2823 ++++++++++++++++++++++++++++++ scripts/esjson5-6/o.json | 335 ++++ scripts/esjson5-6/r.json | 366 ++++ scripts/esjson5-6/settings.json | 378 ++++ scripts/esjson5-6/u.json | 235 +++ scripts/esjson5-6/w.json | 346 ++++ scripts/esjson5-6/x.json | 265 +++ scripts/pycurl-example.py | 30 + 15 files changed, 6098 insertions(+) create mode 100755 scripts/esUpdateMapSettings56.sh create mode 100644 scripts/esjson5-6/a.json create mode 100644 scripts/esjson5-6/b.json create mode 100644 scripts/esjson5-6/c.json create mode 100644 scripts/esjson5-6/cluster.json create mode 100644 scripts/esjson5-6/i.json create mode 100644 scripts/esjson5-6/m.json create mode 100644 scripts/esjson5-6/mappings.json create mode 100644 scripts/esjson5-6/o.json create mode 100644 scripts/esjson5-6/r.json create mode 100644 scripts/esjson5-6/settings.json create mode 100644 scripts/esjson5-6/u.json create mode 100644 scripts/esjson5-6/w.json create mode 100644 scripts/esjson5-6/x.json create mode 100755 scripts/pycurl-example.py diff --git a/scripts/esUpdateMapSettings56.sh b/scripts/esUpdateMapSettings56.sh new file mode 100755 index 000000000..c71d99ff3 --- /dev/null +++ b/scripts/esUpdateMapSettings56.sh @@ -0,0 +1,92 @@ +#!/bin/bash +#change localhost to the correct url if elasticsearch is not hosted on your computer, or in docker +#operate the plugin installation from the server side manually if not dockerized. +#installation du plugin es et reboot +#Install elastic plugin analysis-icu +#echo 'Installing es plugin to elasticsearch docker' +#docker-compose exec -it elasticsearch bin/plugin install analysis-icu +#echo 'Installing done - restarting es docker' +#docker-compose restart elasticsearch +#sleep 10 #in sec to let the docker restart + +#efface tous les indices présents +curl -X DELETE "http://192.168.1.120:9203/c2corg_*" + +#change cluster definition des shards +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/cluster.json http://192.168.1.120:9203/_template/index_defaults + +#créer les index +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_b +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_c +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_i +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_m +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_o +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_r +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_u +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_w +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_x +sleep 2 + +#2. closing the index to upgrade settings +echo -e '\n closing index' +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_b/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_c/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_i/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_m/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_o/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_r/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_u/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_w/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_x/_close +sleep 2 +#3. Updating analyser settings +echo -e '\n updating Settings' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_a/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_b/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_c/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_i/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_m/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_o/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_r/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_u/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_w/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_x/_settings +sleep 2 +#4. restarting the index +echo -e '\n restarting index' +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_b/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_c/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_i/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_m/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_o/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_r/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_u/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_w/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_x/_open +sleep 2 +#5. Docs Mappings update +echo 'updating doc type mappings' +echo 'doc type : a' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/a.json http://192.168.1.120:9203/c2corg_a/_mapping/_doc +echo -e '\n doc type : b' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/b.json http://192.168.1.120:9203/c2corg_b/_mapping/_doc +echo -e '\n doc type : c' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/c.json http://192.168.1.120:9203/c2corg_c/_mapping/_doc +echo -e '\n doc type : i' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/i.json http://192.168.1.120:9203/c2corg_i/_mapping/_doc +echo -e '\n doc type : m' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/m.json http://192.168.1.120:9203/c2corg_m/_mapping/_doc +echo -e '\n doc type : o' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/o.json http://192.168.1.120:9203/c2corg_o/_mapping/_doc +echo -e '\n doc type : r' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/r.json http://192.168.1.120:9203/c2corg_r/_mapping/_doc +echo -e '\n doc type : u' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/u.json http://192.168.1.120:9203/c2corg_u/_mapping/_doc +echo -e '\n doc type : w' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/w.json http://192.168.1.120:9203/c2corg_w/_mapping/_doc +echo -e '\n doc type : x' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/x.json http://192.168.1.120:9203/c2corg_x/_mapping/_doc +sleep 2 diff --git a/scripts/esjson5-6/a.json b/scripts/esjson5-6/a.json new file mode 100644 index 000000000..ef8a0d314 --- /dev/null +++ b/scripts/esjson5-6/a.json @@ -0,0 +1,239 @@ +{ + "properties": { + "area_type": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/b.json b/scripts/esjson5-6/b.json new file mode 100644 index 000000000..bc8b37dab --- /dev/null +++ b/scripts/esjson5-6/b.json @@ -0,0 +1,243 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "book_types": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/c.json b/scripts/esjson5-6/c.json new file mode 100644 index 000000000..91f8ee2a9 --- /dev/null +++ b/scripts/esjson5-6/c.json @@ -0,0 +1,247 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "article_categories": { + "type": "text", + "index": false + }, + "article_type": { + "type": "text", + "index": false + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/cluster.json b/scripts/esjson5-6/cluster.json new file mode 100644 index 000000000..4e2491d7e --- /dev/null +++ b/scripts/esjson5-6/cluster.json @@ -0,0 +1,10 @@ +{ + "index_patterns": [ + "*" + ], + "order": 0, + "settings": { + "number_of_shards": 1, + "number_of_replicas": 0 + } +} \ No newline at end of file diff --git a/scripts/esjson5-6/i.json b/scripts/esjson5-6/i.json new file mode 100644 index 000000000..f32de471d --- /dev/null +++ b/scripts/esjson5-6/i.json @@ -0,0 +1,254 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "categories": { + "type": "text", + "index": false + }, + "date_time": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "image_type": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/m.json b/scripts/esjson5-6/m.json new file mode 100644 index 000000000..1a89299b1 --- /dev/null +++ b/scripts/esjson5-6/m.json @@ -0,0 +1,235 @@ +{ + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/mappings.json b/scripts/esjson5-6/mappings.json new file mode 100644 index 000000000..67c9fff5b --- /dev/null +++ b/scripts/esjson5-6/mappings.json @@ -0,0 +1,2823 @@ +{ + "c2corg": { + "mappings": { + "x": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "avalanche_level": { + "type": "integer" + }, + "avalanche_slope": { + "type": "integer" + }, + "date": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "event_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "nb_impacted": { + "type": "integer" + }, + "nb_participants": { + "type": "integer" + }, + "quality": { + "type": "integer" + }, + "severity": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "c": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "article_categories": { + "type": "text", + "index": "not_analyzed" + }, + "article_type": { + "type": "text", + "index": "not_analyzed" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "u": { + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "o": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "avalanche_signs": { + "type": "text", + "index": "not_analyzed" + }, + "condition_rating": { + "type": "integer" + }, + "date_end": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "date_start": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation_access": { + "type": "integer" + }, + "elevation_down_snow": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_up_snow": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "frequentation": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "glacier_rating": { + "type": "integer" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "length_total": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "public_transport": { + "type": "boolean" + }, + "quality": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "routes": { + "type": "long" + }, + "ski_rating": { + "type": "integer" + }, + "snow_quality": { + "type": "integer" + }, + "snow_quantity": { + "type": "integer" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "users": { + "type": "long" + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + }, + "m": { + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "b": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "book_types": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "w": { + "_all": { + "enabled": false + }, + "properties": { + "access_time": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "best_periods": { + "type": "text", + "index": "not_analyzed" + }, + "capacity": { + "type": "integer" + }, + "capacity_staffed": { + "type": "integer" + }, + "children_proof": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_indoor_types": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_outdoor_types": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_rating_max": { + "type": "integer" + }, + "climbing_rating_median": { + "type": "integer" + }, + "climbing_rating_min": { + "type": "integer" + }, + "climbing_styles": { + "type": "text", + "index": "not_analyzed" + }, + "custodianship": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "equipment_ratings": { + "type": "integer" + }, + "exposition_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "height_max": { + "type": "integer" + }, + "height_median": { + "type": "integer" + }, + "height_min": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "length": { + "type": "integer" + }, + "lift_access": { + "type": "boolean" + }, + "orientations": { + "type": "text", + "index": "not_analyzed" + }, + "paragliding_rating": { + "type": "integer" + }, + "product_types": { + "type": "text", + "index": "not_analyzed" + }, + "prominence": { + "type": "integer" + }, + "public_transportation_rating": { + "type": "text", + "index": "not_analyzed" + }, + "public_transportation_types": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "rain_proof": { + "type": "text", + "index": "not_analyzed" + }, + "rock_types": { + "type": "text", + "index": "not_analyzed" + }, + "routes_quantity": { + "type": "integer" + }, + "snow_clearance_rating": { + "type": "text", + "index": "not_analyzed" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "waypoint_type": { + "type": "text", + "index": "not_analyzed" + }, + "weather_station_types": { + "type": "text", + "index": "not_analyzed" + } + } + }, + "a": { + "_all": { + "enabled": false + }, + "properties": { + "area_type": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + } + } + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "i": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "categories": { + "type": "text", + "index": "not_analyzed" + }, + "date_time": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "image_type": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + } + } + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "r": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "aid_rating": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_outdoor_type": { + "type": "text", + "index": "not_analyzed" + }, + "configuration": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "difficulties_height": { + "type": "integer" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "durations": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_min": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "exposition_rock_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "glacier_gear": { + "type": "text", + "index": "not_analyzed" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_access": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_down": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_mtb_exposition": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "labande_ski_rating": { + "type": "integer" + }, + "mixed_rating": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_height_diff_portages": { + "type": "integer" + }, + "mtb_length_asphalt": { + "type": "integer" + }, + "mtb_length_trail": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "orientations": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "risk_rating": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "rock_required_rating": { + "type": "integer" + }, + "rock_types": { + "type": "text", + "index": "not_analyzed" + }, + "route_length": { + "type": "integer" + }, + "route_types": { + "type": "text", + "index": "not_analyzed" + }, + "ski_exposition": { + "type": "integer" + }, + "ski_rating": { + "type": "integer" + }, + "slackline_type": { + "type": "text", + "index": "not_analyzed" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } + } + } +} diff --git a/scripts/esjson5-6/o.json b/scripts/esjson5-6/o.json new file mode 100644 index 000000000..9100ccce4 --- /dev/null +++ b/scripts/esjson5-6/o.json @@ -0,0 +1,335 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "avalanche_signs": { + "type": "text", + "index": false + }, + "condition_rating": { + "type": "integer" + }, + "date_end": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "date_start": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation_access": { + "type": "integer" + }, + "elevation_down_snow": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_up_snow": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "frequentation": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "glacier_rating": { + "type": "integer" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "length_total": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "public_transport": { + "type": "boolean" + }, + "quality": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "routes": { + "type": "long" + }, + "ski_rating": { + "type": "integer" + }, + "snow_quality": { + "type": "integer" + }, + "snow_quantity": { + "type": "integer" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "users": { + "type": "long" + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/r.json b/scripts/esjson5-6/r.json new file mode 100644 index 000000000..d7384a9d5 --- /dev/null +++ b/scripts/esjson5-6/r.json @@ -0,0 +1,366 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": false + }, + "aid_rating": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "climbing_outdoor_type": { + "type": "text", + "index": false + }, + "configuration": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "difficulties_height": { + "type": "integer" + }, + "doc_type": { + "type": "text", + "index": false + }, + "durations": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_min": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "exposition_rock_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "glacier_gear": { + "type": "text", + "index": false + }, + "global_rating": { + "type": "integer" + }, + "height_diff_access": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_down": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_mtb_exposition": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "labande_ski_rating": { + "type": "integer" + }, + "mixed_rating": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_height_diff_portages": { + "type": "integer" + }, + "mtb_length_asphalt": { + "type": "integer" + }, + "mtb_length_trail": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "orientations": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "risk_rating": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "rock_required_rating": { + "type": "integer" + }, + "rock_types": { + "type": "text", + "index": false + }, + "route_length": { + "type": "integer" + }, + "route_types": { + "type": "text", + "index": false + }, + "ski_exposition": { + "type": "integer" + }, + "ski_rating": { + "type": "integer" + }, + "slackline_type": { + "type": "text", + "index": false + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/settings.json b/scripts/esjson5-6/settings.json new file mode 100644 index 000000000..ff95e3ef8 --- /dev/null +++ b/scripts/esjson5-6/settings.json @@ -0,0 +1,378 @@ +{ + "analysis": { + "filter": { + "autocomplete_filter": { + "type": "edge_ngram", + "min_gram": "2", + "max_gram": "15", + "token_chars": [ + "letter", + "digit" + ] + }, + "english_stop": { + "type": "stop", + "stopwords": "_english_" + }, + "english_stemmer": { + "type": "stemmer", + "language": "english" + }, + "english_possessive_stemmer": { + "type": "stemmer", + "language": "possessive_english" + }, + "french_elision": { + "type": "elision", + "articles_case": true, + "articles": [ + "l", "m", "t", "qu", "n", "s", + "j", "d", "c", "jusqu", "quoiqu", + "lorsqu", "puisqu" + ] + }, + "french_stop": { + "type": "stop", + "stopwords": "_french_" + }, + "french_stemmer": { + "type": "stemmer", + "language": "light_french" + }, + "german_stop": { + "type": "stop", + "stopwords": "_german_" + }, + "german_stemmer": { + "type": "stemmer", + "language": "light_german" + }, + "italian_elision": { + "type": "elision", + "articles_case": true, + "articles": [ + "c", "l", "all", "dall", "dell", + "nell", "sull", "coll", "pell", + "gl", "agl", "dagl", "degl", "negl", + "sugl", "un", "m", "t", "s", "v", "d" + ] + }, + "italian_stop": { + "type": "stop", + "stopwords": "_italian_" + }, + "italian_stemmer": { + "type": "stemmer", + "language": "light_italian" + }, + "spanish_stop": { + "type": "stop", + "stopwords": "_spanish_" + }, + "spanish_stemmer": { + "type": "stemmer", + "language": "light_spanish" + }, + "catalan_elision": { + "type": "elision", + "articles_case": true, + "articles": ["d", "l", "m", "n", "s", "t"] + }, + "catalan_stop": { + "type": "stop", + "stopwords": "_catalan_" + }, + "catalan_stemmer": { + "type": "stemmer", + "language": "catalan" + }, + "basque_stop": { + "type": "stop", + "stopwords": "_basque_" + }, + "basque_stemmer": { + "type": "stemmer", + "language": "basque" + } + }, + "similarity": { + "c2corgsimilarity": { + "type": "BM25" + } + }, + "char_filter": { + "punctuationgreedy": { + "type": "pattern_replace", + "pattern": "[\\.,]" + } + }, + "analyzer": { + "index_ngram": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "icu_folding", + "autocomplete_filter"], + "tokenizer": "icu_tokenizer" + }, + "search_ngram": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "index_raw": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "search_raw": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "index_english": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "english_possessive_stemmer", + "lowercase", + "english_stop", + "english_stemmer", + "autocomplete_filter" + ] + }, + "search_english": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "english_possessive_stemmer", + "lowercase", + "english_stop", + "english_stemmer" + ] + }, + "index_french": { + "tokenizer": "standard", + "filter": [ + "french_elision", + "lowercase", + "french_stop", + "french_stemmer", + "autocomplete_filter" + ] + }, + "search_french": { + "tokenizer": "standard", + "filter": [ + "french_elision", + "lowercase", + "french_stop", + "french_stemmer", + "autocomplete_filter" + ] + }, + "index_german": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "german_stop", + "german_normalization", + "german_stemmer", + "autocomplete_filter" + ] + }, + "search_german": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "german_stop", + "german_normalization", + "german_stemmer" + ] + }, + "index_italian": { + "tokenizer": "standard", + "filter": [ + "italian_elision", + "lowercase", + "italian_stop", + "italian_stemmer", + "autocomplete_filter" + ] + }, + "search_italian": { + "tokenizer": "standard", + "filter": [ + "italian_elision", + "lowercase", + "italian_stop", + "italian_stemmer" + ] + }, + "index_spanish": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer", + "autocomplete_filter" + ] + }, + "search_spanish": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer" + ] + }, + "index_catalan": { + "tokenizer": "standard", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer", + "autocomplete_filter" + ] + }, + "search_catalan": { + "tokenizer": "standard", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer" + ] + }, + "index_basque": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer", + "autocomplete_filter" + ] + }, + "search_basque": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer" + ] + }, + "french_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "french_elision", + "french_stop", + "icu_folding", + "lowercase", + "french_stemmer" + ] + }, + "german_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "german_stop", + "german_stemmer", + "lowercase", + "icu_folding" + ] + }, + "english_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "english_possessive_stemmer", + "english_stop", + "lowercase", + "icu_folding" + ] + }, + "italian_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "italian_elision", + "italian_stop", + "lowercase", + "icu_folding", + "italian_stemmer" + ] + }, + "spanish_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer", + "icu_folding" + ] + }, + "catalan_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer", + "icu_folding" + ] + }, + "basque_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer", + "icu_folding" + ] + }, + "index_slovene": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "autocomplete_filter", + "lowercase" + ] + }, + "search_slovene": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase" + ] + }, + "slovene_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "icu_folding" + ] + }, + "index_chinois": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "autocomplete_filter" + ] + }, + "search_chinois": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase" + ] + }, + "chinois_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "icu_folding" + ] + } + } + } +} \ No newline at end of file diff --git a/scripts/esjson5-6/u.json b/scripts/esjson5-6/u.json new file mode 100644 index 000000000..1a89299b1 --- /dev/null +++ b/scripts/esjson5-6/u.json @@ -0,0 +1,235 @@ +{ + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/w.json b/scripts/esjson5-6/w.json new file mode 100644 index 000000000..757d8959e --- /dev/null +++ b/scripts/esjson5-6/w.json @@ -0,0 +1,346 @@ +{ + "properties": { + "access_time": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "best_periods": { + "type": "text", + "index": false + }, + "capacity": { + "type": "integer" + }, + "capacity_staffed": { + "type": "integer" + }, + "children_proof": { + "type": "text", + "index": false + }, + "climbing_indoor_types": { + "type": "text", + "index": false + }, + "climbing_outdoor_types": { + "type": "text", + "index": false + }, + "climbing_rating_max": { + "type": "integer" + }, + "climbing_rating_median": { + "type": "integer" + }, + "climbing_rating_min": { + "type": "integer" + }, + "climbing_styles": { + "type": "text", + "index": false + }, + "custodianship": { + "type": "text", + "index": false + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "equipment_ratings": { + "type": "integer" + }, + "exposition_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "height_max": { + "type": "integer" + }, + "height_median": { + "type": "integer" + }, + "height_min": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "length": { + "type": "integer" + }, + "lift_access": { + "type": "boolean" + }, + "orientations": { + "type": "text", + "index": false + }, + "paragliding_rating": { + "type": "integer" + }, + "product_types": { + "type": "text", + "index": false + }, + "prominence": { + "type": "integer" + }, + "public_transportation_rating": { + "type": "text", + "index": false + }, + "public_transportation_types": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "rain_proof": { + "type": "text", + "index": false + }, + "rock_types": { + "type": "text", + "index": false + }, + "routes_quantity": { + "type": "integer" + }, + "snow_clearance_rating": { + "type": "text", + "index": false + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "waypoint_type": { + "type": "text", + "index": false + }, + "weather_station_types": { + "type": "text", + "index": false + } + } + } \ No newline at end of file diff --git a/scripts/esjson5-6/x.json b/scripts/esjson5-6/x.json new file mode 100644 index 000000000..693194fa3 --- /dev/null +++ b/scripts/esjson5-6/x.json @@ -0,0 +1,265 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": false + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": false + }, + "avalanche_level": { + "type": "integer" + }, + "avalanche_slope": { + "type": "integer" + }, + "date": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "event_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "nb_impacted": { + "type": "integer" + }, + "nb_participants": { + "type": "integer" + }, + "quality": { + "type": "integer" + }, + "severity": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/pycurl-example.py b/scripts/pycurl-example.py new file mode 100755 index 000000000..e38ac8b73 --- /dev/null +++ b/scripts/pycurl-example.py @@ -0,0 +1,30 @@ +from urllib.parse import urlencode + +import pycurl +from io import BytesIO + +#curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/a.json http://192.168.1.120:9203/c2corg_a/_mapping/_doc + +f = open("./scripts/esjson5-6/a.json") +post_data = f.read() + +buffer = BytesIO() +c = pycurl.Curl() +c.setopt(c.URL, 'http://192.168.1.120:9203/c2corg_tests_a/_mapping/_doc') +header = ['Content-Type: application/json'] +c.setopt(c.HTTPHEADER, header) +c.setopt(c.CUSTOMREQUEST, 'PUT') +# Form data must be provided already urlencoded. +# Sets request method to POST, +# Content-Type header to application/x-www-form-urlencoded +# and data to send in request body. +c.setopt(c.POSTFIELDS, post_data) +c.setopt(c.WRITEDATA, buffer) +c.perform() +c.close() + +body = buffer.getvalue() +# Body is a byte string. +# We have to know the encoding in order to print it to a text file +# such as standard output. +print(body.decode('iso-8859-1')) From f4ae03d69a0e7f377cf64f0f954e78d84ce1ddd2 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Fri, 7 Jun 2024 19:46:30 +0200 Subject: [PATCH 09/53] Upgrade Elasticsearch from 5.6 to 6.8 --- c2corg_api/scripts/es/fill_index.py | 2 +- c2corg_api/scripts/es/sync.py | 6 +- c2corg_api/scripts/initializees.py | 141 ++++++++++++++---- c2corg_api/search/__init__.py | 12 +- c2corg_api/search/advanced_search.py | 4 + c2corg_api/search/mapping.py | 6 +- c2corg_api/search/mapping_types.py | 4 +- c2corg_api/search/mappings/area_mapping.py | 4 +- c2corg_api/search/mappings/article_mapping.py | 2 +- c2corg_api/search/mappings/book_mapping.py | 4 +- c2corg_api/search/mappings/image_mapping.py | 3 +- c2corg_api/search/mappings/outing_mapping.py | 3 +- c2corg_api/search/mappings/route_mapping.py | 3 +- .../search/mappings/topo_map_mapping.py | 4 +- c2corg_api/search/mappings/user_mapping.py | 4 +- .../search/mappings/waypoint_mapping.py | 3 +- c2corg_api/search/mappings/xreport_mapping.py | 4 +- c2corg_api/search/search.py | 12 +- c2corg_api/search/search_filters.py | 6 +- c2corg_api/tests/__init__.py | 1 + .../tests/scripts/es/test_fill_index.py | 30 ++-- c2corg_api/tests/scripts/es/test_syncer.py | 4 +- .../tests/search/test_search_filters.py | 6 +- c2corg_api/tests/views/__init__.py | 27 ++-- c2corg_api/tests/views/test_langs.py | 1 + c2corg_api/tests/views/test_user.py | 6 +- c2corg_api/tests/views/test_user_account.py | 4 +- c2corg_api/tests/views/test_user_profile.py | 4 +- c2corg_api/tests/views/test_waypoint.py | 2 +- c2corg_api/views/document.py | 3 + c2corg_api/views/health.py | 7 +- dataMigration/v6/{logstash => }/logstash.conf | 14 +- docker-compose.yml | 83 +++++++---- elasticsearch6.Dockerfile | 2 + logstash6.Dockerfile | 2 + requirements.txt | 5 +- scripts/esUpdateMapSettings56.sh | 104 ++++++------- scripts/esjson5-6/a.json | 44 +++++- scripts/esjson5-6/b.json | 46 +++++- scripts/esjson5-6/c.json | 48 +++++- scripts/esjson5-6/i.json | 46 +++++- scripts/esjson5-6/m.json | 42 +++++- scripts/esjson5-6/o.json | 46 +++++- scripts/esjson5-6/r.json | 58 +++++-- scripts/esjson5-6/u.json | 42 +++++- scripts/esjson5-6/w.json | 72 +++++++-- scripts/esjson5-6/x.json | 46 +++++- 47 files changed, 796 insertions(+), 226 deletions(-) rename dataMigration/v6/{logstash => }/logstash.conf (63%) create mode 100644 elasticsearch6.Dockerfile create mode 100644 logstash6.Dockerfile diff --git a/c2corg_api/scripts/es/fill_index.py b/c2corg_api/scripts/es/fill_index.py index d69ee0d0f..840c3cf06 100644 --- a/c2corg_api/scripts/es/fill_index.py +++ b/c2corg_api/scripts/es/fill_index.py @@ -81,7 +81,7 @@ def progress(count, total_count): for doc in sync.get_documents(session, doc_type, batch_size, ignore_redirects=True): - batch.add(to_search_document(doc, index_name)) + batch.add(to_search_document(doc, index_name[:-1] + doc_type)) count += 1 progress(count, total) diff --git a/c2corg_api/scripts/es/sync.py b/c2corg_api/scripts/es/sync.py index 8a84e21cc..f85bc197a 100644 --- a/c2corg_api/scripts/es/sync.py +++ b/c2corg_api/scripts/es/sync.py @@ -293,9 +293,9 @@ def sync_deleted_documents(session, deleted_documents, batch_size): with batch: for document_id, doc_type in deleted_documents: batch.add({ - '_index': index, + '_index': index[:-1]+doc_type, '_id': document_id, - '_type': doc_type, + 'c2corg_doc_type': doc_type, 'id': document_id, '_op_type': 'delete' }) @@ -375,7 +375,7 @@ def create_search_documents(doc_type, documents, batch): index = elasticsearch_config['index'] n = 0 for doc in documents: - batch.add(to_search_document(doc, index)) + batch.add(to_search_document(doc, index[:-1]+doc_type)) n += 1 log.info('Sent {} document(s) of type {}'.format(n, doc_type)) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 22c24a8c7..4b291b398 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -1,5 +1,7 @@ import os import sys +import pycurl +from io import BytesIO from c2corg_api.search.mappings.area_mapping import SearchArea from c2corg_api.search.mappings.article_mapping import SearchArticle @@ -11,6 +13,7 @@ from c2corg_api.search.mappings.topo_map_mapping import SearchTopoMap from c2corg_api.search.mappings.user_mapping import SearchUser from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint + from elasticsearch_dsl import Index from pyramid.paster import ( @@ -45,38 +48,38 @@ def main(argv=sys.argv): def setup_es(): """Create the ElasticSearch index and configure the mapping. """ + index_suffix_list=["_a","_b","_c","_i","_m","_o","_r","_u","_w","_x"] + client = elasticsearch_config['client'] index_name = elasticsearch_config['index'] + cible = elasticsearch_config['host']+':'+str(elasticsearch_config['port']) + print('cible es: %s',cible) info = client.info() print('ElasticSearch version: {0}'.format(info['version']['number'])) - if client.indices.exists(index_name): - print('Index "{0}" already exists. To re-create the index, manually ' - 'delete the index and run this script again.'.format(index_name)) - print('To delete the index run:') - print('curl -XDELETE \'http://{0}:{1}/{2}/\''.format( - elasticsearch_config['host'], elasticsearch_config['port'], - index_name)) - sys.exit(0) - - index = Index(index_name) - index.settings(analysis=analysis_settings) - - index.doc_type(SearchArea) - index.doc_type(SearchBook) - index.doc_type(SearchImage) - index.doc_type(SearchOuting) - index.doc_type(SearchXreport) - index.doc_type(SearchRoute) - index.doc_type(SearchTopoMap) - index.doc_type(SearchUser) - index.doc_type(SearchWaypoint) - index.doc_type(SearchArticle) - - index.create() - - print('Index "{0}" created'.format(index_name)) + for index_suffix in index_suffix_list: + print("suffix : ", index_suffix) + if client.indices.exists(index_name[:-2]+index_suffix): + print('Index "{0}" already exists. deleting it {0}... ' + .format(index_name[:-2]+index_suffix)) + #print('To delete the index run:') + #print('curl -XDELETE \'http://{0}:{1}/{2}/\''.format( + # elasticsearch_config['host'], elasticsearch_config['port'], + # index_name+index_suffix)) + delete_indice(cible, index_name[:-2]+index_suffix) + #sys.exit(0) + + for index_suffix in index_suffix_list: + print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) + create_indice(cible,index_name[:-2]+index_suffix) + print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) + indice_settings_update(cible,index_name[:-2]+index_suffix) + print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) + indice_mapping_update(cible,index_name[:-2]+index_suffix,index_suffix[1]) + print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) + + print('all indexes are created') def drop_index(silent=True): @@ -88,3 +91,89 @@ def drop_index(silent=True): except Exception as exc: if not silent: raise exc + + +def delete_indice(cible, indice_name): + buffer = BytesIO() + c = pycurl.Curl() + header = ['Content-Type: application/json'] + c.setopt(c.HTTPHEADER, header) + c.setopt(c.CUSTOMREQUEST, 'DELETE') + c.setopt(c.URL, 'http://' + cible + '/' + indice_name ) + c.setopt(c.WRITEDATA, buffer) + c.perform() + c.close() + + body = buffer.getvalue() + print(body.decode('iso-8859-1')) + + +def create_indice(cible, indice_name): + buffer = BytesIO() + c = pycurl.Curl() + header = ['Content-Type: application/json'] + c.setopt(c.HTTPHEADER, header) + c.setopt(c.CUSTOMREQUEST, 'PUT') + c.setopt(c.URL, 'http://' + cible + '/' + indice_name ) + c.setopt(c.WRITEDATA, buffer) + c.perform() + c.close() + + body = buffer.getvalue() + print(body.decode('iso-8859-1')) + +def indice_settings_update(cible, indice_name): + """curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_close""" + """curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_a/_settings""" + """curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_open""" + buffer = BytesIO() + c = pycurl.Curl() + header = ['Content-Type: application/json'] + c.setopt(c.HTTPHEADER, header) + c.setopt(c.CUSTOMREQUEST, 'POST') + c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_close') + c.setopt(c.WRITEDATA, buffer) + c.perform() + body = buffer.getvalue() + print(body.decode('iso-8859-1')) + + file = "./scripts/esjson5-6/settings.json" + f = open(file) + post_data = f.read() + c.setopt(c.CUSTOMREQUEST, 'PUT') + c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_settings') + c.setopt(c.POSTFIELDS, post_data) + c.setopt(c.WRITEDATA, buffer) + c.perform() + body = buffer.getvalue() + print(body.decode('iso-8859-1')) + + c.setopt(c.HTTPHEADER, header) + c.setopt(c.CUSTOMREQUEST, 'POST') + c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_open') + c.setopt(c.WRITEDATA, buffer) + c.perform() + body = buffer.getvalue() + print(body.decode('iso-8859-1')) + + c.close() + + +def indice_mapping_update(cible, indice_name, mapping_type): + file = "./scripts/esjson5-6/"+mapping_type+".json" + f = open(file) + post_data = f.read() + + buffer = BytesIO() + c = pycurl.Curl() + c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping/_doc') + header = ['Content-Type: application/json'] + c.setopt(c.HTTPHEADER, header) + c.setopt(c.CUSTOMREQUEST, 'PUT') + c.setopt(c.POSTFIELDS, post_data) + c.setopt(c.WRITEDATA, buffer) + c.perform() + c.close() + + body = buffer.getvalue() + print(body.decode('iso-8859-1')) \ No newline at end of file diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index d0307f735..7131ce627 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -1,3 +1,5 @@ +import logging + from c2corg_api.models.area import AREA_TYPE from c2corg_api.models.article import ARTICLE_TYPE from c2corg_api.models.book import BOOK_TYPE @@ -25,6 +27,7 @@ from elasticsearch_dsl.query import MultiMatch from kombu import Exchange, Queue, pools from kombu.connection import Connection +from pprint import pprint # the maximum number of documents that can be returned for each document type SEARCH_LIMIT_MAX = 50 @@ -74,11 +77,14 @@ def __init__(self, settings): return QueueConfiguration(settings) -def create_search(document_type): +def create_search(document_type=None): + #TODO: En fonction du document type envoyer vers le bon index + #en fonction document_type qui contient le suffixe de l'index return Search( using=elasticsearch_config['client'], - index=elasticsearch_config['index'], - doc_type=search_documents[document_type]) + index=(elasticsearch_config['index']), + doc_type="_doc", #search_documents[document_type] + ) def get_text_query_on_title(search_term, search_lang=None): diff --git a/c2corg_api/search/advanced_search.py b/c2corg_api/search/advanced_search.py index 3cf2dc450..f21892682 100644 --- a/c2corg_api/search/advanced_search.py +++ b/c2corg_api/search/advanced_search.py @@ -1,3 +1,7 @@ +import logging + +from IPython.lib.pretty import pprint + from c2corg_api.search.mapping_types import meta_param_keys from c2corg_api.search.search_filters import build_query diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index 29dda3a23..20376f5da 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -61,7 +61,7 @@ class Meta(BaseMeta): pass id = Long() - doc_type = Enum() + c2corg_doc_type = Enum() quality = QEnumRange( 'qa', model_field=Document.quality, enum_mapper=sortable_quality_types) available_locales = QEnumArray('l', enum=default_langs) @@ -138,7 +138,7 @@ def to_search_document(document, index, include_areas=True): search_document = { '_index': index, '_id': document.document_id, - '_type': document.type, + '_type': "_doc",#document.type, 'id': document.document_id } @@ -147,7 +147,7 @@ def to_search_document(document, index, include_areas=True): search_document['_op_type'] = 'delete' else: search_document['_op_type'] = 'index' - search_document['doc_type'] = document.type + search_document['c2corg_doc_type'] = document.type available_locales = [] for locale in document.locales: diff --git a/c2corg_api/search/mapping_types.py b/c2corg_api/search/mapping_types.py index df8964f98..0ce2b3c28 100644 --- a/c2corg_api/search/mapping_types.py +++ b/c2corg_api/search/mapping_types.py @@ -1,4 +1,4 @@ -from elasticsearch_dsl import String, Long, Integer, Boolean, Date +from elasticsearch_dsl import Text, Long, Integer, Boolean, Date # this module contains classes to mark the fields of a mapping that can be # used in a search. @@ -78,7 +78,7 @@ class QClass(QueryableMixin, clazz): return QClass -class Enum(String): +class Enum(Text): """Field type for enums that should not be analyzed before indexing. """ def __init__(self, *args, **kwargs): diff --git a/c2corg_api/search/mappings/area_mapping.py b/c2corg_api/search/mappings/area_mapping.py index f40b8f374..10606b98b 100644 --- a/c2corg_api/search/mappings/area_mapping.py +++ b/c2corg_api/search/mappings/area_mapping.py @@ -4,8 +4,8 @@ class SearchArea(SearchDocument): - class Meta(BaseMeta): - doc_type = AREA_TYPE +# class Meta(BaseMeta): +# doc_type = AREA_TYPE area_type = QEnum('atyp', model_field=Area.area_type) diff --git a/c2corg_api/search/mappings/article_mapping.py b/c2corg_api/search/mappings/article_mapping.py index 930e2c242..34aa537f2 100644 --- a/c2corg_api/search/mappings/article_mapping.py +++ b/c2corg_api/search/mappings/article_mapping.py @@ -5,7 +5,7 @@ class SearchArticle(SearchDocument): class Meta(BaseMeta): - doc_type = ARTICLE_TYPE + c2corg_doc_type = ARTICLE_TYPE activities = QEnumArray( 'act', model_field=Article.activities) diff --git a/c2corg_api/search/mappings/book_mapping.py b/c2corg_api/search/mappings/book_mapping.py index d93fae2bd..d5f5cc3e4 100644 --- a/c2corg_api/search/mappings/book_mapping.py +++ b/c2corg_api/search/mappings/book_mapping.py @@ -4,8 +4,8 @@ class SearchBook(SearchDocument): - class Meta(BaseMeta): - doc_type = BOOK_TYPE +# class Meta(BaseMeta): +# doc_type = BOOK_TYPE activities = QEnumArray( 'act', model_field=Book.activities) diff --git a/c2corg_api/search/mappings/image_mapping.py b/c2corg_api/search/mappings/image_mapping.py index 1cd6ba234..e177a89b1 100644 --- a/c2corg_api/search/mappings/image_mapping.py +++ b/c2corg_api/search/mappings/image_mapping.py @@ -6,7 +6,8 @@ class SearchImage(SearchDocument): class Meta(BaseMeta): - doc_type = IMAGE_TYPE +# doc_type = IMAGE_TYPE + c2corg_doc_type = IMAGE_TYPE activities = QEnumArray( 'act', model_field=Image.activities) diff --git a/c2corg_api/search/mappings/outing_mapping.py b/c2corg_api/search/mappings/outing_mapping.py index 0b6ba166b..879924895 100644 --- a/c2corg_api/search/mappings/outing_mapping.py +++ b/c2corg_api/search/mappings/outing_mapping.py @@ -17,7 +17,8 @@ class SearchOuting(SearchDocument): class Meta(BaseMeta): - doc_type = OUTING_TYPE +# doc_type = OUTING_TYPE + c2corg_doc_type = OUTING_TYPE date_start = Date() date_end = Date() diff --git a/c2corg_api/search/mappings/route_mapping.py b/c2corg_api/search/mappings/route_mapping.py index 758977958..9c7242863 100644 --- a/c2corg_api/search/mappings/route_mapping.py +++ b/c2corg_api/search/mappings/route_mapping.py @@ -17,7 +17,8 @@ class SearchRoute(SearchDocument): class Meta(BaseMeta): - doc_type = ROUTE_TYPE +# doc_type = ROUTE_TYPE + c2corg_doc_type = ROUTE_TYPE # array of waypoint ids waypoints = QLong('w', is_id=True) diff --git a/c2corg_api/search/mappings/topo_map_mapping.py b/c2corg_api/search/mappings/topo_map_mapping.py index a39c82c91..9792779db 100644 --- a/c2corg_api/search/mappings/topo_map_mapping.py +++ b/c2corg_api/search/mappings/topo_map_mapping.py @@ -4,8 +4,8 @@ class SearchTopoMap(SearchDocument): - class Meta(BaseMeta): - doc_type = MAP_TYPE +# class Meta(BaseMeta): +# doc_type = MAP_TYPE FIELDS = [] diff --git a/c2corg_api/search/mappings/user_mapping.py b/c2corg_api/search/mappings/user_mapping.py index 9c80f4b62..a6c98c05d 100644 --- a/c2corg_api/search/mappings/user_mapping.py +++ b/c2corg_api/search/mappings/user_mapping.py @@ -4,8 +4,8 @@ class SearchUser(SearchDocument): - class Meta(BaseMeta): - doc_type = USERPROFILE_TYPE +# class Meta(BaseMeta): +# doc_type = USERPROFILE_TYPE FIELDS = [] diff --git a/c2corg_api/search/mappings/waypoint_mapping.py b/c2corg_api/search/mappings/waypoint_mapping.py index 1ebb6a73e..d4e553f37 100644 --- a/c2corg_api/search/mappings/waypoint_mapping.py +++ b/c2corg_api/search/mappings/waypoint_mapping.py @@ -10,7 +10,8 @@ class SearchWaypoint(SearchDocument): class Meta(BaseMeta): - doc_type = WAYPOINT_TYPE + #doc_type = WAYPOINT_TYPE + c2corg_doc_type = WAYPOINT_TYPE elevation = QInteger( 'walt', range=True) diff --git a/c2corg_api/search/mappings/xreport_mapping.py b/c2corg_api/search/mappings/xreport_mapping.py index a54b86c72..17fc71828 100644 --- a/c2corg_api/search/mappings/xreport_mapping.py +++ b/c2corg_api/search/mappings/xreport_mapping.py @@ -8,8 +8,8 @@ class SearchXreport(SearchDocument): - class Meta(BaseMeta): - doc_type = XREPORT_TYPE +# class Meta(BaseMeta): +# doc_type = XREPORT_TYPE event_activity = QEnumArray( 'act', model_field=Xreport.event_activity) diff --git a/c2corg_api/search/search.py b/c2corg_api/search/search.py index b5149391b..28a6c9ccd 100644 --- a/c2corg_api/search/search.py +++ b/c2corg_api/search/search.py @@ -1,8 +1,13 @@ +import logging + +from elasticsearch_dsl import Q + from c2corg_api.search import create_search, elasticsearch_config, \ get_text_query_on_title from c2corg_api.views.document_listings import get_documents -from elasticsearch_dsl.search import MultiSearch +from elasticsearch_dsl.search import MultiSearch, Search +log = logging.getLogger(__name__) def search_for_types(search_types, search_term, limit, lang): """Get results for all given types. @@ -40,12 +45,13 @@ def do_multi_search_for_types(search_types, search_term, limit, lang): and returns a list of tuples (document_ids, total) containing the results for each type. """ - multi_search = MultiSearch(index=elasticsearch_config['index']) + multi_search = MultiSearch(index=elasticsearch_config['index'],) for search_type in search_types: (_, get_documents_config) = search_type search = create_search(get_documents_config.document_type).\ - query(get_text_query_on_title(search_term, lang)).\ + query(get_text_query_on_title(search_term, lang)). \ + filter(Q("match", c2corg_doc_type=get_documents_config.document_type)). \ extra(from_=0, size=limit) multi_search = multi_search.add(search) diff --git a/c2corg_api/search/search_filters.py b/c2corg_api/search/search_filters.py index d435f979f..7e2e85b7a 100644 --- a/c2corg_api/search/search_filters.py +++ b/c2corg_api/search/search_filters.py @@ -35,6 +35,10 @@ def build_query(url_params, meta_params, doc_type): if filter: search = search.filter(filter) + #TODO: Remettre le filtre + #search.filter(Q("term", c2corg_doc_type=doc_type)) + search = search.filter(Q("match", c2corg_doc_type=doc_type)) + search = search. \ extra(from_=offset, size=limit) @@ -55,8 +59,6 @@ def build_query(url_params, meta_params, doc_type): else: search = search.sort({'id': {'order': 'desc'}}) - log.debug('Search filter: {}'.format(search.to_dict())) - return search diff --git a/c2corg_api/tests/__init__.py b/c2corg_api/tests/__init__.py index ca56146da..61dc2f5c4 100644 --- a/c2corg_api/tests/__init__.py +++ b/c2corg_api/tests/__init__.py @@ -1,4 +1,5 @@ import time +import pycurl import jwt import datetime diff --git a/c2corg_api/tests/scripts/es/test_fill_index.py b/c2corg_api/tests/scripts/es/test_fill_index.py index 88daa53fc..915683d6c 100644 --- a/c2corg_api/tests/scripts/es/test_fill_index.py +++ b/c2corg_api/tests/scripts/es/test_fill_index.py @@ -142,57 +142,57 @@ def test_fill_index(self): # fill the ElasticSearch index fill_index(self.session) - waypoint1 = SearchWaypoint.get(id=71171) + waypoint1 = SearchWaypoint.get(id=71171, index='c2corg_tests_w') self.assertIsNotNone(waypoint1) self.assertEqual(waypoint1.title_en, 'Mont Granier') self.assertEqual(waypoint1.title_fr, 'Mont Granier') # self.assertEqual(waypoint1.summary_fr, 'Le Mont Granier ') - self.assertEqual(waypoint1.doc_type, 'w') + self.assertEqual(waypoint1.c2corg_doc_type, 'w') self.assertEqual(waypoint1.quality, 2) self.assertEqual(waypoint1.access_time, 3) self.assertAlmostEqual(waypoint1.geom[0], 5.71288994) self.assertAlmostEqual(waypoint1.geom[1], 45.64476395) - waypoint2 = SearchWaypoint.get(id=71172) + waypoint2 = SearchWaypoint.get(id=71172, index='c2corg_tests_w') self.assertIsNotNone(waypoint2) self.assertEqual(waypoint2.title_en, 'Mont Blanc') self.assertIsNone(waypoint2.title_fr) - self.assertEqual(waypoint2.doc_type, 'w') + self.assertEqual(waypoint2.c2corg_doc_type, 'w') - route = SearchRoute.get(id=71173) + route = SearchRoute.get(id=71173, index='c2corg_tests_r') self.assertIsNotNone(route) self.assertEqual(route.title_en, 'Mont Blanc : Face N') self.assertIsNone(route.title_fr) - self.assertEqual(route.doc_type, 'r') + self.assertEqual(route.c2corg_doc_type, 'r') self.assertEqual(route.durations, [0]) - outing = SearchOuting.get(id=71175) + outing = SearchOuting.get(id=71175, index='c2corg_tests_o') self.assertIsNotNone(outing) self.assertEqual(outing.title_en, 'Mont Blanc : Face N !') self.assertIsNone(outing.title_fr) - self.assertEqual(outing.doc_type, 'o') + self.assertEqual(outing.c2corg_doc_type, 'o') self.assertEqual(outing.frequentation, 3) - article = SearchArticle.get(id=71176) + article = SearchArticle.get(id=71176, index='c2corg_tests_c') self.assertIsNotNone(article) self.assertEqual(article.title_en, 'Lac d\'Annecy') self.assertEqual(article.title_fr, 'Lac d\'Annecy') - self.assertEqual(article.doc_type, 'c') + self.assertEqual(article.c2corg_doc_type, 'c') - book = SearchBook.get(id=71177) + book = SearchBook.get(id=71177, index='c2corg_tests_b') self.assertIsNotNone(book) self.assertEqual(book.title_en, 'Escalades au Thaurac') self.assertEqual(book.title_fr, 'Escalades au Thaurac') - self.assertEqual(book.doc_type, 'b') + self.assertEqual(book.c2corg_doc_type, 'b') self.assertEqual(book.book_types, ['biography']) - xreport = SearchXreport.get(id=71178) + xreport = SearchXreport.get(id=71178, index='c2corg_tests_x') self.assertIsNotNone(xreport) self.assertEqual(xreport.title_en, 'Death in the mountains') - self.assertEqual(xreport.doc_type, 'x') + self.assertEqual(xreport.c2corg_doc_type, 'x') # merged document is ignored - self.assertIsNone(SearchWaypoint.get(id=71174, ignore=404)) + self.assertIsNone(SearchWaypoint.get(id=71174, index='c2corg_tests_w', ignore=404)) # check that the sync. status was updated last_update, _ = es_sync.get_status(self.session) diff --git a/c2corg_api/tests/scripts/es/test_syncer.py b/c2corg_api/tests/scripts/es/test_syncer.py index fff2c903e..0cefc0517 100644 --- a/c2corg_api/tests/scripts/es/test_syncer.py +++ b/c2corg_api/tests/scripts/es/test_syncer.py @@ -46,9 +46,9 @@ def test_process_task(self): next(syncer.consume(limit=1)) index = elasticsearch_config['index'] - doc = SearchWaypoint.get(id=document_id, index=index) + doc = SearchWaypoint.get(id=document_id, index=index[:-1]+'w') self.assertEqual(doc['title_fr'], 'Mont Granier') - self.assertEqual(doc['doc_type'], 'w') + self.assertEqual(doc['c2corg_doc_type'], 'w') # simulate removing a document self.session.add(ESDeletedDocument( diff --git a/c2corg_api/tests/search/test_search_filters.py b/c2corg_api/tests/search/test_search_filters.py index 707a0693b..fadb3fe12 100644 --- a/c2corg_api/tests/search/test_search_filters.py +++ b/c2corg_api/tests/search/test_search_filters.py @@ -23,12 +23,13 @@ def test_build_query(self): 'limit': 10, 'offset': 0 } - query = build_query(params, meta_params, 'w') + query = build_query(params, meta_params,'w') expected_query = create_search('w'). \ query(get_text_query_on_title('search word')). \ filter(Term(available_locales='fr')).\ filter(Terms(areas=[1234, 4567])). \ filter(Range(elevation={'gte': 1500})). \ + filter(Q("match", c2corg_doc_type='w')). \ extra(from_=0, size=10) self.assertQueryEqual(query, expected_query) @@ -51,6 +52,7 @@ def test_build_query_bbox(self): 'left': 6.28279913, 'bottom': 46.03129072, 'right': 6.28369744, 'top': 46.03191439}, type='indexed')). \ + filter(Q("match", c2corg_doc_type='w')). \ extra(from_=0, size=10) self.assertQueryEqual(query, expected_query) @@ -65,6 +67,7 @@ def test_build_query_limit_offset(self): query = build_query(params, meta_params, 'w') expected_query = create_search('w'). \ query(get_text_query_on_title('search word')). \ + filter(Q("match", c2corg_doc_type='w')). \ extra(from_=40, size=20) self.assertQueryEqual(query, expected_query) @@ -79,6 +82,7 @@ def test_build_query_sort_outing(self): query = build_query(params, meta_params, 'o') expected_query = create_search('o'). \ filter(Term(activities='skitouring')). \ + filter(Q("match", c2corg_doc_type='o')). \ sort({'date_end': {'order': 'desc'}}, {'id': {'order': 'desc'}}). \ extra(from_=40, size=20) self.assertQueryEqual(query, expected_query) diff --git a/c2corg_api/tests/views/__init__.py b/c2corg_api/tests/views/__init__.py index 09cedb27d..bff24d38f 100644 --- a/c2corg_api/tests/views/__init__.py +++ b/c2corg_api/tests/views/__init__.py @@ -1,8 +1,11 @@ import json +import logging import urllib.request import urllib.parse import urllib.error +from IPython.lib.pretty import pprint + from c2corg_api.caching import cache_document_detail from c2corg_api.models.cache_version import CacheVersion, get_cache_key from c2corg_api.models.feed import DocumentChange @@ -584,6 +587,7 @@ def post_success(self, request_body, user='contributor', headers=headers, status=200) body = response.json + if skip_validation: document_id = body.get('document_id') response = self.app.get( @@ -639,10 +643,9 @@ def _validate_document(self, body, headers=None, validate_with_auth=False): self.sync_es() search_doc = search_documents[self._doc_type].get( id=doc.document_id, - index=elasticsearch_config['index']) - - self.assertIsNotNone(search_doc['doc_type']) - self.assertEqual(search_doc['doc_type'], doc.type) + index=elasticsearch_config['index'][:-1]+self._doc_type) + self.assertIsNotNone(search_doc['c2corg_doc_type']) + self.assertEqual(search_doc['c2corg_doc_type'], doc.type) if isinstance(doc, Route): title = waypoint_locale_en.title_prefix + ' : ' + \ @@ -826,9 +829,9 @@ def put_success_all( # check updates to the search index search_doc = search_documents[self._doc_type].get( id=document.document_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+self._doc_type) - self.assertEqual(search_doc['doc_type'], document.type) + self.assertEqual(search_doc['c2corg_doc_type'], document.type) self.assertEqual(search_doc['title_en'], archive_locale.title) self.assertEqual(search_doc['title_fr'], archive_locale_fr.title) @@ -911,9 +914,9 @@ def put_success_figures_only( sync_es(self.session) search_doc = search_documents[self._doc_type].get( id=document.document_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+self._doc_type) - self.assertEqual(search_doc['doc_type'], document.type) + self.assertEqual(search_doc['c2corg_doc_type'], document.type) if isinstance(document, Route) and document.main_waypoint_id: locale_en = document.get_locale('en') @@ -1007,9 +1010,9 @@ def put_success_lang_only( sync_es(self.session) search_doc = search_documents[self._doc_type].get( id=document.document_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+self._doc_type) - self.assertEqual(search_doc['doc_type'], document.type) + self.assertEqual(search_doc['c2corg_doc_type'], document.type) self.assertEqual( search_doc['title_en'], document.get_locale('en').title) self.assertEqual( @@ -1099,9 +1102,9 @@ def put_success_new_lang( sync_es(self.session) search_doc = search_documents[self._doc_type].get( id=document.document_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+self._doc_type) - self.assertEqual(search_doc['doc_type'], document.type) + self.assertEqual(search_doc['c2corg_doc_type'], document.type) self.assertEqual( search_doc['title_en'], document.get_locale('en').title) self.assertEqual( diff --git a/c2corg_api/tests/views/test_langs.py b/c2corg_api/tests/views/test_langs.py index bf4ed4ef9..dc41cd13f 100644 --- a/c2corg_api/tests/views/test_langs.py +++ b/c2corg_api/tests/views/test_langs.py @@ -1,3 +1,4 @@ +import logging from datetime import date, datetime from c2corg_api.models.feed import DocumentChange diff --git a/c2corg_api/tests/views/test_user.py b/c2corg_api/tests/views/test_user.py index ea7c3cc90..610fb0880 100644 --- a/c2corg_api/tests/views/test_user.py +++ b/c2corg_api/tests/views/test_user.py @@ -348,7 +348,7 @@ def test_register_search_index(self, _send_email): sync_es(self.session) search_doc = search_documents[USERPROFILE_TYPE].get( id=user_id, - index=elasticsearch_config['index'], ignore=404) + index=elasticsearch_config['index'][:-1]+'u', ignore=404) self.assertIsNone(search_doc) # Simulate confirmation email validation @@ -360,10 +360,10 @@ def test_register_search_index(self, _send_email): self.sync_es() search_doc = search_documents[USERPROFILE_TYPE].get( id=user_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+'u') self.assertIsNotNone(search_doc) - self.assertIsNotNone(search_doc['doc_type']) + self.assertIsNotNone(search_doc['c2corg_doc_type']) self.assertEqual(search_doc['title_fr'], 'Max Mustermann testf') @patch('c2corg_api.emails.email_service.EmailService._send_email') diff --git a/c2corg_api/tests/views/test_user_account.py b/c2corg_api/tests/views/test_user_account.py index 493f616fb..41a898282 100644 --- a/c2corg_api/tests/views/test_user_account.py +++ b/c2corg_api/tests/views/test_user_account.py @@ -89,12 +89,12 @@ def test_update_account_name_discourse_up(self): self.sync_es() search_doc = search_documents[USERPROFILE_TYPE].get( id=user_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+'u') # and check that the cache version of the user profile was updated self.check_cache_version(user_id, 2) - self.assertIsNotNone(search_doc['doc_type']) + self.assertIsNotNone(search_doc['c2corg_doc_type']) self.assertEqual( search_doc['title_en'], 'changed contributor') diff --git a/c2corg_api/tests/views/test_user_profile.py b/c2corg_api/tests/views/test_user_profile.py index ef723f5a8..42a57b2b3 100644 --- a/c2corg_api/tests/views/test_user_profile.py +++ b/c2corg_api/tests/views/test_user_profile.py @@ -356,8 +356,8 @@ def _check_es_index(self): sync_es(self.session) search_doc = SearchUser.get( id=self.profile1.document_id, - index=elasticsearch_config['index']) - self.assertEqual(search_doc['doc_type'], self.profile1.type) + index=elasticsearch_config['index'][:-1]+'u') + self.assertEqual(search_doc['c2corg_doc_type'], self.profile1.type) self.assertEqual( search_doc['title_en'], 'Contributor contributor') self.assertEqual( diff --git a/c2corg_api/tests/views/test_waypoint.py b/c2corg_api/tests/views/test_waypoint.py index 98885e1f4..58d7d59d8 100644 --- a/c2corg_api/tests/views/test_waypoint.py +++ b/c2corg_api/tests/views/test_waypoint.py @@ -855,7 +855,7 @@ def test_put_success_all(self): # check that the route was updated in the search index search_doc = SearchRoute.get( id=route.document_id, - index=elasticsearch_config['index']) + index=elasticsearch_config['index'][:-1]+'r') self.assertEqual( search_doc['title_en'], 'Mont Granier! : Mont Blanc from the air') diff --git a/c2corg_api/views/document.py b/c2corg_api/views/document.py index 6a65f18ac..09fc435ec 100644 --- a/c2corg_api/views/document.py +++ b/c2corg_api/views/document.py @@ -1,4 +1,5 @@ import logging +from pprint import pprint from c2corg_api.caching import cache_document_detail, cache_document_cooked from c2corg_api.models import DBSession @@ -60,6 +61,8 @@ class DocumentRest(object): def __init__(self, request): + pprint(self) + pprint(request) self.request = request # TODO: remove doc_type, it's in documents_config diff --git a/c2corg_api/views/health.py b/c2corg_api/views/health.py index 875dd14c1..2e00f027b 100644 --- a/c2corg_api/views/health.py +++ b/c2corg_api/views/health.py @@ -1,4 +1,5 @@ import logging +import sys from c2corg_api.caching import cache_document_detail from c2corg_api.models import DBSession, es_sync @@ -65,7 +66,11 @@ def _add_es_status(self, status): client = elasticsearch_config['client'] index = elasticsearch_config['index'] stats = client.indices.stats(index, metric='docs') - es_docs = stats['indices'][index]['total']['docs']['count'] + document_count_all_indice = 0 + for indice in stats['indices']: + document_count_all_indice += stats['indices'][indice]['total']['docs']['count'] + #es_docs = stats['indices'][index]['total']['docs']['count'] + es_docs = document_count_all_indice success = True except Exception: log.exception('Getting indexed docs count failed') diff --git a/dataMigration/v6/logstash/logstash.conf b/dataMigration/v6/logstash.conf similarity index 63% rename from dataMigration/v6/logstash/logstash.conf rename to dataMigration/v6/logstash.conf index 9f23710ee..ee87e1e2d 100644 --- a/dataMigration/v6/logstash/logstash.conf +++ b/dataMigration/v6/logstash.conf @@ -17,20 +17,24 @@ filter { add_field => { "geom" => [0,0] } } } + mutate { + add_field => { "c2corg_doc_type" => "%{[@metadata][_type]}" } + } mutate { - add_field => { "doc_id" => "%{[@metadata][_id]}" } + remove_field => [ "_type", "@timestamp", "@version", "doc_id", "doc_type" ] } + } output { elasticsearch { - hosts => ["elasticsearch6:9200"] + hosts => ["elasticsearch68:9200"] index => "c2corg_%{[@metadata][_type]}" - document_type => "%{doc_type}" - document_id => "%{doc_id}" + document_type => "_doc" + document_id => "%{[@metadata][_id]}" } stdout { @@ -42,4 +46,4 @@ output #stdout { codec => "dots"} #stdout { codec => rubydebug } #stdout { codec => "json" } -#docker-compose run logstash5 /usr/share/logstash/bin/logstash -f /root/logstash.conf \ No newline at end of file +#docker-compose run logstash6 /usr/share/logstash/bin/logstash -f /root/logstash.conf \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 1848772f6..7358d1344 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,21 +1,24 @@ version: "3.7" services: api: + platform: linux/amd64 build: context: . dockerfile: dev_api.Dockerfile network: host depends_on: - postgresql - - elasticsearch +# - elasticsearch - redis ports: - 6543:6543 environment: db_host: 'postgresql' tests_db_host: 'postgresql' - elasticsearch_host: 'elasticsearch' - tests_elasticsearch_host: 'elasticsearch' + elasticsearch_host: '192.168.1.120' + elasticsearch_port: 9203 + tests_elasticsearch_host: '192.168.1.120' + tests_elasticsearch_port: 9203 redis_url: 'redis://redis:6379/' version: '' volumes: @@ -27,10 +30,12 @@ services: - ./test.ini.in:/var/www/test.ini.in - ./pytest.ini:/var/www/pytest.ini - ./requirements.txt:/var/www/requirements.txt + - ./config:/var/www/config + - ./scripts:/var/www/scripts command: make -f config/docker-dev serve links: - postgresql - - elasticsearch +# - elasticsearch - redis postgresql: @@ -41,38 +46,66 @@ services: - ./docker-compose/pgsql-settings.d/:/c2corg_anon/pgsql-settings.d/ - .:/v6_api - elasticsearch26: - image: 'docker.io/c2corg/c2corg_es:anon-2018-11-02' - container_name: elasticsearch26 - ports: - - 9201:9200 - command: -Des.index.number_of_replicas=0 -Des.path.data=/c2corg_anon -Des.script.inline=true +# elasticsearch26: +# platform: linux/amd64 +# image: 'docker.io/c2corg/c2corg_es:anon-2018-11-02' +# container_name: elasticsearch26 +# ports: +# - 9201:9200 +# command: -Des.index.number_of_replicas=0 -Des.path.data=/c2corg_anon -Des.script.inline=true + +# elasticsearch56: +# platform: linux/amd64 +# container_name: elasticsearch56 +# build: +# context: . +# dockerfile: elasticsearch5.Dockerfile +# network: host +# environment: +# - discovery.type=single-node +# ports: +# - 9202:9200 +# - 9302:9300 - elasticsearch: - platform: linux/x86_64 + elasticsearch68: + platform: linux/amd64 container_name: elasticsearch build: context: . - dockerfile: elasticsearch5.Dockerfile + dockerfile: elasticsearch6.Dockerfile network: host environment: - discovery.type=single-node - - cluster.initial_master_nodes="docker-cluster" ports: - 9200:9200 - 9300:9300 +# volumes: +# - ./volumes/es6/data:/usr/share/elasticsearch/data +# - ./volumes/es6/config/jvm.options:/usr/share/elasticsearch/config/jvm.options - logstash5: - platform: linux/x86_64 - name: logstash5 - build: - context: . - dockerfile: logstash5.Dockerfile - network: host - ports: - - 5044:5044 - volumes: - - ./dataMigration/v5/logstash.conf:/root/logstash.conf +# logstash5: +# platform: linux/amd64 +# container_name: logstash5 +# build: +# context: . +# dockerfile: logstash5.Dockerfile +# network: host +# ports: +# - 5045:5044 +# volumes: +# - ./dataMigration/v5/logstash.conf:/root/logstash.conf + +# logstash6: +# platform: linux/amd64 +# container_name: logstash6 +# build: +# context: . +# dockerfile: logstash6.Dockerfile +# network: host +# ports: +# - 5044:5044 +# volumes: +# - ./dataMigration/v6:/usr/share/logstash/pipeline redis: image: 'docker.io/redis:3.2' diff --git a/elasticsearch6.Dockerfile b/elasticsearch6.Dockerfile new file mode 100644 index 000000000..7a6a9fce6 --- /dev/null +++ b/elasticsearch6.Dockerfile @@ -0,0 +1,2 @@ +FROM elasticsearch:6.8.23 +RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu diff --git a/logstash6.Dockerfile b/logstash6.Dockerfile new file mode 100644 index 000000000..e21a2f5a4 --- /dev/null +++ b/logstash6.Dockerfile @@ -0,0 +1,2 @@ +FROM logstash:6.8.23 +RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/requirements.txt b/requirements.txt index cd0fde4c4..e1f5f7402 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,8 +4,8 @@ bcrypt==4.1.2 bleach[css]==6.1.0 colander==2.0 dogpile.cache==1.3.2 -elasticsearch==6.0.0 #elasticsearch==2.4.1 -elasticsearch-dsl==6.0.1 #elasticsearch_dsl==2.2.0 +elasticsearch==7.0.0 #elasticsearch==2.4.1 +elasticsearch-dsl==7.0.0 #elasticsearch_dsl==2.2.0 geoalchemy2==0.4.2 geojson==3.1.0 geomet==1.1.0 @@ -32,6 +32,7 @@ transaction==4.0 waitress==3.0.0 zope.sqlalchemy==3.1 gunicorn==22.0.0 +pycurl # ColanderAlchemy>=0.3.2 # needed for ColanderAlchemy: https://github.com/stefanofontanelli/ColanderAlchemy/pull/90 diff --git a/scripts/esUpdateMapSettings56.sh b/scripts/esUpdateMapSettings56.sh index c71d99ff3..a9f31bfed 100755 --- a/scripts/esUpdateMapSettings56.sh +++ b/scripts/esUpdateMapSettings56.sh @@ -10,83 +10,83 @@ #sleep 10 #in sec to let the docker restart #efface tous les indices présents -curl -X DELETE "http://192.168.1.120:9203/c2corg_*" +curl -X DELETE "http://localhost:9200/c2corg_*" #change cluster definition des shards -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/cluster.json http://192.168.1.120:9203/_template/index_defaults +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/cluster.json http://localhost:9200/_template/index_defaults #créer les index -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_b -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_c -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_i -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_m -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_o -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_r -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_u -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_w -curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_x +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_a +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_b +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_c +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_i +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_m +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_o +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_r +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_u +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_w +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_x sleep 2 #2. closing the index to upgrade settings echo -e '\n closing index' -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_b/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_c/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_i/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_m/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_o/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_r/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_u/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_w/_close -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_x/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_a/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_b/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_c/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_i/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_m/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_o/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_r/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_u/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_w/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_x/_close sleep 2 #3. Updating analyser settings echo -e '\n updating Settings' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_a/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_b/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_c/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_i/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_m/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_o/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_r/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_u/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_w/_settings -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_x/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_a/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_b/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_c/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_i/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_m/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_o/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_r/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_u/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_w/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://localhost:9200/c2corg_x/_settings sleep 2 #4. restarting the index echo -e '\n restarting index' -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_b/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_c/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_i/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_m/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_o/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_r/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_u/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_w/_open -curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_x/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_a/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_b/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_c/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_i/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_m/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_o/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_r/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_u/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_w/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_x/_open sleep 2 #5. Docs Mappings update echo 'updating doc type mappings' echo 'doc type : a' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/a.json http://192.168.1.120:9203/c2corg_a/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/a.json http://localhost:9200/c2corg_a/_mapping/_doc echo -e '\n doc type : b' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/b.json http://192.168.1.120:9203/c2corg_b/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/b.json http://localhost:9200/c2corg_b/_mapping/_doc echo -e '\n doc type : c' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/c.json http://192.168.1.120:9203/c2corg_c/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/c.json http://localhost:9200/c2corg_c/_mapping/_doc echo -e '\n doc type : i' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/i.json http://192.168.1.120:9203/c2corg_i/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/i.json http://localhost:9200/c2corg_i/_mapping/_doc echo -e '\n doc type : m' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/m.json http://192.168.1.120:9203/c2corg_m/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/m.json http://localhost:9200/c2corg_m/_mapping/_doc echo -e '\n doc type : o' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/o.json http://192.168.1.120:9203/c2corg_o/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/o.json http://localhost:9200/c2corg_o/_mapping/_doc echo -e '\n doc type : r' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/r.json http://192.168.1.120:9203/c2corg_r/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/r.json http://localhost:9200/c2corg_r/_mapping/_doc echo -e '\n doc type : u' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/u.json http://192.168.1.120:9203/c2corg_u/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/u.json http://localhost:9200/c2corg_u/_mapping/_doc echo -e '\n doc type : w' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/w.json http://192.168.1.120:9203/c2corg_w/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/w.json http://localhost:9200/c2corg_w/_mapping/_doc echo -e '\n doc type : x' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/x.json http://192.168.1.120:9203/c2corg_x/_mapping/_doc +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/x.json http://localhost:9200/c2corg_x/_mapping/_doc sleep 2 diff --git a/scripts/esjson5-6/a.json b/scripts/esjson5-6/a.json index ef8a0d314..1b3a46031 100644 --- a/scripts/esjson5-6/a.json +++ b/scripts/esjson5-6/a.json @@ -2,14 +2,14 @@ "properties": { "area_type": { "type": "text", - "index": false + "index": true }, "areas": { "type": "long" }, "available_locales": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -234,6 +234,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/b.json b/scripts/esjson5-6/b.json index bc8b37dab..6fa5d297e 100644 --- a/scripts/esjson5-6/b.json +++ b/scripts/esjson5-6/b.json @@ -2,18 +2,18 @@ "properties": { "activities": { "type": "text", - "index": false + "index": true }, "areas": { "type": "long" }, "available_locales": { "type": "text", - "index": false + "index": true }, "book_types": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -238,6 +238,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/c.json b/scripts/esjson5-6/c.json index 91f8ee2a9..a5d6ebf96 100644 --- a/scripts/esjson5-6/c.json +++ b/scripts/esjson5-6/c.json @@ -2,22 +2,22 @@ "properties": { "activities": { "type": "text", - "index": false + "index": true }, "areas": { "type": "long" }, "article_categories": { "type": "text", - "index": false + "index": true }, "article_type": { "type": "text", - "index": false + "index": true }, "available_locales": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -242,6 +242,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/i.json b/scripts/esjson5-6/i.json index f32de471d..a58b744b1 100644 --- a/scripts/esjson5-6/i.json +++ b/scripts/esjson5-6/i.json @@ -2,18 +2,18 @@ "properties": { "activities": { "type": "text", - "index": false + "index": true }, "areas": { "type": "long" }, "available_locales": { "type": "text", - "index": false + "index": true }, "categories": { "type": "text", - "index": false + "index": true }, "date_time": { "type": "date", @@ -249,6 +249,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/m.json b/scripts/esjson5-6/m.json index 1a89299b1..8f23d485f 100644 --- a/scripts/esjson5-6/m.json +++ b/scripts/esjson5-6/m.json @@ -5,7 +5,7 @@ }, "available_locales": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -230,6 +230,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/o.json b/scripts/esjson5-6/o.json index 9100ccce4..b4fa1be8c 100644 --- a/scripts/esjson5-6/o.json +++ b/scripts/esjson5-6/o.json @@ -2,18 +2,18 @@ "properties": { "activities": { "type": "text", - "index": false + "index": true }, "areas": { "type": "long" }, "available_locales": { "type": "text", - "index": false + "index": true }, "avalanche_signs": { "type": "text", - "index": false + "index": true }, "condition_rating": { "type": "integer" @@ -330,6 +330,46 @@ }, "waypoints": { "type": "long" + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/r.json b/scripts/esjson5-6/r.json index d7384a9d5..16ebe15ca 100644 --- a/scripts/esjson5-6/r.json +++ b/scripts/esjson5-6/r.json @@ -2,7 +2,7 @@ "properties": { "activities": { "type": "text", - "index": false + "index": true }, "aid_rating": { "type": "integer" @@ -12,15 +12,15 @@ }, "available_locales": { "type": "text", - "index": false + "index": true }, "climbing_outdoor_type": { "type": "text", - "index": false + "index": true }, "configuration": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -88,7 +88,7 @@ }, "glacier_gear": { "type": "text", - "index": false + "index": true }, "global_rating": { "type": "integer" @@ -143,7 +143,7 @@ }, "orientations": { "type": "text", - "index": false + "index": true }, "quality": { "type": "integer" @@ -159,14 +159,14 @@ }, "rock_types": { "type": "text", - "index": false + "index": true }, "route_length": { "type": "integer" }, "route_types": { "type": "text", - "index": false + "index": true }, "ski_exposition": { "type": "integer" @@ -176,7 +176,7 @@ }, "slackline_type": { "type": "text", - "index": false + "index": true }, "snowshoe_rating": { "type": "integer" @@ -361,6 +361,46 @@ }, "waypoints": { "type": "long" + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/u.json b/scripts/esjson5-6/u.json index 1a89299b1..8f23d485f 100644 --- a/scripts/esjson5-6/u.json +++ b/scripts/esjson5-6/u.json @@ -5,7 +5,7 @@ }, "available_locales": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -230,6 +230,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/w.json b/scripts/esjson5-6/w.json index 757d8959e..4d3e561e0 100644 --- a/scripts/esjson5-6/w.json +++ b/scripts/esjson5-6/w.json @@ -8,11 +8,11 @@ }, "available_locales": { "type": "text", - "index": false + "index": true }, "best_periods": { "type": "text", - "index": false + "index": true }, "capacity": { "type": "integer" @@ -22,15 +22,15 @@ }, "children_proof": { "type": "text", - "index": false + "index": true }, "climbing_indoor_types": { "type": "text", - "index": false + "index": true }, "climbing_outdoor_types": { "type": "text", - "index": false + "index": true }, "climbing_rating_max": { "type": "integer" @@ -43,11 +43,11 @@ }, "climbing_styles": { "type": "text", - "index": false + "index": true }, "custodianship": { "type": "text", - "index": false + "index": true }, "description_ca": { "type": "text", @@ -121,43 +121,43 @@ }, "orientations": { "type": "text", - "index": false + "index": true }, "paragliding_rating": { "type": "integer" }, "product_types": { "type": "text", - "index": false + "index": true }, "prominence": { "type": "integer" }, "public_transportation_rating": { "type": "text", - "index": false + "index": true }, "public_transportation_types": { "type": "text", - "index": false + "index": true }, "quality": { "type": "integer" }, "rain_proof": { "type": "text", - "index": false + "index": true }, "rock_types": { "type": "text", - "index": false + "index": true }, "routes_quantity": { "type": "integer" }, "snow_clearance_rating": { "type": "text", - "index": false + "index": true }, "summary_ca": { "type": "text", @@ -336,11 +336,51 @@ }, "waypoint_type": { "type": "text", - "index": false + "index": true }, "weather_station_types": { "type": "text", - "index": false + "index": true + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file diff --git a/scripts/esjson5-6/x.json b/scripts/esjson5-6/x.json index 693194fa3..9071b569b 100644 --- a/scripts/esjson5-6/x.json +++ b/scripts/esjson5-6/x.json @@ -2,14 +2,14 @@ "properties": { "activities": { "type": "text", - "index": false + "index": true }, "areas": { "type": "long" }, "available_locales": { "type": "text", - "index": false + "index": true }, "avalanche_level": { "type": "integer" @@ -65,7 +65,7 @@ }, "event_type": { "type": "text", - "index": false + "index": true }, "geom": { "type": "geo_point", @@ -260,6 +260,46 @@ "analyzer": "italian_heavy" } } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } } } } \ No newline at end of file From 27109e70283abdee3c4953376635855042ba6bc4 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Fri, 7 Jun 2024 19:47:51 +0200 Subject: [PATCH 10/53] Upgrade Elasticsearch from 5.6 to 6.8 --- c2corg_api/search/__init__.py | 2 -- c2corg_api/search/search_filters.py | 2 -- 2 files changed, 4 deletions(-) diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index 7131ce627..61656d734 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -78,8 +78,6 @@ def __init__(self, settings): def create_search(document_type=None): - #TODO: En fonction du document type envoyer vers le bon index - #en fonction document_type qui contient le suffixe de l'index return Search( using=elasticsearch_config['client'], index=(elasticsearch_config['index']), diff --git a/c2corg_api/search/search_filters.py b/c2corg_api/search/search_filters.py index 7e2e85b7a..a44a99500 100644 --- a/c2corg_api/search/search_filters.py +++ b/c2corg_api/search/search_filters.py @@ -35,8 +35,6 @@ def build_query(url_params, meta_params, doc_type): if filter: search = search.filter(filter) - #TODO: Remettre le filtre - #search.filter(Q("term", c2corg_doc_type=doc_type)) search = search.filter(Q("match", c2corg_doc_type=doc_type)) search = search. \ From 7e3428a8354102f7c55b13a98d68c8369ea6c476 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:28:32 +0200 Subject: [PATCH 11/53] Upgrade Elasticsearch from 5.6 to 6.8 : - lint fixes --- c2corg_api/scripts/initializees.py | 44 ++++++++++--------- c2corg_api/search/__init__.py | 6 +-- c2corg_api/search/advanced_search.py | 4 -- c2corg_api/search/mapping.py | 2 +- c2corg_api/search/mappings/area_mapping.py | 6 +-- c2corg_api/search/mappings/book_mapping.py | 4 +- .../search/mappings/topo_map_mapping.py | 4 +- c2corg_api/search/mappings/user_mapping.py | 4 +- .../search/mappings/waypoint_mapping.py | 1 - c2corg_api/search/mappings/xreport_mapping.py | 4 +- c2corg_api/search/search.py | 3 +- c2corg_api/tests/__init__.py | 1 - .../tests/scripts/es/test_fill_index.py | 3 +- .../tests/search/test_search_filters.py | 2 +- c2corg_api/tests/views/__init__.py | 3 -- c2corg_api/tests/views/test_langs.py | 1 - c2corg_api/views/health.py | 6 +-- docker-compose.yml | 10 ++--- 18 files changed, 50 insertions(+), 58 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 4b291b398..50799faae 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -3,16 +3,16 @@ import pycurl from io import BytesIO -from c2corg_api.search.mappings.area_mapping import SearchArea -from c2corg_api.search.mappings.article_mapping import SearchArticle -from c2corg_api.search.mappings.book_mapping import SearchBook -from c2corg_api.search.mappings.image_mapping import SearchImage -from c2corg_api.search.mappings.outing_mapping import SearchOuting -from c2corg_api.search.mappings.xreport_mapping import SearchXreport -from c2corg_api.search.mappings.route_mapping import SearchRoute -from c2corg_api.search.mappings.topo_map_mapping import SearchTopoMap -from c2corg_api.search.mappings.user_mapping import SearchUser -from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint +#from c2corg_api.search.mappings.area_mapping import SearchArea +#from c2corg_api.search.mappings.article_mapping import SearchArticle +#from c2corg_api.search.mappings.book_mapping import SearchBook +#from c2corg_api.search.mappings.image_mapping import SearchImage +#from c2corg_api.search.mappings.outing_mapping import SearchOuting +#from c2corg_api.search.mappings.xreport_mapping import SearchXreport +#from c2corg_api.search.mappings.route_mapping import SearchRoute +#from c2corg_api.search.mappings.topo_map_mapping import SearchTopoMap +#from c2corg_api.search.mappings.user_mapping import SearchUser +#from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint from elasticsearch_dsl import Index @@ -23,7 +23,7 @@ from pyramid.scripts.common import parse_vars -from c2corg_api.search.mapping import analysis_settings +#from c2corg_api.search.mapping import analysis_settings from c2corg_api.search import configure_es_from_config, elasticsearch_config @@ -48,7 +48,7 @@ def main(argv=sys.argv): def setup_es(): """Create the ElasticSearch index and configure the mapping. """ - index_suffix_list=["_a","_b","_c","_i","_m","_o","_r","_u","_w","_x"] + index_suffix_list = ["_a", "_b", "_c", "_i", "_m", "_o", "_r", "_u", "_w", "_x"] client = elasticsearch_config['client'] index_name = elasticsearch_config['index'] @@ -63,20 +63,20 @@ def setup_es(): if client.indices.exists(index_name[:-2]+index_suffix): print('Index "{0}" already exists. deleting it {0}... ' .format(index_name[:-2]+index_suffix)) - #print('To delete the index run:') - #print('curl -XDELETE \'http://{0}:{1}/{2}/\''.format( + # print('To delete the index run:') + # print('curl -XDELETE \'http://{0}:{1}/{2}/\''.format( # elasticsearch_config['host'], elasticsearch_config['port'], # index_name+index_suffix)) delete_indice(cible, index_name[:-2]+index_suffix) - #sys.exit(0) + # sys.exit(0) for index_suffix in index_suffix_list: print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) - create_indice(cible,index_name[:-2]+index_suffix) + create_indice(cible, index_name[:-2]+index_suffix) print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) - indice_settings_update(cible,index_name[:-2]+index_suffix) + indice_settings_update(cible, index_name[:-2]+index_suffix) print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) - indice_mapping_update(cible,index_name[:-2]+index_suffix,index_suffix[1]) + indice_mapping_update(cible, index_name[:-2]+index_suffix,index_suffix[1]) print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) print('all indexes are created') @@ -99,7 +99,7 @@ def delete_indice(cible, indice_name): header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'DELETE') - c.setopt(c.URL, 'http://' + cible + '/' + indice_name ) + c.setopt(c.URL, 'http://' + cible + '/' + indice_name) c.setopt(c.WRITEDATA, buffer) c.perform() c.close() @@ -114,7 +114,7 @@ def create_indice(cible, indice_name): header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') - c.setopt(c.URL, 'http://' + cible + '/' + indice_name ) + c.setopt(c.URL, 'http://' + cible + '/' + indice_name) c.setopt(c.WRITEDATA, buffer) c.perform() c.close() @@ -122,9 +122,11 @@ def create_indice(cible, indice_name): body = buffer.getvalue() print(body.decode('iso-8859-1')) + def indice_settings_update(cible, indice_name): """curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_close""" - """curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_a/_settings""" + """curl -X PUT -H "Content-Type: application/json" + -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_a/_settings""" """curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_open""" buffer = BytesIO() c = pycurl.Curl() diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index 61656d734..e0a0ce766 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -1,5 +1,3 @@ -import logging - from c2corg_api.models.area import AREA_TYPE from c2corg_api.models.article import ARTICLE_TYPE from c2corg_api.models.book import BOOK_TYPE @@ -27,7 +25,7 @@ from elasticsearch_dsl.query import MultiMatch from kombu import Exchange, Queue, pools from kombu.connection import Connection -from pprint import pprint + # the maximum number of documents that can be returned for each document type SEARCH_LIMIT_MAX = 50 @@ -81,7 +79,7 @@ def create_search(document_type=None): return Search( using=elasticsearch_config['client'], index=(elasticsearch_config['index']), - doc_type="_doc", #search_documents[document_type] + doc_type="_doc" # search_documents[document_type] ) diff --git a/c2corg_api/search/advanced_search.py b/c2corg_api/search/advanced_search.py index f21892682..3cf2dc450 100644 --- a/c2corg_api/search/advanced_search.py +++ b/c2corg_api/search/advanced_search.py @@ -1,7 +1,3 @@ -import logging - -from IPython.lib.pretty import pprint - from c2corg_api.search.mapping_types import meta_param_keys from c2corg_api.search.search_filters import build_query diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index 20376f5da..023b597b4 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -138,7 +138,7 @@ def to_search_document(document, index, include_areas=True): search_document = { '_index': index, '_id': document.document_id, - '_type': "_doc",#document.type, + '_type': "_doc", # document.type, 'id': document.document_id } diff --git a/c2corg_api/search/mappings/area_mapping.py b/c2corg_api/search/mappings/area_mapping.py index 10606b98b..f76110224 100644 --- a/c2corg_api/search/mappings/area_mapping.py +++ b/c2corg_api/search/mappings/area_mapping.py @@ -1,11 +1,11 @@ -from c2corg_api.models.area import AREA_TYPE, Area +from c2corg_api.models.area import Area, AREA_TYPE from c2corg_api.search.mapping import SearchDocument, BaseMeta from c2corg_api.search.mapping_types import QueryableMixin, QEnum class SearchArea(SearchDocument): -# class Meta(BaseMeta): -# doc_type = AREA_TYPE + class Meta(BaseMeta): + c2corg_doc_type = AREA_TYPE area_type = QEnum('atyp', model_field=Area.area_type) diff --git a/c2corg_api/search/mappings/book_mapping.py b/c2corg_api/search/mappings/book_mapping.py index d5f5cc3e4..58a91130f 100644 --- a/c2corg_api/search/mappings/book_mapping.py +++ b/c2corg_api/search/mappings/book_mapping.py @@ -4,8 +4,8 @@ class SearchBook(SearchDocument): -# class Meta(BaseMeta): -# doc_type = BOOK_TYPE + class Meta(BaseMeta): + c2corg_doc_type = BOOK_TYPE activities = QEnumArray( 'act', model_field=Book.activities) diff --git a/c2corg_api/search/mappings/topo_map_mapping.py b/c2corg_api/search/mappings/topo_map_mapping.py index 9792779db..9beb28eb8 100644 --- a/c2corg_api/search/mappings/topo_map_mapping.py +++ b/c2corg_api/search/mappings/topo_map_mapping.py @@ -4,8 +4,8 @@ class SearchTopoMap(SearchDocument): -# class Meta(BaseMeta): -# doc_type = MAP_TYPE + class Meta(BaseMeta): + c2corg_doc_type = MAP_TYPE FIELDS = [] diff --git a/c2corg_api/search/mappings/user_mapping.py b/c2corg_api/search/mappings/user_mapping.py index a6c98c05d..c5443bae3 100644 --- a/c2corg_api/search/mappings/user_mapping.py +++ b/c2corg_api/search/mappings/user_mapping.py @@ -4,8 +4,8 @@ class SearchUser(SearchDocument): -# class Meta(BaseMeta): -# doc_type = USERPROFILE_TYPE + class Meta(BaseMeta): + c2corg_doc_type = USERPROFILE_TYPE FIELDS = [] diff --git a/c2corg_api/search/mappings/waypoint_mapping.py b/c2corg_api/search/mappings/waypoint_mapping.py index d4e553f37..628ff770b 100644 --- a/c2corg_api/search/mappings/waypoint_mapping.py +++ b/c2corg_api/search/mappings/waypoint_mapping.py @@ -10,7 +10,6 @@ class SearchWaypoint(SearchDocument): class Meta(BaseMeta): - #doc_type = WAYPOINT_TYPE c2corg_doc_type = WAYPOINT_TYPE elevation = QInteger( diff --git a/c2corg_api/search/mappings/xreport_mapping.py b/c2corg_api/search/mappings/xreport_mapping.py index 17fc71828..0a68e4936 100644 --- a/c2corg_api/search/mappings/xreport_mapping.py +++ b/c2corg_api/search/mappings/xreport_mapping.py @@ -8,8 +8,8 @@ class SearchXreport(SearchDocument): -# class Meta(BaseMeta): -# doc_type = XREPORT_TYPE + class Meta(BaseMeta): + c2corg_doc_type = XREPORT_TYPE event_activity = QEnumArray( 'act', model_field=Xreport.event_activity) diff --git a/c2corg_api/search/search.py b/c2corg_api/search/search.py index 28a6c9ccd..3711f79ee 100644 --- a/c2corg_api/search/search.py +++ b/c2corg_api/search/search.py @@ -5,10 +5,11 @@ from c2corg_api.search import create_search, elasticsearch_config, \ get_text_query_on_title from c2corg_api.views.document_listings import get_documents -from elasticsearch_dsl.search import MultiSearch, Search +from elasticsearch_dsl.search import MultiSearch # , Search log = logging.getLogger(__name__) + def search_for_types(search_types, search_term, limit, lang): """Get results for all given types. """ diff --git a/c2corg_api/tests/__init__.py b/c2corg_api/tests/__init__.py index 61dc2f5c4..ca56146da 100644 --- a/c2corg_api/tests/__init__.py +++ b/c2corg_api/tests/__init__.py @@ -1,5 +1,4 @@ import time -import pycurl import jwt import datetime diff --git a/c2corg_api/tests/scripts/es/test_fill_index.py b/c2corg_api/tests/scripts/es/test_fill_index.py index 915683d6c..f22e7fb0f 100644 --- a/c2corg_api/tests/scripts/es/test_fill_index.py +++ b/c2corg_api/tests/scripts/es/test_fill_index.py @@ -192,7 +192,8 @@ def test_fill_index(self): self.assertEqual(xreport.c2corg_doc_type, 'x') # merged document is ignored - self.assertIsNone(SearchWaypoint.get(id=71174, index='c2corg_tests_w', ignore=404)) + self.assertIsNone(SearchWaypoint.\ + get(id=71174, index='c2corg_tests_w', ignore=404)) # check that the sync. status was updated last_update, _ = es_sync.get_status(self.session) diff --git a/c2corg_api/tests/search/test_search_filters.py b/c2corg_api/tests/search/test_search_filters.py index fadb3fe12..391d059f7 100644 --- a/c2corg_api/tests/search/test_search_filters.py +++ b/c2corg_api/tests/search/test_search_filters.py @@ -23,7 +23,7 @@ def test_build_query(self): 'limit': 10, 'offset': 0 } - query = build_query(params, meta_params,'w') + query = build_query(params, meta_params, 'w') expected_query = create_search('w'). \ query(get_text_query_on_title('search word')). \ filter(Term(available_locales='fr')).\ diff --git a/c2corg_api/tests/views/__init__.py b/c2corg_api/tests/views/__init__.py index bff24d38f..d2b47c119 100644 --- a/c2corg_api/tests/views/__init__.py +++ b/c2corg_api/tests/views/__init__.py @@ -1,11 +1,8 @@ import json -import logging import urllib.request import urllib.parse import urllib.error -from IPython.lib.pretty import pprint - from c2corg_api.caching import cache_document_detail from c2corg_api.models.cache_version import CacheVersion, get_cache_key from c2corg_api.models.feed import DocumentChange diff --git a/c2corg_api/tests/views/test_langs.py b/c2corg_api/tests/views/test_langs.py index dc41cd13f..bf4ed4ef9 100644 --- a/c2corg_api/tests/views/test_langs.py +++ b/c2corg_api/tests/views/test_langs.py @@ -1,4 +1,3 @@ -import logging from datetime import date, datetime from c2corg_api.models.feed import DocumentChange diff --git a/c2corg_api/views/health.py b/c2corg_api/views/health.py index 2e00f027b..8b1c36495 100644 --- a/c2corg_api/views/health.py +++ b/c2corg_api/views/health.py @@ -1,5 +1,4 @@ import logging -import sys from c2corg_api.caching import cache_document_detail from c2corg_api.models import DBSession, es_sync @@ -68,8 +67,9 @@ def _add_es_status(self, status): stats = client.indices.stats(index, metric='docs') document_count_all_indice = 0 for indice in stats['indices']: - document_count_all_indice += stats['indices'][indice]['total']['docs']['count'] - #es_docs = stats['indices'][index]['total']['docs']['count'] + document_count_all_indice += \ + stats['indices'][indice]['total']['docs']['count'] + # es_docs = stats['indices'][index]['total']['docs']['count'] es_docs = document_count_all_indice success = True except Exception: diff --git a/docker-compose.yml b/docker-compose.yml index 7358d1344..59999053a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -15,10 +15,10 @@ services: environment: db_host: 'postgresql' tests_db_host: 'postgresql' - elasticsearch_host: '192.168.1.120' - elasticsearch_port: 9203 - tests_elasticsearch_host: '192.168.1.120' - tests_elasticsearch_port: 9203 + elasticsearch_host: 'elasticsearch' + elasticsearch_port: 9200 + tests_elasticsearch_host: 'elasticsearch' + tests_elasticsearch_port: 9200 redis_url: 'redis://redis:6379/' version: '' volumes: @@ -67,7 +67,7 @@ services: # - 9202:9200 # - 9302:9300 - elasticsearch68: + elasticsearch: platform: linux/amd64 container_name: elasticsearch build: From 8efbd58c38433f39ff7594c3bcea618acad60997 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:33:41 +0200 Subject: [PATCH 12/53] Upgrade Elasticsearch from 5.6 to 6.8 : - CI/CD : ES Download version --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5feddc541..4014fb679 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,7 +79,7 @@ jobs: - name: Run (old) elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.23.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs From 9ef3c074c59a6b91d340484b3924c1bc2bb37cad Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:33:41 +0200 Subject: [PATCH 13/53] Upgrade Elasticsearch from 5.6 to 6.8 : - lint fixes --- .github/workflows/ci.yml | 2 +- c2corg_api/scripts/initializees.py | 36 +++++++++---------- c2corg_api/search/mappings/image_mapping.py | 1 - c2corg_api/search/mappings/outing_mapping.py | 1 - c2corg_api/search/mappings/route_mapping.py | 1 - c2corg_api/search/search.py | 3 +- .../tests/scripts/es/test_fill_index.py | 4 +-- 7 files changed, 22 insertions(+), 26 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5feddc541..4014fb679 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,7 +79,7 @@ jobs: - name: Run (old) elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.23.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 50799faae..261f4e728 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -3,16 +3,16 @@ import pycurl from io import BytesIO -#from c2corg_api.search.mappings.area_mapping import SearchArea -#from c2corg_api.search.mappings.article_mapping import SearchArticle -#from c2corg_api.search.mappings.book_mapping import SearchBook -#from c2corg_api.search.mappings.image_mapping import SearchImage -#from c2corg_api.search.mappings.outing_mapping import SearchOuting -#from c2corg_api.search.mappings.xreport_mapping import SearchXreport -#from c2corg_api.search.mappings.route_mapping import SearchRoute -#from c2corg_api.search.mappings.topo_map_mapping import SearchTopoMap -#from c2corg_api.search.mappings.user_mapping import SearchUser -#from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint +# from c2corg_api.search.mappings.area_mapping import SearchArea +# from c2corg_api.search.mappings.article_mapping import SearchArticle +# from c2corg_api.search.mappings.book_mapping import SearchBook +# from c2corg_api.search.mappings.image_mapping import SearchImage +# from c2corg_api.search.mappings.outing_mapping import SearchOuting +# from c2corg_api.search.mappings.xreport_mapping import SearchXreport +# from c2corg_api.search.mappings.route_mapping import SearchRoute +# from c2corg_api.search.mappings.topo_map_mapping import SearchTopoMap +# from c2corg_api.search.mappings.user_mapping import SearchUser +# from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint from elasticsearch_dsl import Index @@ -23,7 +23,7 @@ from pyramid.scripts.common import parse_vars -#from c2corg_api.search.mapping import analysis_settings +# from c2corg_api.search.mapping import analysis_settings from c2corg_api.search import configure_es_from_config, elasticsearch_config @@ -48,12 +48,13 @@ def main(argv=sys.argv): def setup_es(): """Create the ElasticSearch index and configure the mapping. """ - index_suffix_list = ["_a", "_b", "_c", "_i", "_m", "_o", "_r", "_u", "_w", "_x"] + index_suffix_list = ["_a", "_b", "_c", "_i", \ + "_m", "_o", "_r", "_u", "_w", "_x"] client = elasticsearch_config['client'] index_name = elasticsearch_config['index'] cible = elasticsearch_config['host']+':'+str(elasticsearch_config['port']) - print('cible es: %s',cible) + print('cible es: %s', cible) info = client.info() print('ElasticSearch version: {0}'.format(info['version']['number'])) @@ -76,7 +77,8 @@ def setup_es(): print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) indice_settings_update(cible, index_name[:-2]+index_suffix) print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) - indice_mapping_update(cible, index_name[:-2]+index_suffix,index_suffix[1]) + indice_mapping_update(cible, \ + index_name[:-2]+index_suffix,index_suffix[1]) print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) print('all indexes are created') @@ -124,10 +126,6 @@ def create_indice(cible, indice_name): def indice_settings_update(cible, indice_name): - """curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_close""" - """curl -X PUT -H "Content-Type: application/json" - -d @./scripts/esjson5-6/settings.json http://192.168.1.120:9203/c2corg_a/_settings""" - """curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9203/c2corg_a/_open""" buffer = BytesIO() c = pycurl.Curl() header = ['Content-Type: application/json'] @@ -178,4 +176,4 @@ def indice_mapping_update(cible, indice_name, mapping_type): c.close() body = buffer.getvalue() - print(body.decode('iso-8859-1')) \ No newline at end of file + print(body.decode('iso-8859-1')) diff --git a/c2corg_api/search/mappings/image_mapping.py b/c2corg_api/search/mappings/image_mapping.py index e177a89b1..13feac38d 100644 --- a/c2corg_api/search/mappings/image_mapping.py +++ b/c2corg_api/search/mappings/image_mapping.py @@ -6,7 +6,6 @@ class SearchImage(SearchDocument): class Meta(BaseMeta): -# doc_type = IMAGE_TYPE c2corg_doc_type = IMAGE_TYPE activities = QEnumArray( diff --git a/c2corg_api/search/mappings/outing_mapping.py b/c2corg_api/search/mappings/outing_mapping.py index 879924895..5a5ffc631 100644 --- a/c2corg_api/search/mappings/outing_mapping.py +++ b/c2corg_api/search/mappings/outing_mapping.py @@ -17,7 +17,6 @@ class SearchOuting(SearchDocument): class Meta(BaseMeta): -# doc_type = OUTING_TYPE c2corg_doc_type = OUTING_TYPE date_start = Date() diff --git a/c2corg_api/search/mappings/route_mapping.py b/c2corg_api/search/mappings/route_mapping.py index 9c7242863..53fcd0d4f 100644 --- a/c2corg_api/search/mappings/route_mapping.py +++ b/c2corg_api/search/mappings/route_mapping.py @@ -17,7 +17,6 @@ class SearchRoute(SearchDocument): class Meta(BaseMeta): -# doc_type = ROUTE_TYPE c2corg_doc_type = ROUTE_TYPE # array of waypoint ids diff --git a/c2corg_api/search/search.py b/c2corg_api/search/search.py index 3711f79ee..429ef51ed 100644 --- a/c2corg_api/search/search.py +++ b/c2corg_api/search/search.py @@ -52,7 +52,8 @@ def do_multi_search_for_types(search_types, search_term, limit, lang): (_, get_documents_config) = search_type search = create_search(get_documents_config.document_type).\ query(get_text_query_on_title(search_term, lang)). \ - filter(Q("match", c2corg_doc_type=get_documents_config.document_type)). \ + filter(Q("match", + c2corg_doc_type=get_documents_config.document_type)). \ extra(from_=0, size=limit) multi_search = multi_search.add(search) diff --git a/c2corg_api/tests/scripts/es/test_fill_index.py b/c2corg_api/tests/scripts/es/test_fill_index.py index f22e7fb0f..7d9355f25 100644 --- a/c2corg_api/tests/scripts/es/test_fill_index.py +++ b/c2corg_api/tests/scripts/es/test_fill_index.py @@ -192,8 +192,8 @@ def test_fill_index(self): self.assertEqual(xreport.c2corg_doc_type, 'x') # merged document is ignored - self.assertIsNone(SearchWaypoint.\ - get(id=71174, index='c2corg_tests_w', ignore=404)) + self.assertIsNone(SearchWaypoint.get(id=71174, + index='c2corg_tests_w', ignore=404)) # check that the sync. status was updated last_update, _ = es_sync.get_status(self.session) From ae35883f2262d72fa2c3ff9b71fe44c3debe0608 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:45:50 +0200 Subject: [PATCH 14/53] Upgrade Elasticsearch from 5.6 to 6.8 : - lint fixes --- c2corg_api/scripts/initializees.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 261f4e728..44f0a39e2 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -48,7 +48,7 @@ def main(argv=sys.argv): def setup_es(): """Create the ElasticSearch index and configure the mapping. """ - index_suffix_list = ["_a", "_b", "_c", "_i", \ + index_suffix_list = ["_a", "_b", "_c", "_i", "_m", "_o", "_r", "_u", "_w", "_x"] client = elasticsearch_config['client'] @@ -77,7 +77,7 @@ def setup_es(): print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) indice_settings_update(cible, index_name[:-2]+index_suffix) print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) - indice_mapping_update(cible, \ + indice_mapping_update(cible, index_name[:-2]+index_suffix,index_suffix[1]) print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) From d92072cc06141432b662a1ae167c7b9be3e04748 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:49:51 +0200 Subject: [PATCH 15/53] Upgrade Elasticsearch from 5.6 to 6.8 : - lint fixes --- c2corg_api/scripts/initializees.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 44f0a39e2..8d4a63e1b 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -49,7 +49,7 @@ def setup_es(): """Create the ElasticSearch index and configure the mapping. """ index_suffix_list = ["_a", "_b", "_c", "_i", - "_m", "_o", "_r", "_u", "_w", "_x"] + "_m", "_o", "_r", "_u", "_w", "_x"] client = elasticsearch_config['client'] index_name = elasticsearch_config['index'] @@ -64,12 +64,11 @@ def setup_es(): if client.indices.exists(index_name[:-2]+index_suffix): print('Index "{0}" already exists. deleting it {0}... ' .format(index_name[:-2]+index_suffix)) - # print('To delete the index run:') - # print('curl -XDELETE \'http://{0}:{1}/{2}/\''.format( - # elasticsearch_config['host'], elasticsearch_config['port'], - # index_name+index_suffix)) + """ print('To delete the index run:') + print('curl -XDELETE \'http://{0}:{1}/{2}/\''.format( + elasticsearch_config['host'], elasticsearch_config['port'], + index_name+index_suffix)) """ delete_indice(cible, index_name[:-2]+index_suffix) - # sys.exit(0) for index_suffix in index_suffix_list: print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) From d2e5c62398028ca6367d6a30f6ad103b222aaeeb Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:53:24 +0200 Subject: [PATCH 16/53] Upgrade Elasticsearch from 5.6 to 6.8 : - lint fixes --- c2corg_api/scripts/initializees.py | 2 +- c2corg_api/search/search.py | 2 +- c2corg_api/tests/scripts/es/test_fill_index.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 8d4a63e1b..4cb58a37f 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -77,7 +77,7 @@ def setup_es(): indice_settings_update(cible, index_name[:-2]+index_suffix) print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) indice_mapping_update(cible, - index_name[:-2]+index_suffix,index_suffix[1]) + index_name[:-2]+index_suffix, index_suffix[1]) print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) print('all indexes are created') diff --git a/c2corg_api/search/search.py b/c2corg_api/search/search.py index 429ef51ed..97795185a 100644 --- a/c2corg_api/search/search.py +++ b/c2corg_api/search/search.py @@ -53,7 +53,7 @@ def do_multi_search_for_types(search_types, search_term, limit, lang): search = create_search(get_documents_config.document_type).\ query(get_text_query_on_title(search_term, lang)). \ filter(Q("match", - c2corg_doc_type=get_documents_config.document_type)). \ + c2corg_doc_type=get_documents_config.document_type)). \ extra(from_=0, size=limit) multi_search = multi_search.add(search) diff --git a/c2corg_api/tests/scripts/es/test_fill_index.py b/c2corg_api/tests/scripts/es/test_fill_index.py index 7d9355f25..f2f42f7f6 100644 --- a/c2corg_api/tests/scripts/es/test_fill_index.py +++ b/c2corg_api/tests/scripts/es/test_fill_index.py @@ -193,7 +193,7 @@ def test_fill_index(self): # merged document is ignored self.assertIsNone(SearchWaypoint.get(id=71174, - index='c2corg_tests_w', ignore=404)) + index='c2corg_tests_w', ignore=404)) # check that the sync. status was updated last_update, _ = es_sync.get_status(self.session) From 8f973a64c0a145f80b4ecbb036650e4da2167fb0 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 10:55:15 +0200 Subject: [PATCH 17/53] Upgrade Elasticsearch from 5.6 to 6.8 : - lint fixes --- c2corg_api/tests/scripts/es/test_fill_index.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/c2corg_api/tests/scripts/es/test_fill_index.py b/c2corg_api/tests/scripts/es/test_fill_index.py index f2f42f7f6..a65da19f0 100644 --- a/c2corg_api/tests/scripts/es/test_fill_index.py +++ b/c2corg_api/tests/scripts/es/test_fill_index.py @@ -193,7 +193,8 @@ def test_fill_index(self): # merged document is ignored self.assertIsNone(SearchWaypoint.get(id=71174, - index='c2corg_tests_w', ignore=404)) + index='c2corg_tests_w', + ignore=404)) # check that the sync. status was updated last_update, _ = es_sync.get_status(self.session) From 060bcbe6b2fde05087ab37b65de7cc95547694d8 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Sat, 8 Jun 2024 11:30:39 +0200 Subject: [PATCH 18/53] Upgrade Elasticsearch from 5.6 to 6.8 : - docker-api ES point --- config/docker-dev | 9 ++++++--- docker-compose.yml | 6 +++--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/config/docker-dev b/config/docker-dev index c5331b9ec..153dab388 100644 --- a/config/docker-dev +++ b/config/docker-dev @@ -7,16 +7,19 @@ include Makefile export db_host = postgresql export db_name = c2corg export elasticsearch_host = elasticsearch -export elasticsearch_index = c2corg +export elasticsearch_port = 9200 +export elasticsearch_index = c2corg_* export tests_db_host = postgresql export tests_db_name = c2corg_tests export tests_elasticsearch_host = elasticsearch -export tests_elasticsearch_index = c2corg_tests +export tests_elasticsearch_port = 9200 +export tests_elasticsearch_index = c2corg_tests_* export redis_db_queue = 6 export redis_db_cache = 7 export redis_url = redis://redis:6379/ # in case of unexpected errors, show the debug toolbar? -export show_debugger_for_errors = false +export logging_level = WARNING +export show_debugger_for_errors = true diff --git a/docker-compose.yml b/docker-compose.yml index 59999053a..7a6f5385c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -8,7 +8,7 @@ services: network: host depends_on: - postgresql -# - elasticsearch + - elasticsearch - redis ports: - 6543:6543 @@ -30,12 +30,12 @@ services: - ./test.ini.in:/var/www/test.ini.in - ./pytest.ini:/var/www/pytest.ini - ./requirements.txt:/var/www/requirements.txt - - ./config:/var/www/config + - ./config/docker-dev:/var/www/config/docker-dev - ./scripts:/var/www/scripts command: make -f config/docker-dev serve links: - postgresql -# - elasticsearch + - elasticsearch - redis postgresql: From 169283baf7131e58ff02fb70618824364f9e4bde Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 11 Jun 2024 12:09:16 +0200 Subject: [PATCH 19/53] Upgrade Elasticsearch from 5.6 to 6.8 : - Update settings es in config file for github-actions --- config/github-actions | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/github-actions b/config/github-actions index 54e6fbd53..5678ff812 100644 --- a/config/github-actions +++ b/config/github-actions @@ -5,7 +5,7 @@ export base_url = /${instanceid} export db_name = c2corg_github_tests export tests_db_name = c2corg_github_tests export elasticsearch_port = 9200 -export elasticsearch_index = c2corg_github_tests +export elasticsearch_index = c2corg_github_tests_* export tests_elasticsearch_port = 9200 -export tests_elasticsearch_index = c2corg_github_tests +export tests_elasticsearch_index = c2corg_github_tests_* export redis_url = redis://localhost:6379/ \ No newline at end of file From 88630bbba44b80148acb5816e34ae17a4afaae8f Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 11 Jun 2024 12:24:22 +0200 Subject: [PATCH 20/53] Upgrade Elasticsearch from 5.6 to 6.8 : - test_fill_index fix coquille --- .../tests/scripts/es/test_fill_index.py | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/c2corg_api/tests/scripts/es/test_fill_index.py b/c2corg_api/tests/scripts/es/test_fill_index.py index a65da19f0..5b719e893 100644 --- a/c2corg_api/tests/scripts/es/test_fill_index.py +++ b/c2corg_api/tests/scripts/es/test_fill_index.py @@ -16,6 +16,7 @@ from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint from c2corg_api.tests import BaseTestCase from c2corg_api.scripts.es.fill_index import fill_index +from c2corg_api.search import elasticsearch_config class FillIndexTest(BaseTestCase): @@ -141,8 +142,8 @@ def test_fill_index(self): # fill the ElasticSearch index fill_index(self.session) - - waypoint1 = SearchWaypoint.get(id=71171, index='c2corg_tests_w') + index = elasticsearch_config['index'] + waypoint1 = SearchWaypoint.get(id=71171, index=index[:-1]+'w') self.assertIsNotNone(waypoint1) self.assertEqual(waypoint1.title_en, 'Mont Granier') self.assertEqual(waypoint1.title_fr, 'Mont Granier') @@ -153,47 +154,47 @@ def test_fill_index(self): self.assertAlmostEqual(waypoint1.geom[0], 5.71288994) self.assertAlmostEqual(waypoint1.geom[1], 45.64476395) - waypoint2 = SearchWaypoint.get(id=71172, index='c2corg_tests_w') + waypoint2 = SearchWaypoint.get(id=71172, index=index[:-1]+'w') self.assertIsNotNone(waypoint2) self.assertEqual(waypoint2.title_en, 'Mont Blanc') self.assertIsNone(waypoint2.title_fr) self.assertEqual(waypoint2.c2corg_doc_type, 'w') - route = SearchRoute.get(id=71173, index='c2corg_tests_r') + route = SearchRoute.get(id=71173, index=index[:-1]+'r') self.assertIsNotNone(route) self.assertEqual(route.title_en, 'Mont Blanc : Face N') self.assertIsNone(route.title_fr) self.assertEqual(route.c2corg_doc_type, 'r') self.assertEqual(route.durations, [0]) - outing = SearchOuting.get(id=71175, index='c2corg_tests_o') + outing = SearchOuting.get(id=71175, index=index[:-1]+'o') self.assertIsNotNone(outing) self.assertEqual(outing.title_en, 'Mont Blanc : Face N !') self.assertIsNone(outing.title_fr) self.assertEqual(outing.c2corg_doc_type, 'o') self.assertEqual(outing.frequentation, 3) - article = SearchArticle.get(id=71176, index='c2corg_tests_c') + article = SearchArticle.get(id=71176, index=index[:-1]+'c') self.assertIsNotNone(article) self.assertEqual(article.title_en, 'Lac d\'Annecy') self.assertEqual(article.title_fr, 'Lac d\'Annecy') self.assertEqual(article.c2corg_doc_type, 'c') - book = SearchBook.get(id=71177, index='c2corg_tests_b') + book = SearchBook.get(id=71177, index=index[:-1]+'b') self.assertIsNotNone(book) self.assertEqual(book.title_en, 'Escalades au Thaurac') self.assertEqual(book.title_fr, 'Escalades au Thaurac') self.assertEqual(book.c2corg_doc_type, 'b') self.assertEqual(book.book_types, ['biography']) - xreport = SearchXreport.get(id=71178, index='c2corg_tests_x') + xreport = SearchXreport.get(id=71178, index=index[:-1]+'x') self.assertIsNotNone(xreport) self.assertEqual(xreport.title_en, 'Death in the mountains') self.assertEqual(xreport.c2corg_doc_type, 'x') # merged document is ignored self.assertIsNone(SearchWaypoint.get(id=71174, - index='c2corg_tests_w', + index=index[:-1]+'w', ignore=404)) # check that the sync. status was updated From ce8c934be93f714415af07fdc643420eb3f344cc Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 11 Jun 2024 15:13:29 +0200 Subject: [PATCH 21/53] Upgrade Elasticsearch from 5.6 to 6.8 : - update doc --- scripts/README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/scripts/README.md b/scripts/README.md index 88a29f035..73b239892 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,3 +1,30 @@ +# ElasticSearch Upgrade from v5.6 to V6.8 for docker env. like dev one. + +The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. +The ElasticSearch 5.6 is still build and run but would not be the main ES for the service. It could be decommissioned after migration. +docker-compose file contains as well : + +- api building with integration of the requirements.txt for python pip installation (component updates) +- ES 5.6 listening on port 9201 instead of 9200 : > elasticsearch26 +- redis no change +- postgresql no change +- ES 6.8 listening on port 9200 and 9300 (elasticsearch6.Dockerfile) > elasticsearch +- Logstash 6.8 for data migration purpose. (logstash6.Dockerfile) + +## migration steps + + 1. **build the environment :** ` docker-compose build ` + 2. **Launch the environment :** ` docker-compose up -d ` + 3. **Create the index and its mapping :** `./scripts/esUpdateMappings56.sh` + 4. **Data Migration :** `docker-compose start logstash6` It will stop by itself when migration would be achieved + 5. **postgresql shema up to date:** `docker-compose exec api .build/venv/bin/alembic upgrade head` + +## post migration + +if all the process has been passed without issues. you can comment or remove in the docker-compose.yml services : +- logstash +- elasticsearch56 + # ElasticSearch Upgrade from v2.6 to V5.6 for docker env. like dev one. The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. From 3d66fcc09caef3ebc2e6c13a05526769cdfe42c2 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 13 Jun 2024 12:48:39 +0200 Subject: [PATCH 22/53] Upgrade Elasticsearch from 6.8 to 7.X : - update mapping and code --- c2corg_api/scripts/es/fill_index.py | 1 + c2corg_api/scripts/initializees.py | 9 +- c2corg_api/search/__init__.py | 2 +- c2corg_api/search/mapping.py | 13 +- c2corg_api/search/mapping_types.py | 3 + c2corg_api/search/mappings/article_mapping.py | 1 + docker-compose.yml | 18 +- elasticsearch7.Dockerfile | 2 + logstash7.Dockerfile | 2 + requirements.txt | 4 +- scripts/esUpdateMapSettings67.sh | 92 + scripts/esjson6-7/a.json | 279 ++ scripts/esjson6-7/b.json | 283 ++ scripts/esjson6-7/c.json | 287 ++ scripts/esjson6-7/cluster.json | 10 + scripts/esjson6-7/i.json | 294 ++ scripts/esjson6-7/m.json | 275 ++ scripts/esjson6-7/mappings.json | 2823 +++++++++++++++++ scripts/esjson6-7/o.json | 375 +++ scripts/esjson6-7/r.json | 406 +++ scripts/esjson6-7/settings.json | 378 +++ scripts/esjson6-7/u.json | 275 ++ scripts/esjson6-7/w.json | 386 +++ scripts/esjson6-7/x.json | 305 ++ 24 files changed, 6511 insertions(+), 12 deletions(-) create mode 100644 elasticsearch7.Dockerfile create mode 100644 logstash7.Dockerfile create mode 100755 scripts/esUpdateMapSettings67.sh create mode 100644 scripts/esjson6-7/a.json create mode 100644 scripts/esjson6-7/b.json create mode 100644 scripts/esjson6-7/c.json create mode 100644 scripts/esjson6-7/cluster.json create mode 100644 scripts/esjson6-7/i.json create mode 100644 scripts/esjson6-7/m.json create mode 100644 scripts/esjson6-7/mappings.json create mode 100644 scripts/esjson6-7/o.json create mode 100644 scripts/esjson6-7/r.json create mode 100644 scripts/esjson6-7/settings.json create mode 100644 scripts/esjson6-7/u.json create mode 100644 scripts/esjson6-7/w.json create mode 100644 scripts/esjson6-7/x.json diff --git a/c2corg_api/scripts/es/fill_index.py b/c2corg_api/scripts/es/fill_index.py index 840c3cf06..cd41f27af 100644 --- a/c2corg_api/scripts/es/fill_index.py +++ b/c2corg_api/scripts/es/fill_index.py @@ -81,6 +81,7 @@ def progress(count, total_count): for doc in sync.get_documents(session, doc_type, batch_size, ignore_redirects=True): + logging.warning("JPR fill_index doc_type : %s", doc_type) batch.add(to_search_document(doc, index_name[:-1] + doc_type)) count += 1 diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 4cb58a37f..713b23fea 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -136,7 +136,7 @@ def indice_settings_update(cible, indice_name): body = buffer.getvalue() print(body.decode('iso-8859-1')) - file = "./scripts/esjson5-6/settings.json" + file = "./scripts/esjson6-7/settings.json" f = open(file) post_data = f.read() c.setopt(c.CUSTOMREQUEST, 'PUT') @@ -146,7 +146,10 @@ def indice_settings_update(cible, indice_name): c.perform() body = buffer.getvalue() print(body.decode('iso-8859-1')) + c.close() + c = pycurl.Curl() + header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'POST') c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_open') @@ -159,13 +162,13 @@ def indice_settings_update(cible, indice_name): def indice_mapping_update(cible, indice_name, mapping_type): - file = "./scripts/esjson5-6/"+mapping_type+".json" + file = "./scripts/esjson6-7/"+mapping_type+".json" f = open(file) post_data = f.read() buffer = BytesIO() c = pycurl.Curl() - c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping/_doc') + c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping') header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index e0a0ce766..38d8c76f8 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -79,7 +79,7 @@ def create_search(document_type=None): return Search( using=elasticsearch_config['client'], index=(elasticsearch_config['index']), - doc_type="_doc" # search_documents[document_type] + #doc_type="_doc" # search_documents[document_type] ) diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index 023b597b4..b85ffa2f1 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -1,4 +1,6 @@ import json +import logging +import pprint from c2corg_api.models.document import Document from c2corg_api.search.mapping_types import Enum, QEnumArray, QLong, \ @@ -6,7 +8,7 @@ from c2corg_api.models.common.attributes import default_langs from c2corg_api.models.common.sortable_search_attributes import \ sortable_quality_types -from elasticsearch_dsl import DocType, MetaField, Long, GeoPoint, Text +from elasticsearch_dsl import MetaField, Long, GeoPoint, Text class BaseMeta: @@ -42,7 +44,7 @@ def default_title_field(lang: None): }) -class SearchDocument(DocType): +class SearchDocument(): """The base ElasticSearch mapping for documents. Each document type has its own specific mapping. @@ -138,7 +140,7 @@ def to_search_document(document, index, include_areas=True): search_document = { '_index': index, '_id': document.document_id, - '_type': "_doc", # document.type, + #'_type': "_doc", # document.type, 'id': document.document_id } @@ -193,7 +195,10 @@ def copy_fields(search_document, document, fields): @staticmethod def copy_enum_range_fields( search_document, document, fields, search_model): - search_fields = search_model._doc_type.mapping + + search_fields = search_model.c2corg_doc_type.mapping + logging.warning("search_fields : %s", search_fields) + for field in fields: search_field = search_fields[field] enum_mapper = search_field._enum_mapper diff --git a/c2corg_api/search/mapping_types.py b/c2corg_api/search/mapping_types.py index 0ce2b3c28..2f56700dc 100644 --- a/c2corg_api/search/mapping_types.py +++ b/c2corg_api/search/mapping_types.py @@ -1,3 +1,5 @@ +import pprint + from elasticsearch_dsl import Text, Long, Integer, Boolean, Date # this module contains classes to mark the fields of a mapping that can be @@ -58,6 +60,7 @@ def __init__(self, query_name, *args, **kwargs): @staticmethod def get_queryable_fields(search_model): queryable_fields = {} + pprint.pprint(search_model) fields = search_model._doc_type.mapping for field_name in fields: field = fields[field_name] diff --git a/c2corg_api/search/mappings/article_mapping.py b/c2corg_api/search/mappings/article_mapping.py index 34aa537f2..2cf7033f8 100644 --- a/c2corg_api/search/mappings/article_mapping.py +++ b/c2corg_api/search/mappings/article_mapping.py @@ -16,6 +16,7 @@ class Meta(BaseMeta): FIELDS = ['activities', 'article_type'] + @staticmethod def to_search_document(document, index): search_document = SearchDocument.to_search_document(document, index) diff --git a/docker-compose.yml b/docker-compose.yml index 7a6f5385c..daab3cf1c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -68,17 +68,31 @@ services: # - 9302:9300 elasticsearch: - platform: linux/amd64 + #platform: linux/amd64 container_name: elasticsearch build: context: . - dockerfile: elasticsearch6.Dockerfile + dockerfile: elasticsearch7.Dockerfile network: host environment: - discovery.type=single-node ports: - 9200:9200 - 9300:9300 +# + +# elasticsearch68: +# platform: linux/amd64 +# container_name: elasticsearch68 +# build: +# context: . +# dockerfile: elasticsearch6.Dockerfile +# network: host +# environment: +# - discovery.type=single-node +# ports: +# - 9200:9200 +# - 9300:9300 # volumes: # - ./volumes/es6/data:/usr/share/elasticsearch/data # - ./volumes/es6/config/jvm.options:/usr/share/elasticsearch/config/jvm.options diff --git a/elasticsearch7.Dockerfile b/elasticsearch7.Dockerfile new file mode 100644 index 000000000..8d70023ae --- /dev/null +++ b/elasticsearch7.Dockerfile @@ -0,0 +1,2 @@ +FROM elasticsearch:7.17.21 +RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu diff --git a/logstash7.Dockerfile b/logstash7.Dockerfile new file mode 100644 index 000000000..8ac2043d8 --- /dev/null +++ b/logstash7.Dockerfile @@ -0,0 +1,2 @@ +FROM logstash:7.17.21 +RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/requirements.txt b/requirements.txt index e1f5f7402..1786c2fcf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,8 +4,8 @@ bcrypt==4.1.2 bleach[css]==6.1.0 colander==2.0 dogpile.cache==1.3.2 -elasticsearch==7.0.0 #elasticsearch==2.4.1 -elasticsearch-dsl==7.0.0 #elasticsearch_dsl==2.2.0 +elasticsearch==7.17.9 #elasticsearch==2.4.1 +elasticsearch-dsl==7.4.1 #elasticsearch_dsl==2.2.0 geoalchemy2==0.4.2 geojson==3.1.0 geomet==1.1.0 diff --git a/scripts/esUpdateMapSettings67.sh b/scripts/esUpdateMapSettings67.sh new file mode 100755 index 000000000..19c9927f4 --- /dev/null +++ b/scripts/esUpdateMapSettings67.sh @@ -0,0 +1,92 @@ +#!/bin/bash +#change localhost to the correct url if elasticsearch is not hosted on your computer, or in docker +#operate the plugin installation from the server side manually if not dockerized. +#installation du plugin es et reboot +#Install elastic plugin analysis-icu +#echo 'Installing es plugin to elasticsearch docker' +#docker-compose exec -it elasticsearch bin/plugin install analysis-icu +#echo 'Installing done - restarting es docker' +#docker-compose restart elasticsearch +#sleep 10 #in sec to let the docker restart + +#efface tous les indices présents +curl -X DELETE "http://localhost:9200/c2corg_*" + +#change cluster definition des shards +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/cluster.json http://localhost:9200/_template/index_defaults + +#créer les index +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_a +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_b +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_c +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_i +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_m +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_o +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_r +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_u +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_w +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_x +sleep 2 + +#2. closing the index to upgrade settings +echo -e '\n closing index' +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_a/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_b/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_c/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_i/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_m/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_o/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_r/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_u/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_w/_close +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_x/_close +sleep 2 +#3. Updating analyser settings +echo -e '\n updating Settings' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_a/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_b/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_c/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_i/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_m/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_o/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_r/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_u/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_w/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_x/_settings +sleep 2 +#4. restarting the index +echo -e '\n restarting index' +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_a/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_b/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_c/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_i/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_m/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_o/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_r/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_u/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_w/_open +curl -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_x/_open +sleep 2 +#5. Docs Mappings update +echo 'updating doc type mappings' +echo 'doc type : a' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://localhost:9200/c2corg_a/_mapping +echo -e '\n doc type : b' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://localhost:9200/c2corg_b/_mapping +echo -e '\n doc type : c' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://localhost:9200/c2corg_c/_mapping +echo -e '\n doc type : i' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://localhost:9200/c2corg_i/_mapping +echo -e '\n doc type : m' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://localhost:9200/c2corg_m/_mapping +echo -e '\n doc type : o' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://localhost:9200/c2corg_o/_mapping +echo -e '\n doc type : r' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://localhost:9200/c2corg_r/_mapping +echo -e '\n doc type : u' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://localhost:9200/c2corg_u/_mapping +echo -e '\n doc type : w' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://localhost:9200/c2corg_w/_mapping +echo -e '\n doc type : x' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://localhost:9200/c2corg_x/_mapping +sleep 2 diff --git a/scripts/esjson6-7/a.json b/scripts/esjson6-7/a.json new file mode 100644 index 000000000..1b3a46031 --- /dev/null +++ b/scripts/esjson6-7/a.json @@ -0,0 +1,279 @@ +{ + "properties": { + "area_type": { + "type": "text", + "index": true + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/b.json b/scripts/esjson6-7/b.json new file mode 100644 index 000000000..6fa5d297e --- /dev/null +++ b/scripts/esjson6-7/b.json @@ -0,0 +1,283 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": true + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "book_types": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/c.json b/scripts/esjson6-7/c.json new file mode 100644 index 000000000..a5d6ebf96 --- /dev/null +++ b/scripts/esjson6-7/c.json @@ -0,0 +1,287 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": true + }, + "areas": { + "type": "long" + }, + "article_categories": { + "type": "text", + "index": true + }, + "article_type": { + "type": "text", + "index": true + }, + "available_locales": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/cluster.json b/scripts/esjson6-7/cluster.json new file mode 100644 index 000000000..4e2491d7e --- /dev/null +++ b/scripts/esjson6-7/cluster.json @@ -0,0 +1,10 @@ +{ + "index_patterns": [ + "*" + ], + "order": 0, + "settings": { + "number_of_shards": 1, + "number_of_replicas": 0 + } +} \ No newline at end of file diff --git a/scripts/esjson6-7/i.json b/scripts/esjson6-7/i.json new file mode 100644 index 000000000..a58b744b1 --- /dev/null +++ b/scripts/esjson6-7/i.json @@ -0,0 +1,294 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": true + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "categories": { + "type": "text", + "index": true + }, + "date_time": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "image_type": { + "type": "text", + "index": false + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/m.json b/scripts/esjson6-7/m.json new file mode 100644 index 000000000..8f23d485f --- /dev/null +++ b/scripts/esjson6-7/m.json @@ -0,0 +1,275 @@ +{ + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/mappings.json b/scripts/esjson6-7/mappings.json new file mode 100644 index 000000000..67c9fff5b --- /dev/null +++ b/scripts/esjson6-7/mappings.json @@ -0,0 +1,2823 @@ +{ + "c2corg": { + "mappings": { + "x": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "avalanche_level": { + "type": "integer" + }, + "avalanche_slope": { + "type": "integer" + }, + "date": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "event_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "nb_impacted": { + "type": "integer" + }, + "nb_participants": { + "type": "integer" + }, + "quality": { + "type": "integer" + }, + "severity": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "c": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "article_categories": { + "type": "text", + "index": "not_analyzed" + }, + "article_type": { + "type": "text", + "index": "not_analyzed" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "u": { + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "o": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "avalanche_signs": { + "type": "text", + "index": "not_analyzed" + }, + "condition_rating": { + "type": "integer" + }, + "date_end": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "date_start": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation_access": { + "type": "integer" + }, + "elevation_down_snow": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_up_snow": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "frequentation": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "glacier_rating": { + "type": "integer" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "length_total": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "public_transport": { + "type": "boolean" + }, + "quality": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "routes": { + "type": "long" + }, + "ski_rating": { + "type": "integer" + }, + "snow_quality": { + "type": "integer" + }, + "snow_quantity": { + "type": "integer" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "users": { + "type": "long" + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + }, + "m": { + "_all": { + "enabled": false + }, + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "b": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "book_types": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "w": { + "_all": { + "enabled": false + }, + "properties": { + "access_time": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "best_periods": { + "type": "text", + "index": "not_analyzed" + }, + "capacity": { + "type": "integer" + }, + "capacity_staffed": { + "type": "integer" + }, + "children_proof": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_indoor_types": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_outdoor_types": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_rating_max": { + "type": "integer" + }, + "climbing_rating_median": { + "type": "integer" + }, + "climbing_rating_min": { + "type": "integer" + }, + "climbing_styles": { + "type": "text", + "index": "not_analyzed" + }, + "custodianship": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "equipment_ratings": { + "type": "integer" + }, + "exposition_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "height_max": { + "type": "integer" + }, + "height_median": { + "type": "integer" + }, + "height_min": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "length": { + "type": "integer" + }, + "lift_access": { + "type": "boolean" + }, + "orientations": { + "type": "text", + "index": "not_analyzed" + }, + "paragliding_rating": { + "type": "integer" + }, + "product_types": { + "type": "text", + "index": "not_analyzed" + }, + "prominence": { + "type": "integer" + }, + "public_transportation_rating": { + "type": "text", + "index": "not_analyzed" + }, + "public_transportation_types": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "rain_proof": { + "type": "text", + "index": "not_analyzed" + }, + "rock_types": { + "type": "text", + "index": "not_analyzed" + }, + "routes_quantity": { + "type": "integer" + }, + "snow_clearance_rating": { + "type": "text", + "index": "not_analyzed" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "waypoint_type": { + "type": "text", + "index": "not_analyzed" + }, + "weather_station_types": { + "type": "text", + "index": "not_analyzed" + } + } + }, + "a": { + "_all": { + "enabled": false + }, + "properties": { + "area_type": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + } + } + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "i": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "categories": { + "type": "text", + "index": "not_analyzed" + }, + "date_time": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "elevation": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "id": { + "type": "long" + }, + "image_type": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + } + } + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + } + } + }, + "r": { + "_all": { + "enabled": false + }, + "properties": { + "activities": { + "type": "text", + "index": "not_analyzed" + }, + "aid_rating": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": "not_analyzed" + }, + "climbing_outdoor_type": { + "type": "text", + "index": "not_analyzed" + }, + "configuration": { + "type": "text", + "index": "not_analyzed" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "difficulties_height": { + "type": "integer" + }, + "doc_type": { + "type": "text", + "index": "not_analyzed" + }, + "durations": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_min": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "exposition_rock_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point" + }, + "glacier_gear": { + "type": "text", + "index": "not_analyzed" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_access": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_down": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_mtb_exposition": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "labande_ski_rating": { + "type": "integer" + }, + "mixed_rating": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_height_diff_portages": { + "type": "integer" + }, + "mtb_length_asphalt": { + "type": "integer" + }, + "mtb_length_trail": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "orientations": { + "type": "text", + "index": "not_analyzed" + }, + "quality": { + "type": "integer" + }, + "risk_rating": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "rock_required_rating": { + "type": "integer" + }, + "rock_types": { + "type": "text", + "index": "not_analyzed" + }, + "route_length": { + "type": "integer" + }, + "route_types": { + "type": "text", + "index": "not_analyzed" + }, + "ski_exposition": { + "type": "integer" + }, + "ski_rating": { + "type": "integer" + }, + "slackline_type": { + "type": "text", + "index": "not_analyzed" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": "not_analyzed", + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + } + } + } + } + } +} diff --git a/scripts/esjson6-7/o.json b/scripts/esjson6-7/o.json new file mode 100644 index 000000000..b4fa1be8c --- /dev/null +++ b/scripts/esjson6-7/o.json @@ -0,0 +1,375 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": true + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "avalanche_signs": { + "type": "text", + "index": true + }, + "condition_rating": { + "type": "integer" + }, + "date_end": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "date_start": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation_access": { + "type": "integer" + }, + "elevation_down_snow": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_up_snow": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "frequentation": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "glacier_rating": { + "type": "integer" + }, + "global_rating": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "length_total": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "public_transport": { + "type": "boolean" + }, + "quality": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "routes": { + "type": "long" + }, + "ski_rating": { + "type": "integer" + }, + "snow_quality": { + "type": "integer" + }, + "snow_quantity": { + "type": "integer" + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "users": { + "type": "long" + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/r.json b/scripts/esjson6-7/r.json new file mode 100644 index 000000000..16ebe15ca --- /dev/null +++ b/scripts/esjson6-7/r.json @@ -0,0 +1,406 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": true + }, + "aid_rating": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "climbing_outdoor_type": { + "type": "text", + "index": true + }, + "configuration": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "difficulties_height": { + "type": "integer" + }, + "doc_type": { + "type": "text", + "index": false + }, + "durations": { + "type": "integer" + }, + "elevation_max": { + "type": "integer" + }, + "elevation_min": { + "type": "integer" + }, + "engagement_rating": { + "type": "integer" + }, + "equipment_rating": { + "type": "integer" + }, + "exposition_rock_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "glacier_gear": { + "type": "text", + "index": true + }, + "global_rating": { + "type": "integer" + }, + "height_diff_access": { + "type": "integer" + }, + "height_diff_difficulties": { + "type": "integer" + }, + "height_diff_down": { + "type": "integer" + }, + "height_diff_up": { + "type": "integer" + }, + "hiking_mtb_exposition": { + "type": "integer" + }, + "hiking_rating": { + "type": "integer" + }, + "ice_rating": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "labande_global_rating": { + "type": "integer" + }, + "labande_ski_rating": { + "type": "integer" + }, + "mixed_rating": { + "type": "integer" + }, + "mtb_down_rating": { + "type": "integer" + }, + "mtb_height_diff_portages": { + "type": "integer" + }, + "mtb_length_asphalt": { + "type": "integer" + }, + "mtb_length_trail": { + "type": "integer" + }, + "mtb_up_rating": { + "type": "integer" + }, + "orientations": { + "type": "text", + "index": true + }, + "quality": { + "type": "integer" + }, + "risk_rating": { + "type": "integer" + }, + "rock_free_rating": { + "type": "integer" + }, + "rock_required_rating": { + "type": "integer" + }, + "rock_types": { + "type": "text", + "index": true + }, + "route_length": { + "type": "integer" + }, + "route_types": { + "type": "text", + "index": true + }, + "ski_exposition": { + "type": "integer" + }, + "ski_rating": { + "type": "integer" + }, + "slackline_type": { + "type": "text", + "index": true + }, + "snowshoe_rating": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "via_ferrata_rating": { + "type": "integer" + }, + "waypoints": { + "type": "long" + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/settings.json b/scripts/esjson6-7/settings.json new file mode 100644 index 000000000..ff95e3ef8 --- /dev/null +++ b/scripts/esjson6-7/settings.json @@ -0,0 +1,378 @@ +{ + "analysis": { + "filter": { + "autocomplete_filter": { + "type": "edge_ngram", + "min_gram": "2", + "max_gram": "15", + "token_chars": [ + "letter", + "digit" + ] + }, + "english_stop": { + "type": "stop", + "stopwords": "_english_" + }, + "english_stemmer": { + "type": "stemmer", + "language": "english" + }, + "english_possessive_stemmer": { + "type": "stemmer", + "language": "possessive_english" + }, + "french_elision": { + "type": "elision", + "articles_case": true, + "articles": [ + "l", "m", "t", "qu", "n", "s", + "j", "d", "c", "jusqu", "quoiqu", + "lorsqu", "puisqu" + ] + }, + "french_stop": { + "type": "stop", + "stopwords": "_french_" + }, + "french_stemmer": { + "type": "stemmer", + "language": "light_french" + }, + "german_stop": { + "type": "stop", + "stopwords": "_german_" + }, + "german_stemmer": { + "type": "stemmer", + "language": "light_german" + }, + "italian_elision": { + "type": "elision", + "articles_case": true, + "articles": [ + "c", "l", "all", "dall", "dell", + "nell", "sull", "coll", "pell", + "gl", "agl", "dagl", "degl", "negl", + "sugl", "un", "m", "t", "s", "v", "d" + ] + }, + "italian_stop": { + "type": "stop", + "stopwords": "_italian_" + }, + "italian_stemmer": { + "type": "stemmer", + "language": "light_italian" + }, + "spanish_stop": { + "type": "stop", + "stopwords": "_spanish_" + }, + "spanish_stemmer": { + "type": "stemmer", + "language": "light_spanish" + }, + "catalan_elision": { + "type": "elision", + "articles_case": true, + "articles": ["d", "l", "m", "n", "s", "t"] + }, + "catalan_stop": { + "type": "stop", + "stopwords": "_catalan_" + }, + "catalan_stemmer": { + "type": "stemmer", + "language": "catalan" + }, + "basque_stop": { + "type": "stop", + "stopwords": "_basque_" + }, + "basque_stemmer": { + "type": "stemmer", + "language": "basque" + } + }, + "similarity": { + "c2corgsimilarity": { + "type": "BM25" + } + }, + "char_filter": { + "punctuationgreedy": { + "type": "pattern_replace", + "pattern": "[\\.,]" + } + }, + "analyzer": { + "index_ngram": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "icu_folding", + "autocomplete_filter"], + "tokenizer": "icu_tokenizer" + }, + "search_ngram": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "index_raw": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "search_raw": { + "char_filter": ["punctuationgreedy"], + "filter": [ + "word_delimiter", "lowercase", "asciifolding", "unique"], + "tokenizer": "standard" + }, + "index_english": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "english_possessive_stemmer", + "lowercase", + "english_stop", + "english_stemmer", + "autocomplete_filter" + ] + }, + "search_english": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "english_possessive_stemmer", + "lowercase", + "english_stop", + "english_stemmer" + ] + }, + "index_french": { + "tokenizer": "standard", + "filter": [ + "french_elision", + "lowercase", + "french_stop", + "french_stemmer", + "autocomplete_filter" + ] + }, + "search_french": { + "tokenizer": "standard", + "filter": [ + "french_elision", + "lowercase", + "french_stop", + "french_stemmer", + "autocomplete_filter" + ] + }, + "index_german": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "german_stop", + "german_normalization", + "german_stemmer", + "autocomplete_filter" + ] + }, + "search_german": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "german_stop", + "german_normalization", + "german_stemmer" + ] + }, + "index_italian": { + "tokenizer": "standard", + "filter": [ + "italian_elision", + "lowercase", + "italian_stop", + "italian_stemmer", + "autocomplete_filter" + ] + }, + "search_italian": { + "tokenizer": "standard", + "filter": [ + "italian_elision", + "lowercase", + "italian_stop", + "italian_stemmer" + ] + }, + "index_spanish": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer", + "autocomplete_filter" + ] + }, + "search_spanish": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer" + ] + }, + "index_catalan": { + "tokenizer": "standard", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer", + "autocomplete_filter" + ] + }, + "search_catalan": { + "tokenizer": "standard", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer" + ] + }, + "index_basque": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer", + "autocomplete_filter" + ] + }, + "search_basque": { + "tokenizer": "standard", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer" + ] + }, + "french_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "french_elision", + "french_stop", + "icu_folding", + "lowercase", + "french_stemmer" + ] + }, + "german_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "german_stop", + "german_stemmer", + "lowercase", + "icu_folding" + ] + }, + "english_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "english_possessive_stemmer", + "english_stop", + "lowercase", + "icu_folding" + ] + }, + "italian_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "italian_elision", + "italian_stop", + "lowercase", + "icu_folding", + "italian_stemmer" + ] + }, + "spanish_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "spanish_stop", + "spanish_stemmer", + "icu_folding" + ] + }, + "catalan_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "catalan_elision", + "lowercase", + "catalan_stop", + "catalan_stemmer", + "icu_folding" + ] + }, + "basque_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "basque_stop", + "basque_stemmer", + "icu_folding" + ] + }, + "index_slovene": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "autocomplete_filter", + "lowercase" + ] + }, + "search_slovene": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase" + ] + }, + "slovene_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "icu_folding" + ] + }, + "index_chinois": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "autocomplete_filter" + ] + }, + "search_chinois": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase" + ] + }, + "chinois_heavy": { + "tokenizer": "icu_tokenizer", + "filter": [ + "lowercase", + "icu_folding" + ] + } + } + } +} \ No newline at end of file diff --git a/scripts/esjson6-7/u.json b/scripts/esjson6-7/u.json new file mode 100644 index 000000000..8f23d485f --- /dev/null +++ b/scripts/esjson6-7/u.json @@ -0,0 +1,275 @@ +{ + "properties": { + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "quality": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/w.json b/scripts/esjson6-7/w.json new file mode 100644 index 000000000..4d3e561e0 --- /dev/null +++ b/scripts/esjson6-7/w.json @@ -0,0 +1,386 @@ +{ + "properties": { + "access_time": { + "type": "integer" + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "best_periods": { + "type": "text", + "index": true + }, + "capacity": { + "type": "integer" + }, + "capacity_staffed": { + "type": "integer" + }, + "children_proof": { + "type": "text", + "index": true + }, + "climbing_indoor_types": { + "type": "text", + "index": true + }, + "climbing_outdoor_types": { + "type": "text", + "index": true + }, + "climbing_rating_max": { + "type": "integer" + }, + "climbing_rating_median": { + "type": "integer" + }, + "climbing_rating_min": { + "type": "integer" + }, + "climbing_styles": { + "type": "text", + "index": true + }, + "custodianship": { + "type": "text", + "index": true + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "equipment_ratings": { + "type": "integer" + }, + "exposition_rating": { + "type": "integer" + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "height_max": { + "type": "integer" + }, + "height_median": { + "type": "integer" + }, + "height_min": { + "type": "integer" + }, + "id": { + "type": "long" + }, + "length": { + "type": "integer" + }, + "lift_access": { + "type": "boolean" + }, + "orientations": { + "type": "text", + "index": true + }, + "paragliding_rating": { + "type": "integer" + }, + "product_types": { + "type": "text", + "index": true + }, + "prominence": { + "type": "integer" + }, + "public_transportation_rating": { + "type": "text", + "index": true + }, + "public_transportation_types": { + "type": "text", + "index": true + }, + "quality": { + "type": "integer" + }, + "rain_proof": { + "type": "text", + "index": true + }, + "rock_types": { + "type": "text", + "index": true + }, + "routes_quantity": { + "type": "integer" + }, + "snow_clearance_rating": { + "type": "text", + "index": true + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "waypoint_type": { + "type": "text", + "index": true + }, + "weather_station_types": { + "type": "text", + "index": true + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file diff --git a/scripts/esjson6-7/x.json b/scripts/esjson6-7/x.json new file mode 100644 index 000000000..9071b569b --- /dev/null +++ b/scripts/esjson6-7/x.json @@ -0,0 +1,305 @@ +{ + "properties": { + "activities": { + "type": "text", + "index": true + }, + "areas": { + "type": "long" + }, + "available_locales": { + "type": "text", + "index": true + }, + "avalanche_level": { + "type": "integer" + }, + "avalanche_slope": { + "type": "integer" + }, + "date": { + "type": "date", + "format": "strict_date_optional_time||epoch_millis" + }, + "description_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "description_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "description_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "description_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "description_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "description_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "description_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "doc_type": { + "type": "text", + "index": false + }, + "elevation": { + "type": "integer" + }, + "event_type": { + "type": "text", + "index": true + }, + "geom": { + "type": "geo_point", + "ignore_malformed": true + }, + "id": { + "type": "long" + }, + "nb_impacted": { + "type": "integer" + }, + "nb_participants": { + "type": "integer" + }, + "quality": { + "type": "integer" + }, + "severity": { + "type": "integer" + }, + "summary_ca": { + "type": "text", + "analyzer": "index_catalan", + "search_analyzer": "search_catalan" + }, + "summary_de": { + "type": "text", + "analyzer": "index_german", + "search_analyzer": "search_german" + }, + "summary_en": { + "type": "text", + "analyzer": "index_english", + "search_analyzer": "search_english" + }, + "summary_es": { + "type": "text", + "analyzer": "index_spanish", + "search_analyzer": "search_spanish" + }, + "summary_eu": { + "type": "text", + "analyzer": "index_basque", + "search_analyzer": "search_basque" + }, + "summary_fr": { + "type": "text", + "analyzer": "index_french", + "search_analyzer": "search_french" + }, + "summary_it": { + "type": "text", + "analyzer": "index_italian", + "search_analyzer": "search_italian" + }, + "title_ca": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "catalan_heavy" + } + } + }, + "title_de": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "german_heavy" + } + } + }, + "title_en": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "english_heavy" + } + } + }, + "title_es": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "spanish_heavy" + } + } + }, + "title_eu": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "basque_heavy" + } + } + }, + "title_fr": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "french_heavy" + } + } + }, + "title_it": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "italian_heavy" + } + } + }, + "title_sl": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "slovene_heavy" + } + } + }, + "title_zh": { + "type": "text", + "index": false, + "fields": { + "ngram": { + "type": "text", + "analyzer": "index_ngram", + "search_analyzer": "search_ngram" + }, + "raw": { + "type": "text", + "analyzer": "index_raw", + "search_analyzer": "search_raw" + }, + "contentheavy": { + "type": "text", + "analyzer": "chinois_heavy" + } + } + } + } + } \ No newline at end of file From c2a8033c6b34e8a253b52a862c693dc78166c203 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 13 Jun 2024 16:19:55 +0200 Subject: [PATCH 23/53] Upgrade Elasticsearch from 6.8 to 7.X : - update mapping --- scripts/esUpdateMapSettings67.sh | 40 ++++++++++++++++---------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/scripts/esUpdateMapSettings67.sh b/scripts/esUpdateMapSettings67.sh index 19c9927f4..3bb799b54 100755 --- a/scripts/esUpdateMapSettings67.sh +++ b/scripts/esUpdateMapSettings67.sh @@ -16,16 +16,16 @@ curl -X DELETE "http://localhost:9200/c2corg_*" curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/cluster.json http://localhost:9200/_template/index_defaults #créer les index -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_a -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_b -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_c -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_i -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_m -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_o -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_r -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_u -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_w -curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_x +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_a?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_b?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_c?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_i?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_m?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_o?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_r?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_u?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_w?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_x?include_type_name=true sleep 2 #2. closing the index to upgrade settings @@ -70,23 +70,23 @@ sleep 2 #5. Docs Mappings update echo 'updating doc type mappings' echo 'doc type : a' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://localhost:9200/c2corg_a/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://localhost:9200/c2corg_a/_mapping/a echo -e '\n doc type : b' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://localhost:9200/c2corg_b/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://localhost:9200/c2corg_b/_mapping/b echo -e '\n doc type : c' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://localhost:9200/c2corg_c/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://localhost:9200/c2corg_c/_mapping/c echo -e '\n doc type : i' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://localhost:9200/c2corg_i/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://localhost:9200/c2corg_i/_mapping/i echo -e '\n doc type : m' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://localhost:9200/c2corg_m/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://localhost:9200/c2corg_m/_mapping/m echo -e '\n doc type : o' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://localhost:9200/c2corg_o/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://localhost:9200/c2corg_o/_mapping/o echo -e '\n doc type : r' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://localhost:9200/c2corg_r/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://localhost:9200/c2corg_r/_mapping/r echo -e '\n doc type : u' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://localhost:9200/c2corg_u/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://localhost:9200/c2corg_u/_mapping/u echo -e '\n doc type : w' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://localhost:9200/c2corg_w/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://localhost:9200/c2corg_w/_mapping/w echo -e '\n doc type : x' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://localhost:9200/c2corg_x/_mapping +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://localhost:9200/c2corg_x/_mapping/x sleep 2 From 7711b0ed839e3cf6cdd5878b7f14712cb8dd4845 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Fri, 14 Jun 2024 10:06:49 +0200 Subject: [PATCH 24/53] Upgrade Elasticsearch from 6.8 to 7.X : - update mapping --- c2corg_api/scripts/initializees.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 713b23fea..a1404b07b 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -115,7 +115,7 @@ def create_indice(cible, indice_name): header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') - c.setopt(c.URL, 'http://' + cible + '/' + indice_name) + c.setopt(c.URL, 'http://' + cible + '/' + indice_name + "?include_type_name=true") c.setopt(c.WRITEDATA, buffer) c.perform() c.close() @@ -168,7 +168,7 @@ def indice_mapping_update(cible, indice_name, mapping_type): buffer = BytesIO() c = pycurl.Curl() - c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping') + c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping'+'/'+mapping_type) header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') From dd72c933f5b1ca9dc2fc11d502cd16b465dd5a1b Mon Sep 17 00:00:00 2001 From: jphilippe Date: Fri, 14 Jun 2024 11:22:11 +0200 Subject: [PATCH 25/53] Upgrade Elasticsearch from 6.8 to 7.X : - update mapping --- scripts/esUpdateMapSettings67.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/scripts/esUpdateMapSettings67.sh b/scripts/esUpdateMapSettings67.sh index 3bb799b54..6b05e5f13 100755 --- a/scripts/esUpdateMapSettings67.sh +++ b/scripts/esUpdateMapSettings67.sh @@ -70,23 +70,23 @@ sleep 2 #5. Docs Mappings update echo 'updating doc type mappings' echo 'doc type : a' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://localhost:9200/c2corg_a/_mapping/a +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://localhost:9200/c2corg_a/_mapping/a?include_type_name=true echo -e '\n doc type : b' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://localhost:9200/c2corg_b/_mapping/b +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://localhost:9200/c2corg_b/_mapping/b?include_type_name=true echo -e '\n doc type : c' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://localhost:9200/c2corg_c/_mapping/c +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://localhost:9200/c2corg_c/_mapping/c?include_type_name=true echo -e '\n doc type : i' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://localhost:9200/c2corg_i/_mapping/i +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://localhost:9200/c2corg_i/_mapping/i?include_type_name=true echo -e '\n doc type : m' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://localhost:9200/c2corg_m/_mapping/m +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://localhost:9200/c2corg_m/_mapping/m?include_type_name=true echo -e '\n doc type : o' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://localhost:9200/c2corg_o/_mapping/o +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://localhost:9200/c2corg_o/_mapping/o?include_type_name=true echo -e '\n doc type : r' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://localhost:9200/c2corg_r/_mapping/r +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://localhost:9200/c2corg_r/_mapping/r?include_type_name=true echo -e '\n doc type : u' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://localhost:9200/c2corg_u/_mapping/u +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://localhost:9200/c2corg_u/_mapping/u?include_type_name=true echo -e '\n doc type : w' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://localhost:9200/c2corg_w/_mapping/w +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://localhost:9200/c2corg_w/_mapping/w?include_type_name=true echo -e '\n doc type : x' -curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://localhost:9200/c2corg_x/_mapping/x +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://localhost:9200/c2corg_x/_mapping/x?include_type_name=true sleep 2 From dd5d94b4d236723c10082c77f1e72461cabe7be7 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Mon, 17 Jun 2024 18:37:28 +0200 Subject: [PATCH 26/53] Upgrade Elasticsearch from 6.8 to 7.0 : - update mapping --- .github/workflows/ci.yml | 2 +- c2corg_api/scripts/es/fill_index.py | 1 - c2corg_api/scripts/initializees.py | 26 +++++++------- c2corg_api/search/advanced_search.py | 12 +++++-- c2corg_api/search/mapping.py | 9 +++-- c2corg_api/tests/views/__init__.py | 6 ++++ c2corg_api/tests/views/test_langs.py | 2 ++ c2corg_api/views/area.py | 1 + c2corg_api/views/document.py | 1 - c2corg_api/views/document_listings.py | 6 ++++ dataMigration/v7/logstash.conf | 50 +++++++++++++++++++++++++++ docker-compose.yml | 12 +++++++ elasticsearch7.Dockerfile | 2 +- logstash7.Dockerfile | 2 +- requirements.txt | 4 +-- scripts/esjson6-7/a.json | 3 +- scripts/esjson6-7/b.json | 3 +- scripts/esjson6-7/c.json | 3 +- scripts/esjson6-7/i.json | 3 +- scripts/esjson6-7/m.json | 3 +- scripts/esjson6-7/o.json | 3 +- scripts/esjson6-7/r.json | 3 +- scripts/esjson6-7/u.json | 3 +- scripts/esjson6-7/w.json | 3 +- scripts/esjson6-7/x.json | 3 +- 25 files changed, 129 insertions(+), 37 deletions(-) create mode 100644 dataMigration/v7/logstash.conf diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4014fb679..9294254b4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,7 +79,7 @@ jobs: - name: Run (old) elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.23.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.0.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs diff --git a/c2corg_api/scripts/es/fill_index.py b/c2corg_api/scripts/es/fill_index.py index cd41f27af..840c3cf06 100644 --- a/c2corg_api/scripts/es/fill_index.py +++ b/c2corg_api/scripts/es/fill_index.py @@ -81,7 +81,6 @@ def progress(count, total_count): for doc in sync.get_documents(session, doc_type, batch_size, ignore_redirects=True): - logging.warning("JPR fill_index doc_type : %s", doc_type) batch.add(to_search_document(doc, index_name[:-1] + doc_type)) count += 1 diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index a1404b07b..e3b82eec5 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -54,13 +54,13 @@ def setup_es(): client = elasticsearch_config['client'] index_name = elasticsearch_config['index'] cible = elasticsearch_config['host']+':'+str(elasticsearch_config['port']) - print('cible es: %s', cible) + #print('cible es: %s', cible) info = client.info() print('ElasticSearch version: {0}'.format(info['version']['number'])) for index_suffix in index_suffix_list: - print("suffix : ", index_suffix) + #print("suffix : ", index_suffix) if client.indices.exists(index_name[:-2]+index_suffix): print('Index "{0}" already exists. deleting it {0}... ' .format(index_name[:-2]+index_suffix)) @@ -71,14 +71,14 @@ def setup_es(): delete_indice(cible, index_name[:-2]+index_suffix) for index_suffix in index_suffix_list: - print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) + #print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) create_indice(cible, index_name[:-2]+index_suffix) - print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) + #print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) indice_settings_update(cible, index_name[:-2]+index_suffix) - print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) + #print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) indice_mapping_update(cible, index_name[:-2]+index_suffix, index_suffix[1]) - print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) + #print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) print('all indexes are created') @@ -106,7 +106,7 @@ def delete_indice(cible, indice_name): c.close() body = buffer.getvalue() - print(body.decode('iso-8859-1')) + #print(body.decode('iso-8859-1')) def create_indice(cible, indice_name): @@ -121,7 +121,7 @@ def create_indice(cible, indice_name): c.close() body = buffer.getvalue() - print(body.decode('iso-8859-1')) + #print(body.decode('iso-8859-1')) def indice_settings_update(cible, indice_name): @@ -134,7 +134,7 @@ def indice_settings_update(cible, indice_name): c.setopt(c.WRITEDATA, buffer) c.perform() body = buffer.getvalue() - print(body.decode('iso-8859-1')) + #print(body.decode('iso-8859-1')) file = "./scripts/esjson6-7/settings.json" f = open(file) @@ -145,7 +145,7 @@ def indice_settings_update(cible, indice_name): c.setopt(c.WRITEDATA, buffer) c.perform() body = buffer.getvalue() - print(body.decode('iso-8859-1')) + #print(body.decode('iso-8859-1')) c.close() c = pycurl.Curl() @@ -156,7 +156,7 @@ def indice_settings_update(cible, indice_name): c.setopt(c.WRITEDATA, buffer) c.perform() body = buffer.getvalue() - print(body.decode('iso-8859-1')) + #print(body.decode('iso-8859-1')) c.close() @@ -168,7 +168,7 @@ def indice_mapping_update(cible, indice_name, mapping_type): buffer = BytesIO() c = pycurl.Curl() - c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping'+'/'+mapping_type) + c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping'+'/'+mapping_type+'?include_type_name=true') header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') @@ -178,4 +178,4 @@ def indice_mapping_update(cible, indice_name, mapping_type): c.close() body = buffer.getvalue() - print(body.decode('iso-8859-1')) + #print(body.decode('iso-8859-1')) diff --git a/c2corg_api/search/advanced_search.py b/c2corg_api/search/advanced_search.py index 3cf2dc450..7a4403d02 100644 --- a/c2corg_api/search/advanced_search.py +++ b/c2corg_api/search/advanced_search.py @@ -1,3 +1,7 @@ +import logging +import sys + + from c2corg_api.search.mapping_types import meta_param_keys from c2corg_api.search.search_filters import build_query @@ -21,8 +25,12 @@ def search(url_params, meta_params, doc_type): # only request the document ids from ES response = query.execute() - document_ids = [int(doc.meta.id) for doc in response] - total = response.hits.total + + for doc in response.hits: + logging.warning('doc.id : %s', doc.meta.id) + + document_ids = [int(doc.meta.id) for doc in response.hits] + total = response.hits.total.value return document_ids, total diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index b85ffa2f1..66511d010 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -8,7 +8,7 @@ from c2corg_api.models.common.attributes import default_langs from c2corg_api.models.common.sortable_search_attributes import \ sortable_quality_types -from elasticsearch_dsl import MetaField, Long, GeoPoint, Text +from elasticsearch_dsl import MetaField, Long, GeoPoint, Text, DocType class BaseMeta: @@ -44,7 +44,7 @@ def default_title_field(lang: None): }) -class SearchDocument(): +class SearchDocument(DocType): """The base ElasticSearch mapping for documents. Each document type has its own specific mapping. @@ -140,7 +140,7 @@ def to_search_document(document, index, include_areas=True): search_document = { '_index': index, '_id': document.document_id, - #'_type': "_doc", # document.type, + '_type': "_doc", # document.type, 'id': document.document_id } @@ -195,8 +195,7 @@ def copy_fields(search_document, document, fields): @staticmethod def copy_enum_range_fields( search_document, document, fields, search_model): - - search_fields = search_model.c2corg_doc_type.mapping + search_fields = search_model._doc_type.mapping logging.warning("search_fields : %s", search_fields) for field in fields: diff --git a/c2corg_api/tests/views/__init__.py b/c2corg_api/tests/views/__init__.py index d2b47c119..50aa18e46 100644 --- a/c2corg_api/tests/views/__init__.py +++ b/c2corg_api/tests/views/__init__.py @@ -1,7 +1,10 @@ import json +import logging import urllib.request import urllib.parse import urllib.error +from inspect import getmembers +from pprint import pprint from c2corg_api.caching import cache_document_detail from c2corg_api.models.cache_version import CacheVersion, get_cache_key @@ -178,12 +181,15 @@ def get_collection(self, params=None, user=None): def get_collection_search(self, params=None, user=None): prefix = self._prefix + if params: prefix += "?" + urllib.parse.urlencode(params) headers = {} if not user else \ self.add_authorization_header(username=user) + #pprint(getmembers(prefix)) + response = self.app.get(prefix, headers=headers, status=200) self.assertEqual(response.content_type, 'application/json') diff --git a/c2corg_api/tests/views/test_langs.py b/c2corg_api/tests/views/test_langs.py index bf4ed4ef9..b24502f0a 100644 --- a/c2corg_api/tests/views/test_langs.py +++ b/c2corg_api/tests/views/test_langs.py @@ -1,3 +1,5 @@ +import logging +import sys from datetime import date, datetime from c2corg_api.models.feed import DocumentChange diff --git a/c2corg_api/views/area.py b/c2corg_api/views/area.py index 50ea3d023..9b9312c52 100644 --- a/c2corg_api/views/area.py +++ b/c2corg_api/views/area.py @@ -1,4 +1,5 @@ import functools +import logging from c2corg_api.models.area import schema_area, Area, schema_update_area, \ AREA_TYPE, schema_create_area, ArchiveArea diff --git a/c2corg_api/views/document.py b/c2corg_api/views/document.py index 09fc435ec..f441c033d 100644 --- a/c2corg_api/views/document.py +++ b/c2corg_api/views/document.py @@ -73,7 +73,6 @@ def _collection_get(self, doc_type, documents_config): 'limit': min(validated.get('limit', LIMIT_DEFAULT), LIMIT_MAX), 'lang': validated.get('lang') } - if meta_params['offset'] + meta_params['limit'] > ES_MAX_RESULT_WINDOW: # ES does not process requests where offset + limit is greater # than 10000, see: diff --git a/c2corg_api/views/document_listings.py b/c2corg_api/views/document_listings.py index d5700aca7..86dd69671 100644 --- a/c2corg_api/views/document_listings.py +++ b/c2corg_api/views/document_listings.py @@ -1,3 +1,6 @@ +import json +import logging + from c2corg_api.caching import cache_document_listing from c2corg_api.models import DBSession from c2corg_api.models.area import schema_listing_area @@ -46,6 +49,9 @@ def get_documents(documents_config, meta_params, search_documents): documents_config.clazz.document_id.desc()) document_ids, total = search_documents(base_query, base_total_query) + if total is not None and type(total) is not int : + total = int(total['value']) + cache_keys = get_cache_keys( document_ids, lang, documents_config.document_type) diff --git a/dataMigration/v7/logstash.conf b/dataMigration/v7/logstash.conf new file mode 100644 index 000000000..83c221913 --- /dev/null +++ b/dataMigration/v7/logstash.conf @@ -0,0 +1,50 @@ +input +{ + elasticsearch + { + hosts => ["192.168.1.120:9203"] + index => "c2corg_*" + docinfo => true + size => 1000 + scroll => "5m" + docinfo_target => "[@metadata]" + } +} + +filter { + if (![geom]) { + mutate { + add_field => { "geom" => [0,0] } + } + } + + #mutate { + # add_field => { "c2corg_doc_type" => "%{[@metadata][_type]}" } + #} + + mutate { + remove_field => [ "_type", "@timestamp", "@version", "doc_id", "doc_type" ] + } + +} + +output +{ + elasticsearch + { + hosts => ["elasticsearch:9200"] + index => "c2corg_%{[c2corg_doc_type]}" + #document_type => "_doc" + document_id => "%{[@metadata][_id]}" + } + + stdout { + codec => "dots" + } + +} + +#stdout { codec => "dots"} +#stdout { codec => rubydebug } +#stdout { codec => "json" } +#docker-compose run logstash6 /usr/share/logstash/bin/logstash -f /root/logstash.conf \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index daab3cf1c..da257c801 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -121,6 +121,18 @@ services: # volumes: # - ./dataMigration/v6:/usr/share/logstash/pipeline + logstash7: + platform: linux/amd64 + container_name: logstash7 + build: + context: . + dockerfile: logstash7.Dockerfile + network: host + ports: + - 5044:5044 + volumes: + - ./dataMigration/v7:/usr/share/logstash/pipeline + redis: image: 'docker.io/redis:3.2' ports: diff --git a/elasticsearch7.Dockerfile b/elasticsearch7.Dockerfile index 8d70023ae..a85280adb 100644 --- a/elasticsearch7.Dockerfile +++ b/elasticsearch7.Dockerfile @@ -1,2 +1,2 @@ -FROM elasticsearch:7.17.21 +FROM elasticsearch:7.0.0 RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu diff --git a/logstash7.Dockerfile b/logstash7.Dockerfile index 8ac2043d8..f2f4dd926 100644 --- a/logstash7.Dockerfile +++ b/logstash7.Dockerfile @@ -1,2 +1,2 @@ -FROM logstash:7.17.21 +FROM logstash:7.0.0 RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/requirements.txt b/requirements.txt index 1786c2fcf..e1f5f7402 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,8 +4,8 @@ bcrypt==4.1.2 bleach[css]==6.1.0 colander==2.0 dogpile.cache==1.3.2 -elasticsearch==7.17.9 #elasticsearch==2.4.1 -elasticsearch-dsl==7.4.1 #elasticsearch_dsl==2.2.0 +elasticsearch==7.0.0 #elasticsearch==2.4.1 +elasticsearch-dsl==7.0.0 #elasticsearch_dsl==2.2.0 geoalchemy2==0.4.2 geojson==3.1.0 geomet==1.1.0 diff --git a/scripts/esjson6-7/a.json b/scripts/esjson6-7/a.json index 1b3a46031..8af0569c0 100644 --- a/scripts/esjson6-7/a.json +++ b/scripts/esjson6-7/a.json @@ -55,7 +55,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "quality": { "type": "integer" diff --git a/scripts/esjson6-7/b.json b/scripts/esjson6-7/b.json index 6fa5d297e..85f82f8db 100644 --- a/scripts/esjson6-7/b.json +++ b/scripts/esjson6-7/b.json @@ -59,7 +59,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "quality": { "type": "integer" diff --git a/scripts/esjson6-7/c.json b/scripts/esjson6-7/c.json index a5d6ebf96..d1fcbab88 100644 --- a/scripts/esjson6-7/c.json +++ b/scripts/esjson6-7/c.json @@ -63,7 +63,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "quality": { "type": "integer" diff --git a/scripts/esjson6-7/i.json b/scripts/esjson6-7/i.json index a58b744b1..ec33e1415 100644 --- a/scripts/esjson6-7/i.json +++ b/scripts/esjson6-7/i.json @@ -66,7 +66,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "image_type": { "type": "text", diff --git a/scripts/esjson6-7/m.json b/scripts/esjson6-7/m.json index 8f23d485f..e776af0da 100644 --- a/scripts/esjson6-7/m.json +++ b/scripts/esjson6-7/m.json @@ -51,7 +51,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "quality": { "type": "integer" diff --git a/scripts/esjson6-7/o.json b/scripts/esjson6-7/o.json index b4fa1be8c..f859309e5 100644 --- a/scripts/esjson6-7/o.json +++ b/scripts/esjson6-7/o.json @@ -109,7 +109,8 @@ "type": "integer" }, "id": { - "type": "long" + "type": "long", + "index": true }, "labande_global_rating": { "type": "integer" diff --git a/scripts/esjson6-7/r.json b/scripts/esjson6-7/r.json index 16ebe15ca..738fc0c95 100644 --- a/scripts/esjson6-7/r.json +++ b/scripts/esjson6-7/r.json @@ -115,7 +115,8 @@ "type": "integer" }, "id": { - "type": "long" + "type": "long", + "index": true }, "labande_global_rating": { "type": "integer" diff --git a/scripts/esjson6-7/u.json b/scripts/esjson6-7/u.json index 8f23d485f..e776af0da 100644 --- a/scripts/esjson6-7/u.json +++ b/scripts/esjson6-7/u.json @@ -51,7 +51,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "quality": { "type": "integer" diff --git a/scripts/esjson6-7/w.json b/scripts/esjson6-7/w.json index 4d3e561e0..640450b38 100644 --- a/scripts/esjson6-7/w.json +++ b/scripts/esjson6-7/w.json @@ -111,7 +111,8 @@ "type": "integer" }, "id": { - "type": "long" + "type": "long", + "index": true }, "length": { "type": "integer" diff --git a/scripts/esjson6-7/x.json b/scripts/esjson6-7/x.json index 9071b569b..7f7ffc5d8 100644 --- a/scripts/esjson6-7/x.json +++ b/scripts/esjson6-7/x.json @@ -72,7 +72,8 @@ "ignore_malformed": true }, "id": { - "type": "long" + "type": "long", + "index": true }, "nb_impacted": { "type": "integer" From cc7e96fb531fb8758960618ef0b0db1644ed6b53 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 18 Jun 2024 12:23:45 +0200 Subject: [PATCH 27/53] Upgrade Elasticsearch from 6.8 to 7.0 : - update script for data migration --- logstash7.Dockerfile | 2 +- scripts/README.md | 27 +++++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/logstash7.Dockerfile b/logstash7.Dockerfile index f2f4dd926..a91bc6967 100644 --- a/logstash7.Dockerfile +++ b/logstash7.Dockerfile @@ -1,2 +1,2 @@ -FROM logstash:7.0.0 +FROM logstash:7.0.1 RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/scripts/README.md b/scripts/README.md index 73b239892..5b3e46960 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,3 +1,30 @@ +# ElasticSearch Upgrade from v6.5 to V7.0 for docker env. like dev one. + +The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. +The ElasticSearch 6.8 is still build and run but would not be the main ES for the service. It could be decommissioned after migration. +docker-compose file contains as well : + +- api building with integration of the requirements.txt for python pip installation (component updates) +- ES 6.8 listening on port 9201 instead of 9200 : > elasticsearch68 +- redis no change +- postgresql no change +- ES 7.0 listening on port 9200 and 9300 (elasticsearch7.Dockerfile) > elasticsearch +- Logstash 7.0 for data migration purpose. (logstash7.Dockerfile) + +## migration steps + + 1. **build the environment :** ` docker-compose build ` + 2. **Launch the environment :** ` docker-compose up -d ` + 3. **Create the index and its mapping :** `./scripts/esUpdateMappings67.sh` + 4. **Data Migration :** `docker-compose start logstash7` It will stop by itself when migration would be achieved + 5. **postgresql shema up to date:** `docker-compose exec api .build/venv/bin/alembic upgrade head` + +## post migration + +if all the process has been passed without issues. you can comment or remove in the docker-compose.yml services : +- logstash +- elasticsearch68 + # ElasticSearch Upgrade from v5.6 to V6.8 for docker env. like dev one. The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. From 674238c0280d13f86ebb0570895268c8e12886f5 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 18 Jun 2024 12:38:12 +0200 Subject: [PATCH 28/53] Upgrade Elasticsearch from 6.8 to 7.0 : - lint fixes --- c2corg_api/scripts/initializees.py | 42 ++++++++++--------- c2corg_api/search/__init__.py | 2 +- c2corg_api/search/advanced_search.py | 3 -- c2corg_api/search/mapping.py | 2 - c2corg_api/search/mappings/article_mapping.py | 1 - c2corg_api/tests/views/__init__.py | 6 --- c2corg_api/tests/views/test_langs.py | 3 -- c2corg_api/views/area.py | 2 - c2corg_api/views/document_listings.py | 6 +-- 9 files changed, 25 insertions(+), 42 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index e3b82eec5..30e5c2359 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -54,13 +54,13 @@ def setup_es(): client = elasticsearch_config['client'] index_name = elasticsearch_config['index'] cible = elasticsearch_config['host']+':'+str(elasticsearch_config['port']) - #print('cible es: %s', cible) + # print('cible es: %s', cible) info = client.info() print('ElasticSearch version: {0}'.format(info['version']['number'])) for index_suffix in index_suffix_list: - #print("suffix : ", index_suffix) + # print("suffix : ", index_suffix) if client.indices.exists(index_name[:-2]+index_suffix): print('Index "{0}" already exists. deleting it {0}... ' .format(index_name[:-2]+index_suffix)) @@ -71,14 +71,14 @@ def setup_es(): delete_indice(cible, index_name[:-2]+index_suffix) for index_suffix in index_suffix_list: - #print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) + # print('Index "{0}" to create'.format(index_name[:-2]+index_suffix)) create_indice(cible, index_name[:-2]+index_suffix) - #print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) + # print('Index "{0}" created'.format(index_name[:-2]+index_suffix)) indice_settings_update(cible, index_name[:-2]+index_suffix) - #print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) + # print('Index "{0}" settings'.format(index_name[:-2] + index_suffix)) indice_mapping_update(cible, index_name[:-2]+index_suffix, index_suffix[1]) - #print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) + # print('Index "{0}" mappings'.format(index_name[:-2] + index_suffix)) print('all indexes are created') @@ -105,8 +105,8 @@ def delete_indice(cible, indice_name): c.perform() c.close() - body = buffer.getvalue() - #print(body.decode('iso-8859-1')) + # body = buffer.getvalue() + # print(body.decode('iso-8859-1')) def create_indice(cible, indice_name): @@ -115,13 +115,14 @@ def create_indice(cible, indice_name): header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') - c.setopt(c.URL, 'http://' + cible + '/' + indice_name + "?include_type_name=true") + c.setopt(c.URL, 'http://' + cible + '/' + indice_name + + "?include_type_name=true") c.setopt(c.WRITEDATA, buffer) c.perform() c.close() - body = buffer.getvalue() - #print(body.decode('iso-8859-1')) + # body = buffer.getvalue() + # print(body.decode('iso-8859-1')) def indice_settings_update(cible, indice_name): @@ -133,8 +134,8 @@ def indice_settings_update(cible, indice_name): c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_close') c.setopt(c.WRITEDATA, buffer) c.perform() - body = buffer.getvalue() - #print(body.decode('iso-8859-1')) + # body = buffer.getvalue() + # print(body.decode('iso-8859-1')) file = "./scripts/esjson6-7/settings.json" f = open(file) @@ -144,8 +145,8 @@ def indice_settings_update(cible, indice_name): c.setopt(c.POSTFIELDS, post_data) c.setopt(c.WRITEDATA, buffer) c.perform() - body = buffer.getvalue() - #print(body.decode('iso-8859-1')) + # body = buffer.getvalue() + # print(body.decode('iso-8859-1')) c.close() c = pycurl.Curl() @@ -155,8 +156,8 @@ def indice_settings_update(cible, indice_name): c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_open') c.setopt(c.WRITEDATA, buffer) c.perform() - body = buffer.getvalue() - #print(body.decode('iso-8859-1')) + # body = buffer.getvalue() + # print(body.decode('iso-8859-1')) c.close() @@ -168,7 +169,8 @@ def indice_mapping_update(cible, indice_name, mapping_type): buffer = BytesIO() c = pycurl.Curl() - c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping'+'/'+mapping_type+'?include_type_name=true') + c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping' + + '/'+mapping_type+'?include_type_name=true') header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') @@ -177,5 +179,5 @@ def indice_mapping_update(cible, indice_name, mapping_type): c.perform() c.close() - body = buffer.getvalue() - #print(body.decode('iso-8859-1')) + # body = buffer.getvalue() + # print(body.decode('iso-8859-1')) diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index 38d8c76f8..0aa9308d7 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -79,7 +79,7 @@ def create_search(document_type=None): return Search( using=elasticsearch_config['client'], index=(elasticsearch_config['index']), - #doc_type="_doc" # search_documents[document_type] + # doc_type="_doc" # search_documents[document_type] ) diff --git a/c2corg_api/search/advanced_search.py b/c2corg_api/search/advanced_search.py index 7a4403d02..ac613ff8b 100644 --- a/c2corg_api/search/advanced_search.py +++ b/c2corg_api/search/advanced_search.py @@ -1,7 +1,4 @@ import logging -import sys - - from c2corg_api.search.mapping_types import meta_param_keys from c2corg_api.search.search_filters import build_query diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index 66511d010..1e4ae4912 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -1,7 +1,5 @@ import json import logging -import pprint - from c2corg_api.models.document import Document from c2corg_api.search.mapping_types import Enum, QEnumArray, QLong, \ QEnumRange diff --git a/c2corg_api/search/mappings/article_mapping.py b/c2corg_api/search/mappings/article_mapping.py index 2cf7033f8..34aa537f2 100644 --- a/c2corg_api/search/mappings/article_mapping.py +++ b/c2corg_api/search/mappings/article_mapping.py @@ -16,7 +16,6 @@ class Meta(BaseMeta): FIELDS = ['activities', 'article_type'] - @staticmethod def to_search_document(document, index): search_document = SearchDocument.to_search_document(document, index) diff --git a/c2corg_api/tests/views/__init__.py b/c2corg_api/tests/views/__init__.py index 50aa18e46..0f49af5b3 100644 --- a/c2corg_api/tests/views/__init__.py +++ b/c2corg_api/tests/views/__init__.py @@ -1,11 +1,7 @@ import json -import logging import urllib.request import urllib.parse import urllib.error -from inspect import getmembers -from pprint import pprint - from c2corg_api.caching import cache_document_detail from c2corg_api.models.cache_version import CacheVersion, get_cache_key from c2corg_api.models.feed import DocumentChange @@ -188,8 +184,6 @@ def get_collection_search(self, params=None, user=None): headers = {} if not user else \ self.add_authorization_header(username=user) - #pprint(getmembers(prefix)) - response = self.app.get(prefix, headers=headers, status=200) self.assertEqual(response.content_type, 'application/json') diff --git a/c2corg_api/tests/views/test_langs.py b/c2corg_api/tests/views/test_langs.py index b24502f0a..e90f50866 100644 --- a/c2corg_api/tests/views/test_langs.py +++ b/c2corg_api/tests/views/test_langs.py @@ -1,7 +1,4 @@ -import logging -import sys from datetime import date, datetime - from c2corg_api.models.feed import DocumentChange from c2corg_api.models.common.attributes import default_langs from c2corg_api.models.user import User diff --git a/c2corg_api/views/area.py b/c2corg_api/views/area.py index 9b9312c52..2d5bb2bc2 100644 --- a/c2corg_api/views/area.py +++ b/c2corg_api/views/area.py @@ -1,6 +1,4 @@ import functools -import logging - from c2corg_api.models.area import schema_area, Area, schema_update_area, \ AREA_TYPE, schema_create_area, ArchiveArea from c2corg_api.models.area_association import update_area diff --git a/c2corg_api/views/document_listings.py b/c2corg_api/views/document_listings.py index 86dd69671..2c8b63f98 100644 --- a/c2corg_api/views/document_listings.py +++ b/c2corg_api/views/document_listings.py @@ -1,6 +1,3 @@ -import json -import logging - from c2corg_api.caching import cache_document_listing from c2corg_api.models import DBSession from c2corg_api.models.area import schema_listing_area @@ -49,7 +46,8 @@ def get_documents(documents_config, meta_params, search_documents): documents_config.clazz.document_id.desc()) document_ids, total = search_documents(base_query, base_total_query) - if total is not None and type(total) is not int : + + if total is not None and type(total) is not int: total = int(total['value']) cache_keys = get_cache_keys( From 72d441eb95f166d504767a644ac69e7e2bdf1c50 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 18 Jun 2024 12:45:11 +0200 Subject: [PATCH 29/53] Upgrade Elasticsearch from 6.8 to 7.0 : - artifact fix for ES download --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9294254b4..5363f085a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,7 +79,7 @@ jobs: - name: Run (old) elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.0.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs From bf42a393f09cf8ba803e1b475ba1572b4a5d10e1 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Tue, 25 Jun 2024 18:25:44 +0200 Subject: [PATCH 30/53] Upgrade Elasticsearch from 6.8 to 7.17.22 : - artifact fix for ES download - mapping - class - requirements.txt - docker definition --- .github/workflows/ci.yml | 2 +- c2corg_api/search/mapping.py | 12 ++++++------ c2corg_api/search/mappings/area_mapping.py | 3 +++ c2corg_api/search/mappings/article_mapping.py | 3 +++ c2corg_api/search/mappings/book_mapping.py | 3 +++ c2corg_api/search/mappings/image_mapping.py | 3 +++ c2corg_api/search/mappings/outing_mapping.py | 3 +++ c2corg_api/search/mappings/route_mapping.py | 3 +++ c2corg_api/search/mappings/topo_map_mapping.py | 3 +++ c2corg_api/search/mappings/user_mapping.py | 3 +++ c2corg_api/search/mappings/waypoint_mapping.py | 3 +++ c2corg_api/search/mappings/xreport_mapping.py | 3 +++ docker-compose.yml | 2 +- elasticsearch7.Dockerfile | 2 +- logstash7.Dockerfile | 2 +- requirements.txt | 4 ++-- scripts/README.md | 6 +++--- 17 files changed, 45 insertions(+), 15 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5363f085a..22f765e70 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,7 +79,7 @@ jobs: - name: Run (old) elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.22-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs diff --git a/c2corg_api/search/mapping.py b/c2corg_api/search/mapping.py index 1e4ae4912..706c8be18 100644 --- a/c2corg_api/search/mapping.py +++ b/c2corg_api/search/mapping.py @@ -1,12 +1,12 @@ import json -import logging from c2corg_api.models.document import Document from c2corg_api.search.mapping_types import Enum, QEnumArray, QLong, \ QEnumRange from c2corg_api.models.common.attributes import default_langs from c2corg_api.models.common.sortable_search_attributes import \ sortable_quality_types -from elasticsearch_dsl import MetaField, Long, GeoPoint, Text, DocType +from elasticsearch_dsl import MetaField, Long, GeoPoint, Text +from elasticsearch_dsl import Document as esDocument class BaseMeta: @@ -42,7 +42,7 @@ def default_title_field(lang: None): }) -class SearchDocument(DocType): +class SearchDocument(esDocument): """The base ElasticSearch mapping for documents. Each document type has its own specific mapping. @@ -62,8 +62,10 @@ class Meta(BaseMeta): id = Long() c2corg_doc_type = Enum() + quality = QEnumRange( 'qa', model_field=Document.quality, enum_mapper=sortable_quality_types) + available_locales = QEnumArray('l', enum=default_langs) geom = GeoPoint() @@ -138,7 +140,7 @@ def to_search_document(document, index, include_areas=True): search_document = { '_index': index, '_id': document.document_id, - '_type': "_doc", # document.type, + # '_type': "_doc", # document.type, 'id': document.document_id } @@ -194,8 +196,6 @@ def copy_fields(search_document, document, fields): def copy_enum_range_fields( search_document, document, fields, search_model): search_fields = search_model._doc_type.mapping - logging.warning("search_fields : %s", search_fields) - for field in fields: search_field = search_fields[field] enum_mapper = search_field._enum_mapper diff --git a/c2corg_api/search/mappings/area_mapping.py b/c2corg_api/search/mappings/area_mapping.py index f76110224..42989a85d 100644 --- a/c2corg_api/search/mappings/area_mapping.py +++ b/c2corg_api/search/mappings/area_mapping.py @@ -7,6 +7,9 @@ class SearchArea(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = AREA_TYPE + class Index: + name = 'c2corg_a' + area_type = QEnum('atyp', model_field=Area.area_type) FIELDS = ['area_type'] diff --git a/c2corg_api/search/mappings/article_mapping.py b/c2corg_api/search/mappings/article_mapping.py index 34aa537f2..3bcd050b7 100644 --- a/c2corg_api/search/mappings/article_mapping.py +++ b/c2corg_api/search/mappings/article_mapping.py @@ -7,6 +7,9 @@ class SearchArticle(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = ARTICLE_TYPE + class Index: + name = 'c2corg_c' + activities = QEnumArray( 'act', model_field=Article.activities) article_categories = QEnumArray( diff --git a/c2corg_api/search/mappings/book_mapping.py b/c2corg_api/search/mappings/book_mapping.py index 58a91130f..e92505739 100644 --- a/c2corg_api/search/mappings/book_mapping.py +++ b/c2corg_api/search/mappings/book_mapping.py @@ -7,6 +7,9 @@ class SearchBook(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = BOOK_TYPE + class Index: + name = 'c2corg_b' + activities = QEnumArray( 'act', model_field=Book.activities) book_types = QEnumArray( diff --git a/c2corg_api/search/mappings/image_mapping.py b/c2corg_api/search/mappings/image_mapping.py index 13feac38d..d7c0f28ab 100644 --- a/c2corg_api/search/mappings/image_mapping.py +++ b/c2corg_api/search/mappings/image_mapping.py @@ -8,6 +8,9 @@ class SearchImage(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = IMAGE_TYPE + class Index: + name = 'c2corg_i' + activities = QEnumArray( 'act', model_field=Image.activities) categories = QEnumArray( diff --git a/c2corg_api/search/mappings/outing_mapping.py b/c2corg_api/search/mappings/outing_mapping.py index 5a5ffc631..b6b250606 100644 --- a/c2corg_api/search/mappings/outing_mapping.py +++ b/c2corg_api/search/mappings/outing_mapping.py @@ -19,6 +19,9 @@ class SearchOuting(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = OUTING_TYPE + class Index: + name = 'c2corg_o' + date_start = Date() date_end = Date() diff --git a/c2corg_api/search/mappings/route_mapping.py b/c2corg_api/search/mappings/route_mapping.py index 53fcd0d4f..277aac025 100644 --- a/c2corg_api/search/mappings/route_mapping.py +++ b/c2corg_api/search/mappings/route_mapping.py @@ -19,6 +19,9 @@ class SearchRoute(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = ROUTE_TYPE + class Index: + name = 'c2corg_r' + # array of waypoint ids waypoints = QLong('w', is_id=True) diff --git a/c2corg_api/search/mappings/topo_map_mapping.py b/c2corg_api/search/mappings/topo_map_mapping.py index 9beb28eb8..1efb05df8 100644 --- a/c2corg_api/search/mappings/topo_map_mapping.py +++ b/c2corg_api/search/mappings/topo_map_mapping.py @@ -7,6 +7,9 @@ class SearchTopoMap(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = MAP_TYPE + class Index: + name = 'c2corg_m' + FIELDS = [] @staticmethod diff --git a/c2corg_api/search/mappings/user_mapping.py b/c2corg_api/search/mappings/user_mapping.py index c5443bae3..d7740d5d3 100644 --- a/c2corg_api/search/mappings/user_mapping.py +++ b/c2corg_api/search/mappings/user_mapping.py @@ -7,6 +7,9 @@ class SearchUser(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = USERPROFILE_TYPE + class Index: + name = 'c2corg_u' + FIELDS = [] @staticmethod diff --git a/c2corg_api/search/mappings/waypoint_mapping.py b/c2corg_api/search/mappings/waypoint_mapping.py index 628ff770b..4716fadab 100644 --- a/c2corg_api/search/mappings/waypoint_mapping.py +++ b/c2corg_api/search/mappings/waypoint_mapping.py @@ -12,6 +12,9 @@ class SearchWaypoint(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = WAYPOINT_TYPE + class Index: + name = 'c2corg_w' + elevation = QInteger( 'walt', range=True) prominence = QInteger( diff --git a/c2corg_api/search/mappings/xreport_mapping.py b/c2corg_api/search/mappings/xreport_mapping.py index 0a68e4936..914435bd7 100644 --- a/c2corg_api/search/mappings/xreport_mapping.py +++ b/c2corg_api/search/mappings/xreport_mapping.py @@ -11,6 +11,9 @@ class SearchXreport(SearchDocument): class Meta(BaseMeta): c2corg_doc_type = XREPORT_TYPE + class Index: + name = 'c2corg_x' + event_activity = QEnumArray( 'act', model_field=Xreport.event_activity) date = QDate('xdate', 'date') diff --git a/docker-compose.yml b/docker-compose.yml index da257c801..7d0a9c816 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -68,7 +68,7 @@ services: # - 9302:9300 elasticsearch: - #platform: linux/amd64 + platform: linux/amd64 container_name: elasticsearch build: context: . diff --git a/elasticsearch7.Dockerfile b/elasticsearch7.Dockerfile index a85280adb..4dbaa80e0 100644 --- a/elasticsearch7.Dockerfile +++ b/elasticsearch7.Dockerfile @@ -1,2 +1,2 @@ -FROM elasticsearch:7.0.0 +FROM elasticsearch:7.17.22 RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu diff --git a/logstash7.Dockerfile b/logstash7.Dockerfile index a91bc6967..72e5908a6 100644 --- a/logstash7.Dockerfile +++ b/logstash7.Dockerfile @@ -1,2 +1,2 @@ -FROM logstash:7.0.1 +FROM logstash:7.17.22 RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/requirements.txt b/requirements.txt index e1f5f7402..9c46c2af2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,8 +4,8 @@ bcrypt==4.1.2 bleach[css]==6.1.0 colander==2.0 dogpile.cache==1.3.2 -elasticsearch==7.0.0 #elasticsearch==2.4.1 -elasticsearch-dsl==7.0.0 #elasticsearch_dsl==2.2.0 +elasticsearch==7.17.9 # elasticsearch==2.4.1 +elasticsearch-dsl==7.4.1 # elasticsearch_dsl==2.2.0 geoalchemy2==0.4.2 geojson==3.1.0 geomet==1.1.0 diff --git a/scripts/README.md b/scripts/README.md index 5b3e46960..aaf445607 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,4 +1,4 @@ -# ElasticSearch Upgrade from v6.5 to V7.0 for docker env. like dev one. +# ElasticSearch Upgrade from v6.8 to V7.22 for docker env. like dev one. The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. The ElasticSearch 6.8 is still build and run but would not be the main ES for the service. It could be decommissioned after migration. @@ -8,8 +8,8 @@ docker-compose file contains as well : - ES 6.8 listening on port 9201 instead of 9200 : > elasticsearch68 - redis no change - postgresql no change -- ES 7.0 listening on port 9200 and 9300 (elasticsearch7.Dockerfile) > elasticsearch -- Logstash 7.0 for data migration purpose. (logstash7.Dockerfile) +- ES 7.22 listening on port 9200 and 9300 (elasticsearch7.Dockerfile) > elasticsearch +- Logstash 7.22 for data migration purpose. (logstash7.Dockerfile) ## migration steps From 2498cd1b9383495c9dcdaf4e543424b429b6b5e3 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 26 Jun 2024 17:28:46 +0200 Subject: [PATCH 31/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - artifact fix for ES download - mapping - class - requirements.txt - docker definition --- c2corg_api/__init__.py | 1 - c2corg_api/scripts/initializees.py | 36 +++---- c2corg_api/search/__init__.py | 15 ++- c2corg_api/search/advanced_search.py | 1 + c2corg_api/search/search_filters.py | 4 +- c2corg_api/tests/scripts/es/test_syncer.py | 2 +- c2corg_api/tests/search/__init__.py | 2 +- .../tests/search/test_search_filters.py | 9 +- c2corg_api/views/health.py | 2 +- common.ini.in | 3 + config/docker-dev | 8 ++ docker-compose.yml | 56 ++++++++-- elasticsearch8.Dockerfile | 2 + logstash8.Dockerfile | 2 + requirements.txt | 4 +- scripts/esUpdateMapSettings71.sh | 92 ++++++++++++++++ scripts/esUpdateMapSettings78.sh | 101 ++++++++++++++++++ scripts/pycurl-example.py | 4 +- test.ini.in | 4 + 19 files changed, 303 insertions(+), 45 deletions(-) create mode 100644 elasticsearch8.Dockerfile create mode 100644 logstash8.Dockerfile create mode 100755 scripts/esUpdateMapSettings71.sh create mode 100755 scripts/esUpdateMapSettings78.sh diff --git a/c2corg_api/__init__.py b/c2corg_api/__init__.py index 4eae9553f..4607e110c 100644 --- a/c2corg_api/__init__.py +++ b/c2corg_api/__init__.py @@ -30,7 +30,6 @@ def __init__(self, request): def main(global_config, **settings): """ This function returns a Pyramid WSGI application. """ - # Configure SQLAlchemy engine = engine_from_config(settings, 'sqlalchemy.') DBSession.configure(bind=engine) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 30e5c2359..993c7beb3 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -2,18 +2,6 @@ import sys import pycurl from io import BytesIO - -# from c2corg_api.search.mappings.area_mapping import SearchArea -# from c2corg_api.search.mappings.article_mapping import SearchArticle -# from c2corg_api.search.mappings.book_mapping import SearchBook -# from c2corg_api.search.mappings.image_mapping import SearchImage -# from c2corg_api.search.mappings.outing_mapping import SearchOuting -# from c2corg_api.search.mappings.xreport_mapping import SearchXreport -# from c2corg_api.search.mappings.route_mapping import SearchRoute -# from c2corg_api.search.mappings.topo_map_mapping import SearchTopoMap -# from c2corg_api.search.mappings.user_mapping import SearchUser -# from c2corg_api.search.mappings.waypoint_mapping import SearchWaypoint - from elasticsearch_dsl import Index from pyramid.paster import ( @@ -22,8 +10,6 @@ ) from pyramid.scripts.common import parse_vars - -# from c2corg_api.search.mapping import analysis_settings from c2corg_api.search import configure_es_from_config, elasticsearch_config @@ -52,6 +38,10 @@ def setup_es(): "_m", "_o", "_r", "_u", "_w", "_x"] client = elasticsearch_config['client'] + elasticsearch_user = elasticsearch_config['user'] + elasticsearch_password = elasticsearch_config['passwd'] + global authentification + authentification = elasticsearch_user+':'+elasticsearch_password index_name = elasticsearch_config['index'] cible = elasticsearch_config['host']+':'+str(elasticsearch_config['port']) # print('cible es: %s', cible) @@ -60,8 +50,8 @@ def setup_es(): print('ElasticSearch version: {0}'.format(info['version']['number'])) for index_suffix in index_suffix_list: - # print("suffix : ", index_suffix) - if client.indices.exists(index_name[:-2]+index_suffix): + indice = index_name[:-2]+index_suffix + if client.indices.exists(index=indice): print('Index "{0}" already exists. deleting it {0}... ' .format(index_name[:-2]+index_suffix)) """ print('To delete the index run:') @@ -102,9 +92,9 @@ def delete_indice(cible, indice_name): c.setopt(c.CUSTOMREQUEST, 'DELETE') c.setopt(c.URL, 'http://' + cible + '/' + indice_name) c.setopt(c.WRITEDATA, buffer) + c.setopt(c.USERPWD, authentification) c.perform() c.close() - # body = buffer.getvalue() # print(body.decode('iso-8859-1')) @@ -115,12 +105,11 @@ def create_indice(cible, indice_name): header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') - c.setopt(c.URL, 'http://' + cible + '/' + indice_name - + "?include_type_name=true") + c.setopt(c.URL, 'http://' + cible + '/' + indice_name) c.setopt(c.WRITEDATA, buffer) + c.setopt(c.USERPWD, authentification) c.perform() c.close() - # body = buffer.getvalue() # print(body.decode('iso-8859-1')) @@ -133,6 +122,7 @@ def indice_settings_update(cible, indice_name): c.setopt(c.CUSTOMREQUEST, 'POST') c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_close') c.setopt(c.WRITEDATA, buffer) + c.setopt(c.USERPWD, authentification) c.perform() # body = buffer.getvalue() # print(body.decode('iso-8859-1')) @@ -144,6 +134,7 @@ def indice_settings_update(cible, indice_name): c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_settings') c.setopt(c.POSTFIELDS, post_data) c.setopt(c.WRITEDATA, buffer) + c.setopt(c.USERPWD, authentification) c.perform() # body = buffer.getvalue() # print(body.decode('iso-8859-1')) @@ -155,6 +146,7 @@ def indice_settings_update(cible, indice_name): c.setopt(c.CUSTOMREQUEST, 'POST') c.setopt(c.URL, 'http://' + cible + '/' + indice_name + '/_open') c.setopt(c.WRITEDATA, buffer) + c.setopt(c.USERPWD, authentification) c.perform() # body = buffer.getvalue() # print(body.decode('iso-8859-1')) @@ -169,13 +161,13 @@ def indice_mapping_update(cible, indice_name, mapping_type): buffer = BytesIO() c = pycurl.Curl() - c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping' - + '/'+mapping_type+'?include_type_name=true') + c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping') header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') c.setopt(c.POSTFIELDS, post_data) c.setopt(c.WRITEDATA, buffer) + c.setopt(c.USERPWD, authentification) c.perform() c.close() diff --git a/c2corg_api/search/__init__.py b/c2corg_api/search/__init__.py index 0aa9308d7..68bb64740 100644 --- a/c2corg_api/search/__init__.py +++ b/c2corg_api/search/__init__.py @@ -38,15 +38,21 @@ 'client': None, 'index': None, 'host': None, - 'port': None + 'port': None, + 'scheme': None, + 'user': None, + 'passwd': None, } def client_from_config(settings): return Elasticsearch([{ 'host': settings['elasticsearch.host'], - 'port': int(settings['elasticsearch.port']) - }], maxsize=int(settings['elasticsearch.pool'])) + 'port': int(settings['elasticsearch.port']), + 'scheme': settings['elasticsearch.scheme'] + }], basic_auth=(settings['elasticsearch.user'], + settings['elasticsearch.passwd']), + maxsize=int(settings['elasticsearch.pool'])) def configure_es_from_config(settings): @@ -57,6 +63,9 @@ def configure_es_from_config(settings): elasticsearch_config['index'] = settings['elasticsearch.index'] elasticsearch_config['host'] = settings['elasticsearch.host'] elasticsearch_config['port'] = int(settings['elasticsearch.port']) + elasticsearch_config['scheme'] = settings['elasticsearch.scheme'] + elasticsearch_config['user'] = settings['elasticsearch.user'] + elasticsearch_config['passwd'] = settings['elasticsearch.passwd'] def get_queue_config(settings): diff --git a/c2corg_api/search/advanced_search.py b/c2corg_api/search/advanced_search.py index ac613ff8b..7aef7a92f 100644 --- a/c2corg_api/search/advanced_search.py +++ b/c2corg_api/search/advanced_search.py @@ -1,4 +1,5 @@ import logging + from c2corg_api.search.mapping_types import meta_param_keys from c2corg_api.search.search_filters import build_query diff --git a/c2corg_api/search/search_filters.py b/c2corg_api/search/search_filters.py index a44a99500..dad21db22 100644 --- a/c2corg_api/search/search_filters.py +++ b/c2corg_api/search/search_filters.py @@ -371,8 +371,8 @@ def create_bbox_filter(query_term): return None return GeoBoundingBox( - geom={'left': xmin, 'bottom': ymin, 'right': xmax, 'top': ymax}, - type='indexed' + geom={'left': xmin, 'bottom': ymin, 'right': xmax, 'top': ymax} + # type='indexed' ) diff --git a/c2corg_api/tests/scripts/es/test_syncer.py b/c2corg_api/tests/scripts/es/test_syncer.py index 0cefc0517..a17ba3fb3 100644 --- a/c2corg_api/tests/scripts/es/test_syncer.py +++ b/c2corg_api/tests/scripts/es/test_syncer.py @@ -61,4 +61,4 @@ def test_process_task(self): next(syncer.consume(limit=1)) with self.assertRaises(NotFoundError): - SearchWaypoint.get(id=document_id, index=index) + SearchWaypoint.get(id=document_id) # JPR8 , index=index diff --git a/c2corg_api/tests/search/__init__.py b/c2corg_api/tests/search/__init__.py index a33b90592..80d748480 100644 --- a/c2corg_api/tests/search/__init__.py +++ b/c2corg_api/tests/search/__init__.py @@ -16,4 +16,4 @@ def force_search_index(): """Force that the search index is updated. """ elasticsearch_config['client'].indices.refresh( - elasticsearch_config['index']) + index=elasticsearch_config['index']) diff --git a/c2corg_api/tests/search/test_search_filters.py b/c2corg_api/tests/search/test_search_filters.py index 391d059f7..a193947e8 100644 --- a/c2corg_api/tests/search/test_search_filters.py +++ b/c2corg_api/tests/search/test_search_filters.py @@ -50,8 +50,8 @@ def test_build_query_bbox(self): filter(GeoBoundingBox( geom={ 'left': 6.28279913, 'bottom': 46.03129072, - 'right': 6.28369744, 'top': 46.03191439}, - type='indexed')). \ + 'right': 6.28369744, 'top': 46.03191439} # , type='indexed' + )). \ filter(Q("match", c2corg_doc_type='w')). \ extra(from_=0, size=10) self.assertQueryEqual(query, expected_query) @@ -460,8 +460,9 @@ def test_create_bbox_filter(self): GeoBoundingBox( geom={ 'left': 6.28279913, 'bottom': 46.03129072, - 'right': 6.28369744, 'top': 46.03191439}, - type='indexed').to_dict()) + 'right': 6.28369744, 'top': 46.03191439} # ,type='indexed' + ).to_dict()) + def assertBboxFilterEqual(self, f1, f2): # noqa self.assertIn('geo_bounding_box', f1) diff --git a/c2corg_api/views/health.py b/c2corg_api/views/health.py index 8b1c36495..149c996e2 100644 --- a/c2corg_api/views/health.py +++ b/c2corg_api/views/health.py @@ -64,7 +64,7 @@ def _add_es_status(self, status): try: client = elasticsearch_config['client'] index = elasticsearch_config['index'] - stats = client.indices.stats(index, metric='docs') + stats = client.indices.stats(index=index, metric='docs') document_count_all_indice = 0 for indice in stats['indices']: document_count_all_indice += \ diff --git a/common.ini.in b/common.ini.in index 337ec44e2..b1f117be0 100644 --- a/common.ini.in +++ b/common.ini.in @@ -15,6 +15,9 @@ sqlalchemy.pool_size=20 elasticsearch.host = {elasticsearch_host} elasticsearch.port = {elasticsearch_port} elasticsearch.index = {elasticsearch_index} +elasticsearch.scheme = {elasticsearch_scheme} +elasticsearch.user = {elasticsearch_user} +elasticsearch.passwd = {elasticsearch_passwd} elasticsearch.pool = 50 # batch size for fill_index diff --git a/config/docker-dev b/config/docker-dev index 153dab388..cab0d2ed3 100644 --- a/config/docker-dev +++ b/config/docker-dev @@ -6,15 +6,23 @@ include Makefile export db_host = postgresql export db_name = c2corg + export elasticsearch_host = elasticsearch export elasticsearch_port = 9200 export elasticsearch_index = c2corg_* +export elasticsearch_user = elastic +export elasticsearch_passwd = elastic2024 +export elasticsearch_scheme = http export tests_db_host = postgresql export tests_db_name = c2corg_tests + export tests_elasticsearch_host = elasticsearch export tests_elasticsearch_port = 9200 export tests_elasticsearch_index = c2corg_tests_* +export tests_elasticsearch_user = elastic +export tests_elasticsearch_passwd = elastic2024 +export tests_elasticsearch_scheme = http export redis_db_queue = 6 export redis_db_cache = 7 diff --git a/docker-compose.yml b/docker-compose.yml index 7d0a9c816..3efbbd3d6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -68,17 +68,33 @@ services: # - 9302:9300 elasticsearch: - platform: linux/amd64 + #platform: linux/amd64 container_name: elasticsearch build: context: . - dockerfile: elasticsearch7.Dockerfile + dockerfile: elasticsearch8.Dockerfile network: host environment: - - discovery.type=single-node + - ELASTIC_PASSWORD='elastic2024' + - discovery.type=single-node + - xpack.security.http.ssl.enabled=false + - xpack.license.self_generated.type=basic ports: - 9200:9200 - 9300:9300 + +# elasticsearch7: +# platform: linux/amd64 +# container_name: elasticsearch +# build: +# context: . +# dockerfile: elasticsearch7.Dockerfile +# network: host +# environment: +# - discovery.type=single-node +# ports: +# - 9200:9200 +# - 9300:9300 # # elasticsearch68: @@ -121,17 +137,43 @@ services: # volumes: # - ./dataMigration/v6:/usr/share/logstash/pipeline - logstash7: - platform: linux/amd64 - container_name: logstash7 + logstash: + #platform: linux/amd64 + container_name: logstash build: context: . - dockerfile: logstash7.Dockerfile + dockerfile: logstash8.Dockerfile network: host ports: - 5044:5044 volumes: - ./dataMigration/v7:/usr/share/logstash/pipeline + environment: + - ELASTICSEARCH_URL=http://elasticsearch:9200 + - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 + - ELASTICSEARCH_USERNAME=kibana_system + - ELASTICSEARCH_PASSWORD="kibana2024" + - xpack.security.enabled=false + #- xpack.license.self_generated.type=trial + + ## configure the Kibana password in the ES container + #curl -u elastic:$ELASTIC_PASSWORD \ + # -X POST \ + # http://localhost:9200/_security/user/kibana_system/_password \ + # -d '{"password":"'"kibana2024"'"}' \ + # -H 'Content-Type: application/json' + +# logstash7: +# platform: linux/amd64 +# container_name: logstash7 +# build: +# context: . +# dockerfile: logstash7.Dockerfile +# network: host +# ports: +# - 5044:5044 +# volumes: +# - ./dataMigration/v7:/usr/share/logstash/pipeline redis: image: 'docker.io/redis:3.2' diff --git a/elasticsearch8.Dockerfile b/elasticsearch8.Dockerfile new file mode 100644 index 000000000..e40774ef3 --- /dev/null +++ b/elasticsearch8.Dockerfile @@ -0,0 +1,2 @@ +FROM elasticsearch:8.14.1 +RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu diff --git a/logstash8.Dockerfile b/logstash8.Dockerfile new file mode 100644 index 000000000..c306e1aa4 --- /dev/null +++ b/logstash8.Dockerfile @@ -0,0 +1,2 @@ +FROM logstash:8.14.1 +RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-prune diff --git a/requirements.txt b/requirements.txt index 9c46c2af2..0c8ebf5ac 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,8 +4,8 @@ bcrypt==4.1.2 bleach[css]==6.1.0 colander==2.0 dogpile.cache==1.3.2 -elasticsearch==7.17.9 # elasticsearch==2.4.1 -elasticsearch-dsl==7.4.1 # elasticsearch_dsl==2.2.0 +elasticsearch==8.14.0 # elasticsearch==2.4.1 +elasticsearch-dsl==8.14.0 # elasticsearch_dsl==2.2.0 geoalchemy2==0.4.2 geojson==3.1.0 geomet==1.1.0 diff --git a/scripts/esUpdateMapSettings71.sh b/scripts/esUpdateMapSettings71.sh new file mode 100755 index 000000000..878d5726c --- /dev/null +++ b/scripts/esUpdateMapSettings71.sh @@ -0,0 +1,92 @@ +#!/bin/bash +#change localhost to the correct url if elasticsearch is not hosted on your computer, or in docker +#operate the plugin installation from the server side manually if not dockerized. +#installation du plugin es et reboot +#Install elastic plugin analysis-icu +#echo 'Installing es plugin to elasticsearch docker' +#docker-compose exec -it elasticsearch bin/plugin install analysis-icu +#echo 'Installing done - restarting es docker' +#docker-compose restart elasticsearch +#sleep 10 #in sec to let the docker restart + +#efface tous les indices présents +curl -X DELETE "http://192.168.1.120:9205/c2corg_*" + +#change cluster definition des shards +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/cluster.json http://192.168.1.120:9205/_template/index_defaults + +#créer les index +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_a?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_b?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_c?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_i?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_m?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_o?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_r?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_u?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_w?include_type_name=true +curl -X PUT -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_x?include_type_name=true +sleep 2 + +#2. closing the index to upgrade settings +echo -e '\n closing index' +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_a/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_b/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_c/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_i/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_m/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_o/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_r/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_u/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_w/_close +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_x/_close +sleep 2 +#3. Updating analyser settings +echo -e '\n updating Settings' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_a/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_b/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_c/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_i/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_m/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_o/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_r/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_u/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_w/_settings +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://192.168.1.120:9205/c2corg_x/_settings +sleep 2 +#4. restarting the index +echo -e '\n restarting index' +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_a/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_b/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_c/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_i/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_m/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_o/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_r/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_u/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_w/_open +curl -X POST -H "Content-Type: application/json" http://192.168.1.120:9205/c2corg_x/_open +sleep 2 +#5. Docs Mappings update +echo 'updating doc type mappings' +echo 'doc type : a' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://192.168.1.120:9205/c2corg_a/_mapping/a?include_type_name=true +echo -e '\n doc type : b' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://192.168.1.120:9205/c2corg_b/_mapping/b?include_type_name=true +echo -e '\n doc type : c' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://192.168.1.120:9205/c2corg_c/_mapping/c?include_type_name=true +echo -e '\n doc type : i' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://192.168.1.120:9205/c2corg_i/_mapping/i?include_type_name=true +echo -e '\n doc type : m' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://192.168.1.120:9205/c2corg_m/_mapping/m?include_type_name=true +echo -e '\n doc type : o' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://192.168.1.120:9205/c2corg_o/_mapping/o?include_type_name=true +echo -e '\n doc type : r' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://192.168.1.120:9205/c2corg_r/_mapping/r?include_type_name=true +echo -e '\n doc type : u' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://192.168.1.120:9205/c2corg_u/_mapping/u?include_type_name=true +echo -e '\n doc type : w' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://192.168.1.120:9205/c2corg_w/_mapping/w?include_type_name=true +echo -e '\n doc type : x' +curl -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://192.168.1.120:9205/c2corg_x/_mapping/x?include_type_name=true +sleep 2 diff --git a/scripts/esUpdateMapSettings78.sh b/scripts/esUpdateMapSettings78.sh new file mode 100755 index 000000000..ec7545695 --- /dev/null +++ b/scripts/esUpdateMapSettings78.sh @@ -0,0 +1,101 @@ +#!/bin/bash +#change localhost to the correct url if elasticsearch is not hosted on your computer, or in docker +#operate the plugin installation from the server side manually if not dockerized. +#installation du plugin es et reboot +#Install elastic plugin analysis-icu +#echo 'Installing es plugin to elasticsearch docker' +#docker-compose exec -it elasticsearch bin/plugin install analysis-icu +#echo 'Installing done - restarting es docker' +#docker-compose restart elasticsearch +#sleep 10 #in sec to let the docker restart + +#efface tous les indices présents +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_a" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_b" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_c" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_i" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_m" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_o" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_r" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_u" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_w" +curl -u elastic:elastic2024 -X DELETE "http://localhost:9200/c2corg_x" + +#change cluster definition des shards +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/cluster.json http://localhost:9200/_template/index_defaults + +#créer les index +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_a +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_b +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_c +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_i +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_m +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_o +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_r +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_u +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_w +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" http://localhost:9200/c2corg_x +sleep 2 + +#2. closing the index to upgrade settings +echo -e '\n closing index' +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_a/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_b/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_c/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_i/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_m/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_o/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_r/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_u/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_w/_close +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_x/_close +sleep 2 +#3. Updating analyser settings +echo -e '\n updating Settings' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_a/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_b/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_c/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_i/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_m/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_o/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_r/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_u/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_w/_settings +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/settings.json http://localhost:9200/c2corg_x/_settings +sleep 2 +#4. restarting the index +echo -e '\n restarting index' +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_a/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_b/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_c/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_i/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_m/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_o/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_r/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_u/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_w/_open +curl -u elastic:elastic2024 -X POST -H "Content-Type: application/json" http://localhost:9200/c2corg_x/_open +sleep 2 +#5. Docs Mappings update +echo 'updating doc type mappings' +echo 'doc type : a' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/a.json http://localhost:9200/c2corg_a/_mapping +echo -e '\n doc type : b' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/b.json http://localhost:9200/c2corg_b/_mapping +echo -e '\n doc type : c' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/c.json http://localhost:9200/c2corg_c/_mapping +echo -e '\n doc type : i' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/i.json http://localhost:9200/c2corg_i/_mapping +echo -e '\n doc type : m' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/m.json http://localhost:9200/c2corg_m/_mapping +echo -e '\n doc type : o' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/o.json http://localhost:9200/c2corg_o/_mapping +echo -e '\n doc type : r' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/r.json http://localhost:9200/c2corg_r/_mapping +echo -e '\n doc type : u' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/u.json http://localhost:9200/c2corg_u/_mapping +echo -e '\n doc type : w' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/w.json http://localhost:9200/c2corg_w/_mapping +echo -e '\n doc type : x' +curl -u elastic:elastic2024 -X PUT -H "Content-Type: application/json" -d @./scripts/esjson6-7/x.json http://localhost:9200/c2corg_x/_mapping +sleep 2 diff --git a/scripts/pycurl-example.py b/scripts/pycurl-example.py index e38ac8b73..c6979ac96 100755 --- a/scripts/pycurl-example.py +++ b/scripts/pycurl-example.py @@ -1,3 +1,4 @@ +#!/usr/bin/python3 from urllib.parse import urlencode import pycurl @@ -10,7 +11,7 @@ buffer = BytesIO() c = pycurl.Curl() -c.setopt(c.URL, 'http://192.168.1.120:9203/c2corg_tests_a/_mapping/_doc') +c.setopt(c.URL, 'http://127.0.0.1:9200/c2corg_tests_a/_mapping') header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') @@ -18,6 +19,7 @@ # Sets request method to POST, # Content-Type header to application/x-www-form-urlencoded # and data to send in request body. +c.setopt(c.USERPWD, 'elastic:elastic2024') c.setopt(c.POSTFIELDS, post_data) c.setopt(c.WRITEDATA, buffer) c.perform() diff --git a/test.ini.in b/test.ini.in index 1915e91d5..631c1c89c 100644 --- a/test.ini.in +++ b/test.ini.in @@ -10,6 +10,10 @@ jwtauth.master_secret = The master key elasticsearch.host = {tests_elasticsearch_host} elasticsearch.port = {tests_elasticsearch_port} elasticsearch.index = {tests_elasticsearch_index} +elasticsearch.scheme = {tests_elasticsearch_scheme} +elasticsearch.user = {tests_elasticsearch_user} +elasticsearch.passwd = {tests_elasticsearch_passwd} + redis.url = {redis_url} redis.cache_key_prefix = {redis_cache_key_prefix}_tests cache_version_timestamp = True From 37055ab023a472058b56edde9ebe7fa7569d1b40 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 26 Jun 2024 18:50:24 +0200 Subject: [PATCH 32/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - artifact fix for ES download - mapping - class - requirements.txt - docker definition - CI update - logstash data migration update --- .github/workflows/ci.yml | 7 ++--- dataMigration/v8/logstash.conf | 47 ++++++++++++++++++++++++++++++++++ docker-compose.yml | 12 ++++----- 3 files changed, 57 insertions(+), 9 deletions(-) create mode 100644 dataMigration/v8/logstash.conf diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 22f765e70..dab9ee622 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -75,13 +75,14 @@ jobs: uses: actions/setup-java@v4 with: distribution: 'adopt' - java-version: 8 - - name: Run (old) elasticsearch + java-version: 22 + - name: Run elasticsearch run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.22-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize + /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - run: curl -v http://localhost:9200 - name: Run tests diff --git a/dataMigration/v8/logstash.conf b/dataMigration/v8/logstash.conf new file mode 100644 index 000000000..dfa7da252 --- /dev/null +++ b/dataMigration/v8/logstash.conf @@ -0,0 +1,47 @@ +input +{ + elasticsearch + { + hosts => ["http://192.168.1.120:9205"] + index => "c2corg_*" + docinfo => true + size => 100 + scroll => "5m" + docinfo_target => "[@metadata]" + } +} + +filter { + if (![geom]) { + mutate { + add_field => { "geom" => [0,0] } + } + } + + #mutate { + # add_field => { "c2corg_doc_type" => "%{[@metadata][_type]}" } + #} + + mutate { + remove_field => [ "_type", "@timestamp", "@version", "doc_id", "doc_type" ] + } + +} + +output +{ + elasticsearch + { + hosts => [elasticsearch] + index => "c2corg_%{[c2corg_doc_type]}" + #document_type => "_doc" + document_id => "%{[@metadata][_id]}" + user => 'elastic' + password => 'elastic2024' + } + + stdout { + codec => "dots" + } + +} diff --git a/docker-compose.yml b/docker-compose.yml index 3efbbd3d6..acbff0883 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -147,14 +147,14 @@ services: ports: - 5044:5044 volumes: - - ./dataMigration/v7:/usr/share/logstash/pipeline + - ./dataMigration/v8:/usr/share/logstash/pipeline environment: - - ELASTICSEARCH_URL=http://elasticsearch:9200 - - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 - - ELASTICSEARCH_USERNAME=kibana_system - - ELASTICSEARCH_PASSWORD="kibana2024" + - ELASTICSEARCH_URL=elasticsearch + - ELASTICSEARCH_HOSTS=elasticsearch + - ELASTICSEARCH_USERNAME='elastic' + - ELASTICSEARCH_PASSWORD='elastic2024' - xpack.security.enabled=false - #- xpack.license.self_generated.type=trial + - xpack.license.self_generated.type=basic ## configure the Kibana password in the ES container #curl -u elastic:$ELASTIC_PASSWORD \ From 0495840bde948c8c3d68d51268bb669bbb179b7c Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 26 Jun 2024 18:59:19 +0200 Subject: [PATCH 33/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - lint --- c2corg_api/tests/search/test_search_filters.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/c2corg_api/tests/search/test_search_filters.py b/c2corg_api/tests/search/test_search_filters.py index a193947e8..237771f1d 100644 --- a/c2corg_api/tests/search/test_search_filters.py +++ b/c2corg_api/tests/search/test_search_filters.py @@ -50,7 +50,8 @@ def test_build_query_bbox(self): filter(GeoBoundingBox( geom={ 'left': 6.28279913, 'bottom': 46.03129072, - 'right': 6.28369744, 'top': 46.03191439} # , type='indexed' + 'right': 6.28369744, 'top': 46.03191439} + # , type='indexed' )). \ filter(Q("match", c2corg_doc_type='w')). \ extra(from_=0, size=10) @@ -460,7 +461,8 @@ def test_create_bbox_filter(self): GeoBoundingBox( geom={ 'left': 6.28279913, 'bottom': 46.03129072, - 'right': 6.28369744, 'top': 46.03191439} # ,type='indexed' + 'right': 6.28369744, 'top': 46.03191439} + # ,type='indexed' ).to_dict()) From 1b21516b1c4870b1bb37eb973e7d2cdb921ed8eb Mon Sep 17 00:00:00 2001 From: jphilippe Date: Wed, 26 Jun 2024 19:01:30 +0200 Subject: [PATCH 34/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - lint --- c2corg_api/tests/search/test_search_filters.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/c2corg_api/tests/search/test_search_filters.py b/c2corg_api/tests/search/test_search_filters.py index 237771f1d..9956af4bf 100644 --- a/c2corg_api/tests/search/test_search_filters.py +++ b/c2corg_api/tests/search/test_search_filters.py @@ -51,7 +51,6 @@ def test_build_query_bbox(self): geom={ 'left': 6.28279913, 'bottom': 46.03129072, 'right': 6.28369744, 'top': 46.03191439} - # , type='indexed' )). \ filter(Q("match", c2corg_doc_type='w')). \ extra(from_=0, size=10) @@ -462,7 +461,6 @@ def test_create_bbox_filter(self): geom={ 'left': 6.28279913, 'bottom': 46.03129072, 'right': 6.28369744, 'top': 46.03191439} - # ,type='indexed' ).to_dict()) From 066c55518745af028de1fcae1c668e34339cdabd Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 09:48:57 +0200 Subject: [PATCH 35/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - templates --- config/default | 6 ++++++ config/github-actions | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/config/default b/config/default index e6b4da1f5..060c8311d 100644 --- a/config/default +++ b/config/default @@ -17,6 +17,9 @@ export db_name = c2corg export elasticsearch_host = localhost export elasticsearch_port = 9200 export elasticsearch_index = c2corg +export elasticsearch_user = elastic +export elasticsearch_passwd = elastic2024 +export elasticsearch_scheme = http export elasticsearch_batch_size_fill_index = 1000 export elasticsearch_batch_size_syncer = 1000 @@ -46,6 +49,9 @@ export tests_db_name = c2corg_tests export tests_elasticsearch_host = localhost export tests_elasticsearch_port = 9200 export tests_elasticsearch_index = c2corg_tests +export tests_elasticsearch_user = elastic +export tests_elasticsearch_passwd = elastic2024 +export tests_elasticsearch_scheme = http # Discourse instance to run the unit tests export tests_discourse_url = http://localhost:3000 diff --git a/config/github-actions b/config/github-actions index 5678ff812..8f97a6d0a 100644 --- a/config/github-actions +++ b/config/github-actions @@ -6,6 +6,12 @@ export db_name = c2corg_github_tests export tests_db_name = c2corg_github_tests export elasticsearch_port = 9200 export elasticsearch_index = c2corg_github_tests_* +export elasticsearch_user = elastic +export elasticsearch_passwd = elastic2024 +export elasticsearch_scheme = http export tests_elasticsearch_port = 9200 export tests_elasticsearch_index = c2corg_github_tests_* +export tests_elasticsearch_user = elastic +export tests_elasticsearch_passwd = elastic2024 +export tests_elasticsearch_scheme = http export redis_url = redis://localhost:6379/ \ No newline at end of file From 6f86d28d15b907a6d06eae2219af0959c7af5d63 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 10:14:04 +0200 Subject: [PATCH 36/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 2 +- docker-compose.yml | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dab9ee622..7773bb35e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -84,7 +84,7 @@ jobs: /tmp/elasticsearch/bin/elasticsearch --daemonize /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - - run: curl -v http://localhost:9200 + - run: curl -u elastic:elastic2024 -v http://localhost:9200 - name: Run tests run: pytest --cov-report term --cov-report xml --cov=c2corg_api - name: Send coverage to codacy diff --git a/docker-compose.yml b/docker-compose.yml index acbff0883..da510c9a9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -75,7 +75,7 @@ services: dockerfile: elasticsearch8.Dockerfile network: host environment: - - ELASTIC_PASSWORD='elastic2024' + - ELASTIC_PASSWORD=elastic2024 - discovery.type=single-node - xpack.security.http.ssl.enabled=false - xpack.license.self_generated.type=basic @@ -151,8 +151,8 @@ services: environment: - ELASTICSEARCH_URL=elasticsearch - ELASTICSEARCH_HOSTS=elasticsearch - - ELASTICSEARCH_USERNAME='elastic' - - ELASTICSEARCH_PASSWORD='elastic2024' + - ELASTICSEARCH_USERNAME=elastic + - ELASTICSEARCH_PASSWORD=elastic2024 - xpack.security.enabled=false - xpack.license.self_generated.type=basic From f0106ceb382f41344c2c3c6ad617d056744a90fd Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 12:05:37 +0200 Subject: [PATCH 37/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7773bb35e..015045bce 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,6 +15,16 @@ jobs: --health-retries 5 ports: - 6379:6379 + elasticsearch: + image: logstash:8.14.1 + env: + ELASTIC_PASSWORD=elastic2024 + discovery.type=single-node + xpack.security.http.ssl.enabled=false + xpack.license.self_generated.type=basic + ports: + - 9200:9200 + - 9300:9300 # elasticsearch: # image: elasticsearch:6.8.14 # options: >- @@ -71,20 +81,22 @@ jobs: PGPORT: 5432 - name: Create config from templates run: make -f config/github-actions template - - name: Install java 8 for (old) elasticsearch - uses: actions/setup-java@v4 - with: - distribution: 'adopt' - java-version: 22 - - name: Run elasticsearch - run: | - mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu - /tmp/elasticsearch/bin/elasticsearch --daemonize - /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - - run: curl -u elastic:elastic2024 -v http://localhost:9200 + #- name: Install java 8 for (old) elasticsearch + # uses: actions/setup-java@v4 + # with: + # distribution: 'adopt' + # java-version: 22 + - name: Check elasticsearch + # run: | + # mkdir /tmp/elasticsearch + # wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + # /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu + # /tmp/elasticsearch/bin/elasticsearch --daemonize + # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" + # sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs + run: curl -u elastic:elastic2024 -v http://localhost:9200 + - name: add elastic analyser + run: docker exec elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu - name: Run tests run: pytest --cov-report term --cov-report xml --cov=c2corg_api - name: Send coverage to codacy From 63eb4ba21275b44b059695564a4ef54960777b29 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 12:27:36 +0200 Subject: [PATCH 38/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 42 +++++++++++++++++----------------------- 1 file changed, 18 insertions(+), 24 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 015045bce..d1f0ed458 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,16 +15,6 @@ jobs: --health-retries 5 ports: - 6379:6379 - elasticsearch: - image: logstash:8.14.1 - env: - ELASTIC_PASSWORD=elastic2024 - discovery.type=single-node - xpack.security.http.ssl.enabled=false - xpack.license.self_generated.type=basic - ports: - - 9200:9200 - - 9300:9300 # elasticsearch: # image: elasticsearch:6.8.14 # options: >- @@ -81,22 +71,26 @@ jobs: PGPORT: 5432 - name: Create config from templates run: make -f config/github-actions template - #- name: Install java 8 for (old) elasticsearch - # uses: actions/setup-java@v4 - # with: - # distribution: 'adopt' - # java-version: 22 + - name: Install java 8 for elasticsearch + uses: actions/setup-java@v4 + with: + distribution: 'adopt' + java-version: 22 + - name: Install elasticsearch + run: | + mkdir /tmp/elasticsearch + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 + truncate -s0 /tmp/elasticsearch/config/elasticsearch.yml + cat >> /tmp/elasticsearch/config/elasticsearch.yml << EOF + cluster.name: "docker-cluster" + network.host: 0.0.0.0 + EOF + /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" + /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu + /tmp/elasticsearch/bin/elasticsearch --daemonize + sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - name: Check elasticsearch - # run: | - # mkdir /tmp/elasticsearch - # wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - # /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu - # /tmp/elasticsearch/bin/elasticsearch --daemonize - # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - # sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs run: curl -u elastic:elastic2024 -v http://localhost:9200 - - name: add elastic analyser - run: docker exec elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu - name: Run tests run: pytest --cov-report term --cov-report xml --cov=c2corg_api - name: Send coverage to codacy From 11687083a37c87735ba575e7a3ebbd793beee843 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 12:34:48 +0200 Subject: [PATCH 39/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d1f0ed458..8f1656612 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -71,7 +71,7 @@ jobs: PGPORT: 5432 - name: Create config from templates run: make -f config/github-actions template - - name: Install java 8 for elasticsearch + - name: Install java 22 for elasticsearch uses: actions/setup-java@v4 with: distribution: 'adopt' @@ -85,10 +85,11 @@ jobs: cluster.name: "docker-cluster" network.host: 0.0.0.0 EOF - /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs + - name: ChangeMe password update + run: /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - name: Check elasticsearch run: curl -u elastic:elastic2024 -v http://localhost:9200 - name: Run tests From 473a5a5178187c73351d86d1caee6fc8840acb53 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 12:41:09 +0200 Subject: [PATCH 40/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8f1656612..604c01d26 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -71,11 +71,11 @@ jobs: PGPORT: 5432 - name: Create config from templates run: make -f config/github-actions template - - name: Install java 22 for elasticsearch - uses: actions/setup-java@v4 - with: - distribution: 'adopt' - java-version: 22 + #- name: Install java 22 for elasticsearch + # uses: actions/setup-java@v4 + # with: + # distribution: 'adopt' + # java-version: 22 - name: Install elasticsearch run: | mkdir /tmp/elasticsearch @@ -88,8 +88,10 @@ jobs: /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - - name: ChangeMe password update - run: /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" + - name: ElasticSearch ChangeMe password update + run: | + /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" + sleep 15 # Password change requires time - name: Check elasticsearch run: curl -u elastic:elastic2024 -v http://localhost:9200 - name: Run tests From 64698188ca22acac495eb133505732b0048711d6 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 13:10:27 +0200 Subject: [PATCH 41/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 604c01d26..2a2d0e66c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -81,19 +81,16 @@ jobs: mkdir /tmp/elasticsearch wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 truncate -s0 /tmp/elasticsearch/config/elasticsearch.yml - cat >> /tmp/elasticsearch/config/elasticsearch.yml << EOF - cluster.name: "docker-cluster" - network.host: 0.0.0.0 - EOF + sed -i 's/true/false/g' /tmp/elasticsearch/config/elasticsearch.yml /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - - name: ElasticSearch ChangeMe password update - run: | - /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - sleep 15 # Password change requires time + #- name: ElasticSearch ChangeMe password update + # run: | + # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" + # sleep 15 # Password change requires time - name: Check elasticsearch - run: curl -u elastic:elastic2024 -v http://localhost:9200 + run: curl -v http://localhost:9200 - name: Run tests run: pytest --cov-report term --cov-report xml --cov=c2corg_api - name: Send coverage to codacy From 353a6d1bf2272aeacfc43bf927706e8626bd2158 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 13:17:55 +0200 Subject: [PATCH 42/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2a2d0e66c..8e2d1b0e2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -84,7 +84,7 @@ jobs: sed -i 's/true/false/g' /tmp/elasticsearch/config/elasticsearch.yml /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize - sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs + sleep 180 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs #- name: ElasticSearch ChangeMe password update # run: | # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" From 18f64821444762b033f8c44c934880709bbac553 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 13:20:00 +0200 Subject: [PATCH 43/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8e2d1b0e2..a85fce90d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -84,7 +84,7 @@ jobs: sed -i 's/true/false/g' /tmp/elasticsearch/config/elasticsearch.yml /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize - sleep 180 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs + sleep 200 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs #- name: ElasticSearch ChangeMe password update # run: | # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" From 88460d7cb7511df75eab32a23083da63e193d603 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 13:21:26 +0200 Subject: [PATCH 44/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a85fce90d..07eec537b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,11 +80,10 @@ jobs: run: | mkdir /tmp/elasticsearch wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - truncate -s0 /tmp/elasticsearch/config/elasticsearch.yml sed -i 's/true/false/g' /tmp/elasticsearch/config/elasticsearch.yml /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize - sleep 200 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs + sleep 180 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs #- name: ElasticSearch ChangeMe password update # run: | # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" From c2a66e929843be3efa90378f9e19c4c1cde56cb7 Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 15:10:21 +0200 Subject: [PATCH 45/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 update --- .github/workflows/ci.yml | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 07eec537b..9710d2865 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,18 +76,20 @@ jobs: # with: # distribution: 'adopt' # java-version: 22 - - name: Install elasticsearch + - name: Install elasticsearch & plugin & 1st start run: | mkdir /tmp/elasticsearch - wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.0-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 - sed -i 's/true/false/g' /tmp/elasticsearch/config/elasticsearch.yml + wget -O - https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.1-linux-x86_64.tar.gz | tar xz --directory=/tmp/elasticsearch --strip-components=1 /tmp/elasticsearch/bin/elasticsearch-plugin install analysis-icu /tmp/elasticsearch/bin/elasticsearch --daemonize - sleep 180 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs - #- name: ElasticSearch ChangeMe password update - # run: | - # /tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - # sleep 15 # Password change requires time + sleep 30 # ElasticSearch takes few seconds to start, make sure it is available when the build script runs + - name: ElasticSearch settings update after auto configuration at 1st start & restart + run: | + sed -i 's/true/false/g' /tmp/elasticsearch/config/elasticsearch.yml + pgrep -f elasticsearch | xargs kill -9 + /tmp/elasticsearch/bin/elasticsearch --daemonize + sleep 30 # Password change requires time + #/tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - name: Check elasticsearch run: curl -v http://localhost:9200 - name: Run tests From efb3ddaeb795dd1c8bf86cc4fd54010ebb7d3b7e Mon Sep 17 00:00:00 2001 From: jphilippe Date: Thu, 27 Jun 2024 15:43:09 +0200 Subject: [PATCH 46/53] Upgrade Elasticsearch from 7.17.22 to 8.14 : - CI elastic 8.22 - README.md Update --- .github/workflows/ci.yml | 1 - scripts/README.md | 60 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9710d2865..af487aecd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -89,7 +89,6 @@ jobs: pgrep -f elasticsearch | xargs kill -9 /tmp/elasticsearch/bin/elasticsearch --daemonize sleep 30 # Password change requires time - #/tmp/elasticsearch/bin/elasticsearch-keystore add -x -f "bootstrap.password" "elastic2024" - name: Check elasticsearch run: curl -v http://localhost:9200 - name: Run tests diff --git a/scripts/README.md b/scripts/README.md index aaf445607..514564f06 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,3 +1,63 @@ +# ElasticSearch Upgrade from v7.22 to V8.14 for docker env. like dev one. + +The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. +The ElasticSearch 7.22 is still build and run but would not be the main ES for the service. It could be decommissioned after migration. +docker-compose file contains as well : + +- api building with integration of the requirements.txt for python pip installation (component updates) +- ES 7.22 listening on port 9201 instead of 9200 : > elasticsearch72 +- redis no change +- postgresql no change +- ES 8.14 listening on port 9200 and 9300 (elasticsearch8.Dockerfile) > elasticsearch +- Logstash 8.14 for data migration purpose. (logstash8.Dockerfile) + +## migration steps + + 1. **build the environment :** ` docker-compose build ` + 2. **Launch the environment :** ` docker-compose up -d ` + 3. **Create the index and its mapping :** `./scripts/esUpdateMappings78.sh` + 4. **Data Migration :** `docker-compose start logstash8` It will stop by itself when migration would be achieved + 5. **postgresql shema up to date:** `docker-compose exec api .build/venv/bin/alembic upgrade head` + +## post migration + +if all the process has been passed without issues. you can comment or remove in the docker-compose.yml services : +- logstash +- elasticsearch72 + +## special note +ElasticSearch 8 integrate new security settings and configuration by design like: +- https +- user account (id, pwd) + +in the docker for dev, I let a basic authentification working and keep http. + +### Base upon the way you make the configuration some adjustments have to be realized. + +- with the docker provided: + - elasticsearch listen on http => app config file : `elasticsearch_sheme = http` && `tests_elasticsearch_scheme = http` https instead for tls + - elastic user elastic have a bootstrap passwd define : `elasticsearch/bin/elasticsearch-keystore add "bootstrap.password" `value `elastic2024` + +- in app config file, it is necessary to report this authentification : + - `elasticsearch_user = elastic` & `elasticsearch_passwd = elastic2024` + - `tests_elasticsearch_user = elastic` & `tests_elasticsearch_passwd = elastic2024` + +- in the logstash pipeline file for data migration purpose, it is also required to report this authentification as well : + ``` + output + { + elasticsearch + { + hosts => [elasticsearch] + index => "c2corg_%{[c2corg_doc_type]}" + #document_type => "_doc" + document_id => "%{[@metadata][_id]}" + user => 'elastic' + password => 'elastic2024' + } + } + + # ElasticSearch Upgrade from v6.8 to V7.22 for docker env. like dev one. The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. From 9045ce5ff530f9ba34d427c310f8c0eb1c565f32 Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Mon, 20 Jul 2026 01:11:13 +0200 Subject: [PATCH 47/53] Add connect/read timeouts to ES pycurl calls in initializees.py The pycurl calls used to create/configure ES indices had no timeout set, so a stalled connection (e.g. under I/O contention) could hang the process indefinitely instead of failing and letting the caller retry. Observed this hang concretely while running the test suite locally: the process sat idle (near-0% CPU, no network I/O) for several minutes with no way to recover short of killing it. --- c2corg_api/scripts/initializees.py | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/c2corg_api/scripts/initializees.py b/c2corg_api/scripts/initializees.py index 993c7beb3..477d43533 100644 --- a/c2corg_api/scripts/initializees.py +++ b/c2corg_api/scripts/initializees.py @@ -84,9 +84,16 @@ def drop_index(silent=True): raise exc +def _configure_curl(c): + # avoid indefinite hangs if the ES connection stalls + c.setopt(c.CONNECTTIMEOUT, 10) + c.setopt(c.TIMEOUT, 30) + + def delete_indice(cible, indice_name): buffer = BytesIO() c = pycurl.Curl() + _configure_curl(c) header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'DELETE') @@ -95,13 +102,12 @@ def delete_indice(cible, indice_name): c.setopt(c.USERPWD, authentification) c.perform() c.close() - # body = buffer.getvalue() - # print(body.decode('iso-8859-1')) def create_indice(cible, indice_name): buffer = BytesIO() c = pycurl.Curl() + _configure_curl(c) header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'PUT') @@ -110,13 +116,12 @@ def create_indice(cible, indice_name): c.setopt(c.USERPWD, authentification) c.perform() c.close() - # body = buffer.getvalue() - # print(body.decode('iso-8859-1')) def indice_settings_update(cible, indice_name): buffer = BytesIO() c = pycurl.Curl() + _configure_curl(c) header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'POST') @@ -124,8 +129,6 @@ def indice_settings_update(cible, indice_name): c.setopt(c.WRITEDATA, buffer) c.setopt(c.USERPWD, authentification) c.perform() - # body = buffer.getvalue() - # print(body.decode('iso-8859-1')) file = "./scripts/esjson6-7/settings.json" f = open(file) @@ -136,11 +139,10 @@ def indice_settings_update(cible, indice_name): c.setopt(c.WRITEDATA, buffer) c.setopt(c.USERPWD, authentification) c.perform() - # body = buffer.getvalue() - # print(body.decode('iso-8859-1')) c.close() c = pycurl.Curl() + _configure_curl(c) header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) c.setopt(c.CUSTOMREQUEST, 'POST') @@ -148,8 +150,6 @@ def indice_settings_update(cible, indice_name): c.setopt(c.WRITEDATA, buffer) c.setopt(c.USERPWD, authentification) c.perform() - # body = buffer.getvalue() - # print(body.decode('iso-8859-1')) c.close() @@ -161,6 +161,7 @@ def indice_mapping_update(cible, indice_name, mapping_type): buffer = BytesIO() c = pycurl.Curl() + _configure_curl(c) c.setopt(c.URL, 'http://'+cible+'/'+indice_name+'/_mapping') header = ['Content-Type: application/json'] c.setopt(c.HTTPHEADER, header) @@ -170,6 +171,3 @@ def indice_mapping_update(cible, indice_name, mapping_type): c.setopt(c.USERPWD, authentification) c.perform() c.close() - - # body = buffer.getvalue() - # print(body.decode('iso-8859-1')) From faf492239d5c8498b80285f9b2b0c2af9def2639 Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Mon, 20 Jul 2026 08:50:30 +0200 Subject: [PATCH 48/53] Ignore local sandbox env file config/env.sandbox is a throwaway env file pointing at local migration-rehearsal container hostnames, not meant to be committed. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 3b2cd35ac..3b3e58184 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,4 @@ coverage.xml /config/env.local project.tar scripts/anonymize-database/*/ +config/env.sandbox From 30c794b5181899c545db6252a7a00c1ab43b555f Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Mon, 20 Jul 2026 13:25:05 +0200 Subject: [PATCH 49/53] Migrate to Python 3.12 and SQLAlchemy 2.0 - Bump base images (docker/Dockerfile, docker/Dockerfile.dev, .devcontainer/Dockerfile) from python:3.9 to python:3.12-slim-bookworm, and fix bullseye-only package name (libffi7 -> libffi8) for bookworm. - Bump CI to Python 3.12. - requirements.txt: SQLAlchemy 1.3.24 -> 2.0.51, geoalchemy2 0.4.2 -> 0.20.0. ColanderAlchemy temporarily points at a local, unpublished patch (branch sqlalchemy-2.0-support in the ColanderAlchemy repo) that relaxes its own SQLAlchemy pin to >=1.4 - needs write access to c2corg/ColanderAlchemy to publish for real. - models/__init__.py: declarative_base import moved from sqlalchemy.ext.declarative (pre-1.4) to sqlalchemy.orm. - ext/colander_ext.py: removed the geoalchemy2.compat import (buffer/bytes were just Python 2/3 shims; bytes is the builtin, buffer() is memoryview() on Python 3). - models/document.py, models/stoparea.py: removed the `management=True` Geometry() kwarg, which geoalchemy2 0.20 no longer supports (modern PostGIS uses typmod columns, not the old catalog-managed style). - models/association_views.py, area_association.py, topo_map_association.py, feed.py: ported old-style select([col1, col2]) calls to the SQLAlchemy 1.4+/2.0 select(col1, col2) signature; added .scalar_subquery() to two column_property() selects in feed.py per the 2.0 deprecation warning. Verified: `import c2corg_api.models` (all ~20 model files, every document type) now succeeds cleanly, and ColanderAlchemy schema generation (SQLAlchemySchemaNode direct instantiation, the pattern this codebase actually uses) produces correct field lists. Not yet done: the view layer (c2corg_api/views/) and scripts/ almost certainly have their own SQLAlchemy 1.3-era Query patterns that need the same kind of review; this commit only gets the model layer to import and construct schemas cleanly under SQLAlchemy 2.0. --- .devcontainer/Dockerfile | 2 +- .github/workflows/ci.yml | 2 +- c2corg_api/ext/colander_ext.py | 3 +- c2corg_api/models/__init__.py | 3 +- c2corg_api/models/area_association.py | 6 ++-- c2corg_api/models/association_views.py | 36 +++++++++++------------ c2corg_api/models/document.py | 2 -- c2corg_api/models/feed.py | 10 ++++--- c2corg_api/models/stoparea.py | 2 +- c2corg_api/models/topo_map_association.py | 6 ++-- docker/Dockerfile | 4 +-- docker/Dockerfile.dev | 4 +-- requirements.txt | 10 +++++-- 13 files changed, 46 insertions(+), 44 deletions(-) diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index 305dff5f4..d1579ade9 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.9-slim-bookworm +FROM python:3.12-slim-bookworm # Avoid prompts from apt ENV DEBIAN_FRONTEND=noninteractive diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4c8f396d1..70823b40f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,7 +44,7 @@ jobs: uses: actions/checkout@v6 - uses: actions/setup-python@v6 with: - python-version: 3.9 + python-version: "3.12" - name: Cache pip uses: actions/cache@v4.3.0 with: diff --git a/c2corg_api/ext/colander_ext.py b/c2corg_api/ext/colander_ext.py index 767401eac..e592443ac 100644 --- a/c2corg_api/ext/colander_ext.py +++ b/c2corg_api/ext/colander_ext.py @@ -5,7 +5,6 @@ from geoalchemy2 import WKBElement from geomet import wkb -from geoalchemy2.compat import buffer, bytes import geojson from numbers import Number @@ -87,7 +86,7 @@ def geojson_from_wkbelement(wkb_element): def from_wkb(wkb, srid=-1): - return WKBElement(buffer(wkb), srid=srid) + return WKBElement(memoryview(wkb), srid=srid) def is_valid_geometry(obj): diff --git a/c2corg_api/models/__init__.py b/c2corg_api/models/__init__.py index e6cb04924..890bb82ec 100644 --- a/c2corg_api/models/__init__.py +++ b/c2corg_api/models/__init__.py @@ -1,6 +1,5 @@ -from sqlalchemy.ext.declarative import declarative_base - from sqlalchemy.orm import ( + declarative_base, scoped_session, sessionmaker, ) diff --git a/c2corg_api/models/area_association.py b/c2corg_api/models/area_association.py index baa4ae858..cfa40c211 100644 --- a/c2corg_api/models/area_association.py +++ b/c2corg_api/models/area_association.py @@ -66,7 +66,7 @@ def update_area(area, reset=False): # ignore forwarded areas return - area_geom = select([DocumentGeometry.geom_detail]). \ + area_geom = select(DocumentGeometry.geom_detail). \ where(DocumentGeometry.document_id == area.document_id) intersecting_documents = DBSession. \ query( @@ -111,9 +111,9 @@ def update_areas_for_document(document, reset=False): # ignore forwarded areas return - document_geom = select([DocumentGeometry.geom]). \ + document_geom = select(DocumentGeometry.geom). \ where(DocumentGeometry.document_id == document.document_id) - document_geom_detail = select([DocumentGeometry.geom_detail]). \ + document_geom_detail = select(DocumentGeometry.geom_detail). \ where(DocumentGeometry.document_id == document.document_id) intersecting_areas = DBSession. \ query( diff --git a/c2corg_api/models/association_views.py b/c2corg_api/models/association_views.py index d2fea4ec9..cd833f737 100644 --- a/c2corg_api/models/association_views.py +++ b/c2corg_api/models/association_views.py @@ -30,10 +30,10 @@ def _get_select_waypoints_for_routes(): route_type = text('\'' + ROUTE_TYPE + '\'') select_linked_waypoints = \ - select([ + select( Association.child_document_id.label('route_id'), Association.parent_document_id.label('waypoint_id') - ]). \ + ). \ where( and_( Association.parent_document_type == waypoint_type, @@ -41,10 +41,10 @@ def _get_select_waypoints_for_routes(): cte('linked_waypoints') select_waypoint_parents = \ - select([ + select( select_linked_waypoints.c.route_id, Association.parent_document_id.label('waypoint_id') - ]). \ + ). \ select_from(join( select_linked_waypoints, Association, @@ -56,10 +56,10 @@ def _get_select_waypoints_for_routes(): cte('waypoint_parents') select_waypoint_grandparents = \ - select([ + select( select_waypoint_parents.c.route_id, Association.parent_document_id.label('waypoint_id') - ]). \ + ). \ select_from(join( select_waypoint_parents, Association, @@ -89,12 +89,12 @@ def _get_select_waypoints_for_routes_aggregated(): """ all_waypoints = _get_select_waypoints_for_routes() return \ - select([ + select( all_waypoints.c.route_id.label('route_id'), func.array_agg( all_waypoints.c.waypoint_id, type_=postgresql.ARRAY(Integer)).label('waypoint_ids') - ]). \ + ). \ select_from(all_waypoints). \ group_by(all_waypoints.c.route_id) @@ -135,10 +135,10 @@ def _get_select_waypoints_for_outings_aggregated(): route_type = text('\'' + ROUTE_TYPE + '\'') all_waypoints_for_routes = _get_select_waypoints_for_routes() waypoints_for_outings = \ - select([ + select( Association.child_document_id.label('outing_id'), all_waypoints_for_routes.c.waypoint_id - ]). \ + ). \ select_from(join( Association, all_waypoints_for_routes, @@ -150,12 +150,12 @@ def _get_select_waypoints_for_outings_aggregated(): ))). \ cte('waypoints_for_outings') return \ - select([ + select( waypoints_for_outings.c.outing_id.label('outing_id'), func.array_agg( waypoints_for_outings.c.waypoint_id, type_=postgresql.ARRAY(Integer)).label('waypoint_ids') - ]). \ + ). \ select_from(waypoints_for_outings). \ group_by(waypoints_for_outings.c.outing_id) @@ -189,12 +189,12 @@ def _get_select_users_for_outings_aggregated(): outing_type = text('\'' + OUTING_TYPE + '\'') user_type = text('\'' + USERPROFILE_TYPE + '\'') return \ - select([ + select( Association.child_document_id.label('outing_id'), func.array_agg( Association.parent_document_id, type_=postgresql.ARRAY(Integer)).label('user_ids') - ]). \ + ). \ select_from(Association). \ where(and_( Association.parent_document_type == user_type, @@ -232,12 +232,12 @@ def _get_select_routes_for_outings_aggregated(): outing_type = text('\'' + OUTING_TYPE + '\'') route_type = text('\'' + ROUTE_TYPE + '\'') return \ - select([ + select( Association.child_document_id.label('outing_id'), func.array_agg( Association.parent_document_id, type_=postgresql.ARRAY(Integer)).label('route_ids') - ]). \ + ). \ select_from(Association). \ where(and_( Association.parent_document_type == route_type, @@ -273,12 +273,12 @@ def _get_select_users_for_routes_aggregated(): associated users. """ return \ - select([ + select( DocumentTag.document_id.label('route_id'), func.array_agg( DocumentTag.user_id, type_=postgresql.ARRAY(Integer)).label('user_ids') - ]). \ + ). \ select_from(DocumentTag). \ group_by(DocumentTag.document_id) diff --git a/c2corg_api/models/document.py b/c2corg_api/models/document.py index a9754d777..581593d5d 100644 --- a/c2corg_api/models/document.py +++ b/c2corg_api/models/document.py @@ -295,7 +295,6 @@ def geom(self): return Column( Geometry( geometry_type='POINT', srid=3857, dimension=2, - management=True, spatial_index=self.__name__ != 'ArchiveDocumentGeometry'), info={ 'colanderalchemy': { @@ -309,7 +308,6 @@ def geom_detail(self): return Column( Geometry( geometry_type='GEOMETRY', srid=3857, - management=True, use_typmod=False, spatial_index=self.__name__ != 'ArchiveDocumentGeometry'), info={ diff --git a/c2corg_api/models/feed.py b/c2corg_api/models/feed.py index 941d4b50e..cbef57b81 100644 --- a/c2corg_api/models/feed.py +++ b/c2corg_api/models/feed.py @@ -56,9 +56,10 @@ class FilterArea(Base): User.has_area_filter = column_property( - select([func.count(FilterArea.area_id) > 0]). + select(func.count(FilterArea.area_id) > 0). where(FilterArea.user_id == User.id). - correlate_except(FilterArea), + correlate_except(FilterArea). + scalar_subquery(), deferred=True ) User.feed_filter_areas = relationship('Area', secondary=FilterArea.__table__) @@ -96,9 +97,10 @@ class FollowedUser(Base): User.is_following_users = column_property( - select([func.count(FollowedUser.followed_user_id) > 0]). + select(func.count(FollowedUser.followed_user_id) > 0). where(FollowedUser.follower_user_id == User.id). - correlate_except(FollowedUser), + correlate_except(FollowedUser). + scalar_subquery(), deferred=True ) diff --git a/c2corg_api/models/stoparea.py b/c2corg_api/models/stoparea.py index 6b238726e..db8b7a454 100644 --- a/c2corg_api/models/stoparea.py +++ b/c2corg_api/models/stoparea.py @@ -14,7 +14,7 @@ class _StopareaMixin(object): stoparea_name = Column(String, nullable=False) line = Column(String, nullable=False) operator = Column(String, nullable=False) - geom = Column(Geometry('POINT', srid=4326, management=True)) + geom = Column(Geometry('POINT', srid=4326)) attributes = ['navitia_id', 'stoparea_name', 'line', 'operator', 'geom'] diff --git a/c2corg_api/models/topo_map_association.py b/c2corg_api/models/topo_map_association.py index 4bfcd6c23..d400a61e8 100644 --- a/c2corg_api/models/topo_map_association.py +++ b/c2corg_api/models/topo_map_association.py @@ -66,7 +66,7 @@ def update_map(topo_map, reset=False): # ignore forwarded maps return - map_geom = select([DocumentGeometry.geom_detail]). \ + map_geom = select(DocumentGeometry.geom_detail). \ where(DocumentGeometry.document_id == topo_map.document_id) intersecting_documents = DBSession. \ query( @@ -111,9 +111,9 @@ def update_maps_for_document(document, reset=False): # ignore forwarded maps return - document_geom = select([DocumentGeometry.geom]). \ + document_geom = select(DocumentGeometry.geom). \ where(DocumentGeometry.document_id == document.document_id) - document_geom_detail = select([DocumentGeometry.geom_detail]). \ + document_geom_detail = select(DocumentGeometry.geom_detail). \ where(DocumentGeometry.document_id == document.document_id) intersecting_maps = DBSession. \ query( diff --git a/docker/Dockerfile b/docker/Dockerfile index 8025a757a..e2b2391a5 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,4 +1,4 @@ -FROM docker.io/python:3.9-slim-bullseye +FROM docker.io/python:3.12-slim-bookworm ENV DEBIAN_FRONTEND=noninteractive ENV LC_ALL=en_US.UTF-8 @@ -15,7 +15,7 @@ RUN set -x \ RUN set -x \ && apt-get -y --no-install-recommends install \ git \ - libffi7 \ + libffi8 \ libgeos-c1v5 \ libpq5 diff --git a/docker/Dockerfile.dev b/docker/Dockerfile.dev index e44342a7e..4bf003e8e 100644 --- a/docker/Dockerfile.dev +++ b/docker/Dockerfile.dev @@ -1,4 +1,4 @@ -FROM docker.io/python:3.9-slim-bullseye +FROM docker.io/python:3.12-slim-bookworm ENV DEBIAN_FRONTEND=noninteractive ENV LC_ALL=en_US.UTF-8 @@ -15,7 +15,7 @@ RUN set -x \ RUN set -x \ && apt-get -y --no-install-recommends install \ git \ - libffi7 \ + libffi8 \ libgeos-c1v5 \ libpq5 RUN set -x \ diff --git a/requirements.txt b/requirements.txt index 32b593c46..e79f341b8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,7 +6,7 @@ colander==2.0 dogpile.cache==1.3.4 elasticsearch==8.19.3 elasticsearch-dsl==8.18.0 -geoalchemy2==0.4.2 +geoalchemy2==0.20.0 geojson==3.2.0 geomet==1.1.0 kombu==5.6.2 @@ -27,7 +27,7 @@ redis==5.2.1 requests==2.32.3 setuptools==78.1.1 Shapely==2.0.7 -SQLAlchemy==1.3.24 +SQLAlchemy==2.0.51 transaction==5.0 waitress==3.0.2 zope.sqlalchemy==3.1 @@ -37,7 +37,11 @@ pytz==2025.2 pycurl==7.45.6 # needs: https://github.com/stefanofontanelli/ColanderAlchemy/pull/90 + 91 -ColanderAlchemy @ git+https://github.com/c2corg/ColanderAlchemy.git@v0.3.4+c2corg.1 +# TEMPORARY: points at a local, unpublished patch (branch +# sqlalchemy-2.0-support) that relaxes the SQLAlchemy pin to >=1.4 for +# SQLAlchemy 2.0 support. Needs write access to c2corg/ColanderAlchemy +# to publish properly; replace with the real git URL once pushed. +ColanderAlchemy @ file:///d:/Yoann/workspace/c2c/ColanderAlchemy # needs: https://github.com/mozilla-services/cornice/pull/359 cornice @ git+https://github.com/c2corg/cornice.git@6.1.0+c2corg From e2737e892f38f98184a2d2d3afc74a63a2a54738 Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Mon, 20 Jul 2026 22:35:36 +0200 Subject: [PATCH 50/53] Fix remaining SQLAlchemy 2.0 runtime incompatibilities in views/scripts Completes the Python 3.12 / SQLAlchemy 2.0 migration for c2corg_api/views and c2corg_api/scripts, found empirically by running the full test suite: - sqlalchemy.orm.util no longer exports with_polymorphic/aliased (moved to sqlalchemy.orm); sqlalchemy.sql.elements.not_ moved to top-level sqlalchemy. - String-based loader options (load_only, joinedload, contains_eager, undefer) are rejected outright in 2.0; switched to class-bound attributes across document_listings.py, document.py, document_changes.py, document_history.py, document_version.py, user_preferences.py, feed.py and scripts/es/sync.py. Route.locales needed .of_type(RouteLocale) when load_only-ing polymorphic locale columns. - Query.from_self() was removed in 2.0; rewrote the windowed_query/ column_windows pagination helper in models/utils.py using an explicit subquery. - collections.Mapping removed in Python 3.10+; use collections.abc.Mapping. - SQLAlchemy 1.4+ no longer implicitly adds order_by() columns to the SELECT list under .distinct(), causing InvalidColumnReference; fixed in waypoint.py/route.py's set_recent_outings and generalized get_first_column() to unpack Row objects positionally. - Several single-column query results were used directly as scalar ids without unpacking (silently worked in 1.3, "can't adapt type 'Row'" in 2.0); fixed in document_delete.py and scripts/es/sync.py. - geoalchemy2 0.20 routes use_typmod=False columns (needed on geom_detail to store mixed 2D/3D/4D geometries) through its legacy "managed" AddGeometryColumn() codepath, whose automatic spatial index creation collides with alembic's own index creation. Set spatial_index=False on that column in the init migration and create the index explicitly. - Leftover legacy select([...]) list-syntax + .as_scalar() in models/feed.py, missed by the earlier SQLAlchemy 2.0 select() sweep. - Document.update() mixed transient and persistent locale objects in the same loop without their FK set yet, letting an unrelated expired- attribute access trigger a premature autoflush that inserted a locale row before its document_id was set. Wrapped in DBSession.no_autoflush. Full pytest suite: 956 passed, 2 failed (958 total). The two failures are a PostGIS-side loss of the M coordinate on 4D geometry round-trips, reproducible only against the pg18 sandbox (PostGIS 3.6) used for the separate, not-yet-started database migration - not caused by this change. --- .../versions/2c90b2e5ca7e_add_chinese.py | 2 +- .../versions/305b064bdf66_add_slovenian.py | 2 +- .../versions/38df9393c9a9_init.py | 17 ++++++++--- ...6d557f_create_stops_and_waypoints_stops.py | 2 +- c2corg_api/models/document.py | 15 +++++----- c2corg_api/models/feed.py | 17 ++++++----- c2corg_api/models/utils.py | 10 +++---- c2corg_api/scripts/es/sync.py | 17 ++++++----- c2corg_api/views/__init__.py | 2 +- c2corg_api/views/document.py | 18 ++++++------ c2corg_api/views/document_associations.py | 2 +- c2corg_api/views/document_changes.py | 11 ++++---- c2corg_api/views/document_delete.py | 15 ++++++---- c2corg_api/views/document_history.py | 6 ++-- c2corg_api/views/document_info.py | 2 +- c2corg_api/views/document_listings.py | 28 ++++++++++++------- c2corg_api/views/document_merge.py | 2 +- c2corg_api/views/document_revert.py | 4 +-- c2corg_api/views/document_version.py | 6 ++-- c2corg_api/views/feed.py | 4 +-- c2corg_api/views/route.py | 2 +- c2corg_api/views/user_preferences.py | 7 +++-- c2corg_api/views/waypoint.py | 6 ++-- 23 files changed, 115 insertions(+), 82 deletions(-) diff --git a/alembic_migration/versions/2c90b2e5ca7e_add_chinese.py b/alembic_migration/versions/2c90b2e5ca7e_add_chinese.py index dba9e3f89..b3a501818 100644 --- a/alembic_migration/versions/2c90b2e5ca7e_add_chinese.py +++ b/alembic_migration/versions/2c90b2e5ca7e_add_chinese.py @@ -48,7 +48,7 @@ def upgrade(): # if there is some value in the table. If yes, it's not a test DB and we have to complete them. conn = op.get_bind() - res = conn.execute("SELECT count(1) FROM guidebook.langs").fetchall() + res = conn.execute(sa.text("SELECT count(1) FROM guidebook.langs")).fetchall() if res[0][0] != 0: op.execute("INSERT INTO guidebook.langs VALUES ('zh');") diff --git a/alembic_migration/versions/305b064bdf66_add_slovenian.py b/alembic_migration/versions/305b064bdf66_add_slovenian.py index 389aa62aa..cf66ef4b7 100644 --- a/alembic_migration/versions/305b064bdf66_add_slovenian.py +++ b/alembic_migration/versions/305b064bdf66_add_slovenian.py @@ -21,7 +21,7 @@ def upgrade(): conn = op.get_bind() - res = conn.execute("SELECT count(1) FROM guidebook.langs").fetchall() + res = conn.execute(sa.text("SELECT count(1) FROM guidebook.langs")).fetchall() if res[0][0] != 0: op.execute("INSERT INTO guidebook.langs VALUES ('sl');") diff --git a/alembic_migration/versions/38df9393c9a9_init.py b/alembic_migration/versions/38df9393c9a9_init.py index 6b5907e27..be7eba281 100644 --- a/alembic_migration/versions/38df9393c9a9_init.py +++ b/alembic_migration/versions/38df9393c9a9_init.py @@ -702,18 +702,27 @@ def upgrade(): op.create_table('documents_geometries', sa.Column('version', sa.Integer(), nullable=False), sa.Column('document_id', sa.Integer(), nullable=False), - sa.Column('geom', geoalchemy2.types.Geometry(geometry_type='POINT', srid=3857, management=True), nullable=True), - sa.Column('geom_detail', geoalchemy2.types.Geometry(srid=3857, management=True, use_typmod=False), nullable=True), + sa.Column('geom', geoalchemy2.types.Geometry(geometry_type='POINT', srid=3857), nullable=True), + sa.Column('geom_detail', geoalchemy2.types.Geometry(srid=3857, use_typmod=False, spatial_index=False), nullable=True), sa.ForeignKeyConstraint(['document_id'], ['guidebook.documents.document_id'], ), sa.PrimaryKeyConstraint('document_id'), schema='guidebook' ) + # geom_detail uses use_typmod=False (see below, dimension constraint is dropped to allow + # mixed 2D/3D/4D geometries), which routes it through geoalchemy2's legacy "managed" + # AddGeometryColumn() codepath. That codepath's automatic spatial-index creation collides + # with alembic's own index creation (the Index object re-attaches itself to the table), + # raising a DuplicateTable error - so spatial_index is disabled above and the index is + # created explicitly here instead. + op.create_index( + 'idx_documents_geometries_geom_detail', 'documents_geometries', ['geom_detail'], + unique=False, schema='guidebook', postgresql_using='gist') op.create_table('documents_geometries_archives', sa.Column('version', sa.Integer(), nullable=False), sa.Column('id', sa.Integer(), nullable=False), sa.Column('document_id', sa.Integer(), nullable=False), - sa.Column('geom', geoalchemy2.types.Geometry(geometry_type='POINT', srid=3857, spatial_index=False, management=True), nullable=True), - sa.Column('geom_detail', geoalchemy2.types.Geometry(srid=3857, spatial_index=False, management=True, use_typmod=False), nullable=True), + sa.Column('geom', geoalchemy2.types.Geometry(geometry_type='POINT', srid=3857, spatial_index=False), nullable=True), + sa.Column('geom_detail', geoalchemy2.types.Geometry(srid=3857, spatial_index=False, use_typmod=False), nullable=True), sa.ForeignKeyConstraint(['document_id'], ['guidebook.documents.document_id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('version', 'document_id', name='uq_documents_geometries_archives_document_id_version_lang'), diff --git a/alembic_migration/versions/bb61456d557f_create_stops_and_waypoints_stops.py b/alembic_migration/versions/bb61456d557f_create_stops_and_waypoints_stops.py index 8489b32d7..751f6e902 100644 --- a/alembic_migration/versions/bb61456d557f_create_stops_and_waypoints_stops.py +++ b/alembic_migration/versions/bb61456d557f_create_stops_and_waypoints_stops.py @@ -23,7 +23,7 @@ def upgrade(): sa.Column('stoparea_name', sa.String(), nullable=False), sa.Column('line', sa.String(), nullable=False), sa.Column('operator', sa.String(), nullable=False), - sa.Column('geom', geoalchemy2.types.Geometry(geometry_type='POINT', srid=3857, management=True), nullable=True), + sa.Column('geom', geoalchemy2.types.Geometry(geometry_type='POINT', srid=3857), nullable=True), schema='guidebook' ) diff --git a/c2corg_api/models/document.py b/c2corg_api/models/document.py index 581593d5d..fdc23dccf 100644 --- a/c2corg_api/models/document.py +++ b/c2corg_api/models/document.py @@ -118,13 +118,14 @@ def update(self, other): """ copy_attributes(other, self, Document._ATTRIBUTES_WHITELISTED) - for locale_in in other.locales: - locale = self.get_locale(locale_in.lang) - if locale: - locale.update(locale_in) - locale.document_id = self.document_id - else: - self.locales.append(locale_in) + with DBSession.no_autoflush: + for locale_in in other.locales: + locale = self.get_locale(locale_in.lang) + if locale: + locale.update(locale_in) + locale.document_id = self.document_id + else: + self.locales.append(locale_in) if other.geometry: if self.geometry: diff --git a/c2corg_api/models/feed.py b/c2corg_api/models/feed.py index cbef57b81..7542f3bd3 100644 --- a/c2corg_api/models/feed.py +++ b/c2corg_api/models/feed.py @@ -436,20 +436,19 @@ def update_areas_of_changes(document): """Update the area ids of all feed entries of the given document. """ areas_select = select( - [ - # concatenate with empty array to avoid null values - # select ARRAY[]::integer[] || array_agg(area_id) - literal_column('ARRAY[]::integer[]').op('||')( - func.array_agg( - AreaAssociation.area_id, - type_=postgresql.ARRAY(Integer))) - ]).\ + # concatenate with empty array to avoid null values + # select ARRAY[]::integer[] || array_agg(area_id) + literal_column('ARRAY[]::integer[]').op('||')( + func.array_agg( + AreaAssociation.area_id, + type_=postgresql.ARRAY(Integer))) + ).\ where(AreaAssociation.document_id == document.document_id) DBSession.execute( DocumentChange.__table__.update(). where(DocumentChange.document_id == document.document_id). - values(area_ids=areas_select.as_scalar()) + values(area_ids=areas_select.scalar_subquery()) ) diff --git a/c2corg_api/models/utils.py b/c2corg_api/models/utils.py index 6741b82fb..3868b3702 100644 --- a/c2corg_api/models/utils.py +++ b/c2corg_api/models/utils.py @@ -151,13 +151,13 @@ def int_for_range(start_id, end_id): else: return column >= start_id - q = session.query( - column, + subq = session.query( + column.label('_window_col'), func.row_number().over(order_by=column).label('rownum') - ). \ - from_self(column) + ).subquery() + q = session.query(subq.c._window_col) if windowsize > 1: - q = q.filter(sa.text("rownum %% %d=1" % windowsize)) + q = q.filter(subq.c.rownum % windowsize == 1) intervals = [id for id, in q] diff --git a/c2corg_api/scripts/es/sync.py b/c2corg_api/scripts/es/sync.py index e4da5181f..535da6fa2 100644 --- a/c2corg_api/scripts/es/sync.py +++ b/c2corg_api/scripts/es/sync.py @@ -1,6 +1,8 @@ from c2corg_api.models import es_sync, document_types, document_locale_types from c2corg_api.models.area import Area from c2corg_api.models.association import AssociationLog, Association +from c2corg_api.models.association_views import WaypointsForRoutesView, \ + WaypointsForOutingsView from c2corg_api.models.document import Document, DocumentGeometry from c2corg_api.models.document_history import DocumentVersion, HistoryMetaData from c2corg_api.models.document_tag import DocumentTagLog @@ -319,8 +321,9 @@ def add_routes_for_waypoints(session, docs_per_type): if not changed_waypoint_ids: return - linked_route_ids = session.query(Route.document_id). \ - filter(Route.main_waypoint_id.in_(changed_waypoint_ids)).all() + linked_route_ids = [ + row[0] for row in session.query(Route.document_id). + filter(Route.main_waypoint_id.in_(changed_waypoint_ids)).all()] route_ids = docs_per_type.setdefault(ROUTE_TYPE, set()) route_ids.update(linked_route_ids) @@ -345,9 +348,9 @@ def get_documents(session, doc_type, batch_size, document_ids=None, if document_ids: base_query = base_query.filter(clazz.document_id.in_(document_ids)) - locale_fields = ['title'] + locale_fields = [locales_clazz.title] if clazz == Route: - locale_fields.append('title_prefix') + locale_fields.append(locales_clazz.title_prefix) base_query = base_query. \ options(joinedload(clazz.locales.of_type(locales_clazz)). @@ -356,19 +359,19 @@ def get_documents(session, doc_type, batch_size, document_ids=None, if clazz != Area: base_query = base_query. \ - options(joinedload(clazz._areas).load_only('document_id')) + options(joinedload(clazz._areas).load_only(Area.document_id)) if clazz == Route: base_query = base_query. \ options( joinedload(Route.associated_waypoints_ids). - load_only('waypoint_ids')) + load_only(WaypointsForRoutesView.waypoint_ids)) if clazz == Outing: base_query = base_query. \ options( joinedload(Outing.associated_waypoints_ids). - load_only('waypoint_ids')) + load_only(WaypointsForOutingsView.waypoint_ids)) base_query = add_load_for_profiles(base_query, clazz) diff --git a/c2corg_api/views/__init__.py b/c2corg_api/views/__init__.py index 666617614..86360abdd 100644 --- a/c2corg_api/views/__init__.py +++ b/c2corg_api/views/__init__.py @@ -156,7 +156,7 @@ def serialize(data): """ if isinstance(data, str): return str(data) - if isinstance(data, collections.Mapping): + if isinstance(data, collections.abc.Mapping): return dict(list(map(serialize, iter(data.items())))) if isinstance(data, collections.abc.Iterable): return type(data)(list(map(serialize, data))) diff --git a/c2corg_api/views/document.py b/c2corg_api/views/document.py index d58487041..17e9e3ba0 100644 --- a/c2corg_api/views/document.py +++ b/c2corg_api/views/document.py @@ -13,8 +13,9 @@ from c2corg_api.models.cache_version import update_cache_version, \ update_cache_version_associations, get_cache_key from c2corg_api.models.document import ( - UpdateType, DocumentLocale, ArchiveDocumentLocale, ArchiveDocument, - ArchiveDocumentGeometry, set_available_langs, get_available_langs) + Document, UpdateType, DocumentLocale, ArchiveDocumentLocale, + ArchiveDocument, ArchiveDocumentGeometry, set_available_langs, + get_available_langs) from c2corg_api.models.document_history import HistoryMetaData, DocumentVersion from c2corg_api.models.feed import update_feed_document_create, \ update_feed_document_update @@ -41,7 +42,7 @@ HTTPBadRequest, HTTPForbidden from sqlalchemy.orm import joinedload, contains_eager, load_only from sqlalchemy.orm.exc import StaleDataError -from sqlalchemy.orm.util import with_polymorphic +from sqlalchemy.orm import with_polymorphic log = logging.getLogger(__name__) @@ -95,7 +96,8 @@ def _search_documents_paginated( offset = meta_params['offset'] limit = meta_params['limit'] documents = base_query. \ - options(load_only('document_id', 'type', 'version')). \ + options(load_only( + Document.document_id, Document.type, Document.version)). \ slice(offset, offset + limit). \ limit(limit). \ all() @@ -303,7 +305,7 @@ def _get_document(self, clazz, id, clazz_locale=None, lang=None): document_query = DBSession. \ query(clazz). \ filter(getattr(clazz, 'document_id') == id). \ - options(joinedload('geometry')) + options(joinedload(getattr(clazz, 'geometry'))) document_query = add_load_for_locales( document_query, clazz, clazz_locale) document_query = add_load_for_profiles(document_query, clazz) @@ -320,7 +322,7 @@ def _get_document(self, clazz, id, clazz_locale=None, lang=None): join(locales_type). \ filter(getattr(clazz, 'document_id') == id). \ filter(DocumentLocale.lang == lang). \ - options(joinedload('geometry')).\ + options(joinedload(getattr(clazz, 'geometry'))).\ options(contains_eager(locales_type_eager, alias=locales_type)) document_query = add_load_for_profiles(document_query, clazz) document = document_query.first() @@ -331,7 +333,7 @@ def _get_document(self, clazz, id, clazz_locale=None, lang=None): document_query = DBSession. \ query(clazz). \ filter(getattr(clazz, 'document_id') == id). \ - options(joinedload('geometry')) + options(joinedload(getattr(clazz, 'geometry'))) document_query = add_load_for_profiles(document_query, clazz) document = document_query.first() @@ -358,7 +360,7 @@ def _get_document_for_cooking(clazz, document_id, lang, clazz_locale=None): document_query = DBSession. \ query(clazz). \ filter(getattr(clazz, 'document_id') == document_id). \ - options(joinedload('geometry')) + options(joinedload(getattr(clazz, 'geometry'))) document_query = add_load_for_locales( document_query, clazz, clazz_locale) document_query = add_load_for_profiles(document_query, clazz) diff --git a/c2corg_api/views/document_associations.py b/c2corg_api/views/document_associations.py index 5e61daeca..31904d50a 100644 --- a/c2corg_api/views/document_associations.py +++ b/c2corg_api/views/document_associations.py @@ -302,4 +302,4 @@ def get_linked_xreports(document, lang): def get_first_column(rows): - return [r for (r, ) in rows] + return [row[0] for row in rows] diff --git a/c2corg_api/views/document_changes.py b/c2corg_api/views/document_changes.py index 8104cff65..166f0eee8 100644 --- a/c2corg_api/views/document_changes.py +++ b/c2corg_api/views/document_changes.py @@ -79,23 +79,24 @@ def load_feed(doc_ids, limit, user_id=None): doc_changes = [] else: doc_changes = DBSession.query(DocumentVersion) \ - .options(load_only('lang'), - joinedload('history_metadata').load_only( + .options(load_only(DocumentVersion.lang), + joinedload(DocumentVersion.history_metadata).load_only( HistoryMetaData.id, HistoryMetaData.user_id, HistoryMetaData.comment, HistoryMetaData.written_at). - joinedload('user').load_only( + joinedload(HistoryMetaData.user).load_only( User.id, User.name, User.username, User.lang)) \ - .options(joinedload('document').load_only( + .options(joinedload(DocumentVersion.document).load_only( Document.version, Document.document_id, Document.type, Document.quality)) \ - .options(joinedload('document_locales_archive').load_only( + .options(joinedload( + DocumentVersion.document_locales_archive).load_only( ArchiveDocumentLocale.title)) \ .order_by(desc(DocumentVersion.id)) \ .filter(DocumentVersion.history_metadata_id.in_(doc_ids)) \ diff --git a/c2corg_api/views/document_delete.py b/c2corg_api/views/document_delete.py index a4328c23d..9ff96a2f8 100644 --- a/c2corg_api/views/document_delete.py +++ b/c2corg_api/views/document_delete.py @@ -254,8 +254,10 @@ def _delete_document(self, document_id, document_type, redirecting=False): archive_clazz, archive_clazz_locale) def _remove_merged_documents(self, document_id, document_type): - merged_document_ids = DBSession.query(ArchiveDocument.document_id). \ - filter(ArchiveDocument.redirects_to == document_id).all() + merged_document_ids = [ + row[0] for row in DBSession. + query(ArchiveDocument.document_id). + filter(ArchiveDocument.redirects_to == document_id).all()] for merged_document_id in merged_document_ids: self._delete_document(merged_document_id, document_type, True) @@ -367,10 +369,11 @@ def remove_from_cache(document_id): def _remove_versions(document_id): - history_metadata_ids = DBSession. \ - query(DocumentVersion.history_metadata_id). \ - filter(DocumentVersion.document_id == document_id). \ - all() + history_metadata_ids = [row[0] for row in DBSession. + query(DocumentVersion.history_metadata_id). + filter( + DocumentVersion.document_id == document_id). + all()] DBSession.query(DocumentVersion). \ filter(DocumentVersion.document_id == document_id). \ delete() diff --git a/c2corg_api/views/document_history.py b/c2corg_api/views/document_history.py index b67394d72..0f5790735 100644 --- a/c2corg_api/views/document_history.py +++ b/c2corg_api/views/document_history.py @@ -2,7 +2,8 @@ from c2corg_api.models import DBSession from c2corg_api.models.cache_version import get_cache_key from c2corg_api.models.document import DocumentLocale, DOCUMENT_TYPE -from c2corg_api.models.document_history import DocumentVersion +from c2corg_api.models.document_history import DocumentVersion, \ + HistoryMetaData from c2corg_api.views import cors_policy, etag_cache from c2corg_api.views.document_version import serialize_version from c2corg_api.views.validation import validate_lang, validate_id @@ -62,7 +63,8 @@ def _get_history(self, document_id, lang): raise HTTPNotFound('no locale document for "{0}"'.format(lang)) versions = DBSession.query(DocumentVersion) \ - .options(joinedload('history_metadata').joinedload('user')) \ + .options(joinedload(DocumentVersion.history_metadata). + joinedload(HistoryMetaData.user)) \ .filter(DocumentVersion.document_id == document_id) \ .filter(DocumentVersion.lang == lang) \ .order_by(DocumentVersion.id) \ diff --git a/c2corg_api/views/document_info.py b/c2corg_api/views/document_info.py index a818c71b0..174677d82 100644 --- a/c2corg_api/views/document_info.py +++ b/c2corg_api/views/document_info.py @@ -11,7 +11,7 @@ from c2corg_api.caching import get_or_create from pyramid.httpexceptions import HTTPNotFound from sqlalchemy.orm import contains_eager, load_only, joinedload -from sqlalchemy.orm.util import with_polymorphic +from sqlalchemy.orm import with_polymorphic class DocumentInfoRest(object): diff --git a/c2corg_api/views/document_listings.py b/c2corg_api/views/document_listings.py index 2c8b63f98..11f4ddc3d 100644 --- a/c2corg_api/views/document_listings.py +++ b/c2corg_api/views/document_listings.py @@ -1,11 +1,11 @@ from c2corg_api.caching import cache_document_listing from c2corg_api.models import DBSession -from c2corg_api.models.area import schema_listing_area +from c2corg_api.models.area import Area, schema_listing_area from c2corg_api.models.association import Association from c2corg_api.models.cache_version import get_document_id, \ get_cache_keys from c2corg_api.models.document import ( - set_available_langs) + DocumentGeometry, DocumentLocale, set_available_langs) from c2corg_api.models.image import IMAGE_TYPE from c2corg_api.models.outing import Outing from c2corg_api.models.user import User @@ -89,7 +89,9 @@ def _get_documents_from_ids( loaded, and the list has the same order has the document id list. """ base_query = base_query.options( - load_only(*documents_config.get_load_only_fields()) + load_only(*[ + getattr(documents_config.clazz, field) + for field in documents_config.get_load_only_fields()]) ) base_query = add_load_for_locales( base_query, documents_config.clazz, documents_config.clazz_locale, @@ -100,7 +102,9 @@ def _get_documents_from_ids( # 'version' base_query = base_query.options( joinedload(getattr(documents_config.clazz, 'geometry')). - load_only(*documents_config.get_load_only_fields_geometry()) + load_only(*[ + getattr(DocumentGeometry, field) + for field in documents_config.get_load_only_fields_geometry()]) ) if documents_config.include_areas: @@ -108,11 +112,12 @@ def _get_documents_from_ids( options( joinedload(getattr(documents_config.clazz, '_areas')). load_only( - 'document_id', 'area_type', 'version', 'protected', - 'type'). - joinedload('locales'). + Area.document_id, Area.area_type, Area.version, + Area.protected, Area.type). + joinedload(Area.locales). load_only( - 'lang', 'title', 'version') + DocumentLocale.lang, DocumentLocale.title, + DocumentLocale.version) ) documents = _load_documents( @@ -165,7 +170,7 @@ def add_load_for_profiles(document_query, clazz): if clazz == UserProfile: # for profiles load names together from the associated user document_query = add_profile_filter(document_query, clazz). \ - options(contains_eager('user').load_only( + options(contains_eager(UserProfile.user).load_only( User.id, User.name, User.forum_username)) return document_query @@ -184,11 +189,14 @@ def add_load_for_locales( if clazz_locale: locales_load = subqueryload( getattr(clazz, 'locales').of_type(clazz_locale)) + locale_clazz = clazz_locale else: locales_load = subqueryload(getattr(clazz, 'locales')) + locale_clazz = DocumentLocale if load_only_fields: - locales_load = locales_load.load_only(*load_only_fields) + locales_load = locales_load.load_only( + *[getattr(locale_clazz, field) for field in load_only_fields]) return base_query.options(locales_load) diff --git a/c2corg_api/views/document_merge.py b/c2corg_api/views/document_merge.py index 954b71c8b..d45fa7e93 100644 --- a/c2corg_api/views/document_merge.py +++ b/c2corg_api/views/document_merge.py @@ -18,7 +18,7 @@ from colander import MappingSchema, required, SchemaNode, Integer from cornice.resource import resource from cornice.validators import colander_body_validator -from sqlalchemy.sql.elements import not_ +from sqlalchemy import not_ from sqlalchemy.sql.expression import and_, or_ from sqlalchemy.sql.functions import func diff --git a/c2corg_api/views/document_revert.py b/c2corg_api/views/document_revert.py index 7eb0be4ab..523e58b4b 100644 --- a/c2corg_api/views/document_revert.py +++ b/c2corg_api/views/document_revert.py @@ -30,7 +30,7 @@ from cornice.validators import colander_body_validator from pyramid.httpexceptions import HTTPBadRequest from sqlalchemy.orm import joinedload, contains_eager -from sqlalchemy.orm.util import with_polymorphic +from sqlalchemy.orm import with_polymorphic from sqlalchemy.sql.expression import exists, and_ log = logging.getLogger(__name__) @@ -192,7 +192,7 @@ def _get_current_document(self, document_id, lang, clazz, clazz_locale): join(locales_type). \ filter(getattr(clazz, 'document_id') == document_id). \ filter(DocumentLocale.lang == lang). \ - options(joinedload('geometry')). \ + options(joinedload(getattr(clazz, 'geometry'))). \ options(contains_eager(locales_type_eager, alias=locales_type)) return document_query.first() diff --git a/c2corg_api/views/document_version.py b/c2corg_api/views/document_version.py index 5c5532d97..8da665331 100644 --- a/c2corg_api/views/document_version.py +++ b/c2corg_api/views/document_version.py @@ -2,7 +2,8 @@ from c2corg_api.caching import cache_document_version from c2corg_api.models import DBSession from c2corg_api.models.cache_version import get_cache_key -from c2corg_api.models.document_history import DocumentVersion +from c2corg_api.models.document_history import DocumentVersion, \ + HistoryMetaData from c2corg_api.models.user import User from c2corg_api.views import to_json_dict, etag_cache from c2corg_api.caching import get_or_create @@ -52,7 +53,8 @@ def _load_version( self, document_id, lang, version_id, clazz, locale_clazz, schema, adapt_schema): version = DBSession.query(DocumentVersion) \ - .options(joinedload('history_metadata').joinedload('user'). + .options(joinedload(DocumentVersion.history_metadata). + joinedload(HistoryMetaData.user). load_only(User.id, User.name)) \ .options(joinedload( DocumentVersion.document_archive.of_type(clazz))) \ diff --git a/c2corg_api/views/feed.py b/c2corg_api/views/feed.py index 43423d274..0ab7d2d34 100644 --- a/c2corg_api/views/feed.py +++ b/c2corg_api/views/feed.py @@ -162,8 +162,8 @@ def get_changes_of_personal_feed( user_id, token_id, token_time, limit, ignore_admin_changes_filter): user = DBSession.query(User). \ filter(User.id == user_id). \ - options(undefer('has_area_filter')). \ - options(undefer('is_following_users')). \ + options(undefer(User.has_area_filter)). \ + options(undefer(User.is_following_users)). \ first() if has_no_custom_filter(user): diff --git a/c2corg_api/views/route.py b/c2corg_api/views/route.py index 22f5738d3..f9f7e603f 100644 --- a/c2corg_api/views/route.py +++ b/c2corg_api/views/route.py @@ -148,7 +148,7 @@ def set_recent_outings(route, lang): """Set last 10 outings on the given route. """ recent_outing_ids = get_first_column( - DBSession.query(Outing.document_id). + DBSession.query(Outing.document_id, Outing.date_end). filter(Outing.redirects_to.is_(None)). join( Association, diff --git a/c2corg_api/views/user_preferences.py b/c2corg_api/views/user_preferences.py index 1e20f0f8d..61d6edf5c 100644 --- a/c2corg_api/views/user_preferences.py +++ b/c2corg_api/views/user_preferences.py @@ -1,6 +1,7 @@ from c2corg_api.security.acl import ACLDefault from c2corg_api import DBSession from c2corg_api.models.area import schema_listing_area, Area +from c2corg_api.models.document import DocumentLocale from c2corg_api.models.schema_utils import SchemaAssociationDoc from c2corg_api.models.user import User from c2corg_api.views import cors_policy, restricted_json_view, \ @@ -42,8 +43,10 @@ def get_user(self, with_area_locales=True): if with_area_locales: area_joinedload = area_joinedload. \ - joinedload('locales'). \ - load_only('lang', 'title', 'version') + joinedload(Area.locales). \ + load_only( + DocumentLocale.lang, DocumentLocale.title, + DocumentLocale.version) return DBSession. \ query(User). \ diff --git a/c2corg_api/views/waypoint.py b/c2corg_api/views/waypoint.py index d037fc50d..35e7a57b5 100644 --- a/c2corg_api/views/waypoint.py +++ b/c2corg_api/views/waypoint.py @@ -35,7 +35,7 @@ from c2corg_api.models.common.fields_waypoint import fields_waypoint from c2corg_api.models.common.attributes import waypoint_types from sqlalchemy.orm import joinedload, load_only -from sqlalchemy.orm.util import aliased +from sqlalchemy.orm import aliased from sqlalchemy.sql.elements import literal_column from sqlalchemy.sql.expression import and_, text, union, column from c2corg_api.models.area import Area @@ -266,7 +266,7 @@ def set_recent_outings(waypoint, lang): with_query_waypoints = _get_select_children(waypoint) recent_outing_ids = get_first_column( - DBSession.query(Outing.document_id). + DBSession.query(Outing.document_id, Outing.date_end). filter(Outing.redirects_to.is_(None)). join( t_outing_route, @@ -494,7 +494,7 @@ def update_linked_route_titles(waypoint, update_types, user_id): linked_routes = DBSession.query(Route). \ filter(Route.main_waypoint_id == waypoint.document_id). \ - options(joinedload(Route.locales).load_only( + options(joinedload(Route.locales.of_type(RouteLocale)).load_only( RouteLocale.lang, RouteLocale.id)). \ options(load_only(Route.document_id)). \ all() From aaccdb451ef1ca89faef7322986935bb95583964 Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Mon, 20 Jul 2026 22:52:58 +0200 Subject: [PATCH 51/53] Fix 4D geometry M-coordinate loss and a scalar subquery warning geoalchemy2 binds a non-extended WKBElement by converting it through Shapely, which has no support for M (4th dimension) coordinates and silently drops them on write. wkbelement_from_geojson() now embeds the SRID directly in the WKB header (turning it into real EWKB), which geoalchemy2 detects as already-extended and binds as a hex string directly instead, preserving all dimensions. Fixes the last two failures from the SQLAlchemy 2.0 migration test run (test_ext_colander::test_save_and_load_4d, test_route::test_post_success_4d). Also replace an implicit scalar-subquery coercion in update_langs_of_changes() with an explicit .scalar_subquery() call, as recommended by the SQLAlchemy 2.0 deprecation warning. --- c2corg_api/ext/colander_ext.py | 25 +++++++++++++++++++++++++ c2corg_api/models/feed.py | 2 +- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/c2corg_api/ext/colander_ext.py b/c2corg_api/ext/colander_ext.py index e592443ac..ddc8b2f4f 100644 --- a/c2corg_api/ext/colander_ext.py +++ b/c2corg_api/ext/colander_ext.py @@ -1,6 +1,8 @@ # Inspired from the c2cgeoform colander extension # https://github.com/camptocamp/c2cgeoform/blob/master/c2cgeoform/ext/ +import struct + from colander import (null, Invalid, SchemaType) from geoalchemy2 import WKBElement @@ -8,6 +10,10 @@ import geojson from numbers import Number +# EWKB SRID flag, see https://git.osgeo.org/gitea/postgis/postgis/raw/ +# master/doc/ZMSgeoms.txt +_EWKB_SRID_FLAG = 0x20000000 + class Geometry(SchemaType): """ A Colander type meant to be used with GeoAlchemy 2 geometry columns. @@ -77,9 +83,28 @@ def cstruct_children(self, node, cstruct): def wkbelement_from_geojson(geojson, srid): geometry = wkb.dumps(geojson, big_endian=False) + if srid and srid > 0: + # Embed the SRID directly in the WKB header, turning it into EWKB. + # geoalchemy2 binds a non-extended WKBElement by converting it + # through Shapely, which does not support M (4th dimension) + # coordinates and silently drops them. A WKBElement built from real + # EWKB is detected as already-extended and bound as a hex string + # directly instead, preserving all dimensions. + geometry = _to_ewkb(geometry, srid) return from_wkb(geometry, srid) +def _to_ewkb(wkb_bytes, srid): + byte_order = wkb_bytes[0] + fmt = 'I' + geom_type = struct.unpack(fmt, wkb_bytes[1:5])[0] + ewkb_type = geom_type | _EWKB_SRID_FLAG + return ( + wkb_bytes[0:1] + struct.pack(fmt, ewkb_type) + + struct.pack(fmt, srid) + wkb_bytes[5:] + ) + + def geojson_from_wkbelement(wkb_element): geometry = wkb.loads(bytes(wkb_element.data)) return geojson.dumps(geometry) diff --git a/c2corg_api/models/feed.py b/c2corg_api/models/feed.py index 7542f3bd3..6df3cd7af 100644 --- a/c2corg_api/models/feed.py +++ b/c2corg_api/models/feed.py @@ -475,7 +475,7 @@ def update_langs_of_changes(document_id): DBSession.execute( DocumentChange.__table__.update(). where(DocumentChange.document_id == document_id). - values(langs=langs.select())) + values(langs=langs.select().scalar_subquery())) def get_linked_document(images_in): From 13e685ecd10ef0989573a647d874cf3417e4c86d Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Mon, 20 Jul 2026 23:20:54 +0200 Subject: [PATCH 52/53] fix codacy PR --- c2corg_api/views/document.py | 1 - c2corg_api/views/document_listings.py | 2 +- scripts/esjson/mappings.json | 16 ---------------- scripts/esjson2-5/mappings.json | 16 ---------------- scripts/esjson5-6/mappings.json | 16 ---------------- scripts/esjson6-7/mappings.json | 16 ---------------- scripts/pycurl-example.py | 2 +- 7 files changed, 2 insertions(+), 67 deletions(-) diff --git a/c2corg_api/views/document.py b/c2corg_api/views/document.py index 17e9e3ba0..7712eccd4 100644 --- a/c2corg_api/views/document.py +++ b/c2corg_api/views/document.py @@ -1,5 +1,4 @@ import logging -from pprint import pprint from c2corg_api.security.acl import ACLDefault from c2corg_api.caching import cache_document_detail, cache_document_cooked diff --git a/c2corg_api/views/document_listings.py b/c2corg_api/views/document_listings.py index 11f4ddc3d..031f21e09 100644 --- a/c2corg_api/views/document_listings.py +++ b/c2corg_api/views/document_listings.py @@ -47,7 +47,7 @@ def get_documents(documents_config, meta_params, search_documents): document_ids, total = search_documents(base_query, base_total_query) - if total is not None and type(total) is not int: + if total is not None and not isinstance(total, int): total = int(total['value']) cache_keys = get_cache_keys( diff --git a/scripts/esjson/mappings.json b/scripts/esjson/mappings.json index f4269bfa7..095ae6b78 100644 --- a/scripts/esjson/mappings.json +++ b/scripts/esjson/mappings.json @@ -2020,22 +2020,6 @@ "analyzer": "index_italian", "search_analyzer": "search_italian" }, - "title_ca": { - "type": "string", - "index": "not_analyzed", - "fields": { - "ngram": { - "type": "string", - "analyzer": "index_ngram", - "search_analyzer": "search_ngram" - }, - "raw": { - "type": "string", - "analyzer": "index_raw", - "search_analyzer": "search_raw" - } - } - }, "title_ca": { "type": "string", "index": "not_analyzed", diff --git a/scripts/esjson2-5/mappings.json b/scripts/esjson2-5/mappings.json index 67c9fff5b..4ac3ab7f4 100644 --- a/scripts/esjson2-5/mappings.json +++ b/scripts/esjson2-5/mappings.json @@ -2020,22 +2020,6 @@ "analyzer": "index_italian", "search_analyzer": "search_italian" }, - "title_ca": { - "type": "text", - "index": "not_analyzed", - "fields": { - "ngram": { - "type": "text", - "analyzer": "index_ngram", - "search_analyzer": "search_ngram" - }, - "raw": { - "type": "text", - "analyzer": "index_raw", - "search_analyzer": "search_raw" - } - } - }, "title_ca": { "type": "text", "index": "not_analyzed", diff --git a/scripts/esjson5-6/mappings.json b/scripts/esjson5-6/mappings.json index 67c9fff5b..4ac3ab7f4 100644 --- a/scripts/esjson5-6/mappings.json +++ b/scripts/esjson5-6/mappings.json @@ -2020,22 +2020,6 @@ "analyzer": "index_italian", "search_analyzer": "search_italian" }, - "title_ca": { - "type": "text", - "index": "not_analyzed", - "fields": { - "ngram": { - "type": "text", - "analyzer": "index_ngram", - "search_analyzer": "search_ngram" - }, - "raw": { - "type": "text", - "analyzer": "index_raw", - "search_analyzer": "search_raw" - } - } - }, "title_ca": { "type": "text", "index": "not_analyzed", diff --git a/scripts/esjson6-7/mappings.json b/scripts/esjson6-7/mappings.json index 67c9fff5b..4ac3ab7f4 100644 --- a/scripts/esjson6-7/mappings.json +++ b/scripts/esjson6-7/mappings.json @@ -2020,22 +2020,6 @@ "analyzer": "index_italian", "search_analyzer": "search_italian" }, - "title_ca": { - "type": "text", - "index": "not_analyzed", - "fields": { - "ngram": { - "type": "text", - "analyzer": "index_ngram", - "search_analyzer": "search_ngram" - }, - "raw": { - "type": "text", - "analyzer": "index_raw", - "search_analyzer": "search_raw" - } - } - }, "title_ca": { "type": "text", "index": "not_analyzed", diff --git a/scripts/pycurl-example.py b/scripts/pycurl-example.py index c6979ac96..82a0ed5dc 100755 --- a/scripts/pycurl-example.py +++ b/scripts/pycurl-example.py @@ -1,5 +1,5 @@ #!/usr/bin/python3 -from urllib.parse import urlencode +# from urllib.parse import urlencode import pycurl from io import BytesIO From aab4703f5896c821f98af3e8a0a932c0bebc0f6b Mon Sep 17 00:00:00 2001 From: Nayor <11463011+Nayor@users.noreply.github.com> Date: Fri, 11 Sep 2026 23:35:35 +0200 Subject: [PATCH 53/53] Point ColanderAlchemy dependency at pushed c2corg fork commit requirements.txt referenced ColanderAlchemy via a local file:// path, which only worked on the machine that had the patched checkout. Now that sqlalchemy-2.0-support has push access on c2corg/ColanderAlchemy, pin to that branch's commit via a git URL instead. Co-Authored-By: Claude Sonnet 5 --- requirements.txt | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/requirements.txt b/requirements.txt index e79f341b8..917b456b8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -37,11 +37,8 @@ pytz==2025.2 pycurl==7.45.6 # needs: https://github.com/stefanofontanelli/ColanderAlchemy/pull/90 + 91 -# TEMPORARY: points at a local, unpublished patch (branch -# sqlalchemy-2.0-support) that relaxes the SQLAlchemy pin to >=1.4 for -# SQLAlchemy 2.0 support. Needs write access to c2corg/ColanderAlchemy -# to publish properly; replace with the real git URL once pushed. -ColanderAlchemy @ file:///d:/Yoann/workspace/c2c/ColanderAlchemy +# SQLAlchemy 2.0 support, pushed to c2corg/ColanderAlchemy (branch sqlalchemy-2.0-support). +ColanderAlchemy @ git+https://github.com/c2corg/ColanderAlchemy.git@d2cc04a041d111deac307666e20fc24e47cc62e6 # needs: https://github.com/mozilla-services/cornice/pull/359 cornice @ git+https://github.com/c2corg/cornice.git@6.1.0+c2corg