Existing status.json causes issue when creating document index

[AlexBlueSteele]

[BUG]
When a old status file exists, the document index does not get created. However, the folder index does get created. In the example case below, poc1_folder exists but poc1 does not exist. This was not a bug in the last snapshot.

Job Settings

name: poc1
fs:
  url: 'W:\'
  update_rate: 15m
  indexed_chars: 32768
  lang_detect: true
  continue_on_error: true
  raw_metadata: true
  store_source: false
  remove_deleted: true
  attributes_support: true
elasticsearch:
  urls:
    - https://REDACTED
  ssl_verification: false

Logs
Elasticsearch

[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.593Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_alias]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","pr
ocess.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.clus
ter.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elasticsearch.node
.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docker-cluster"
}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.631Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_settings_total_fields]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticse
arch.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.el
asticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","el
asticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":
"docker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.688Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_mapping_attributes]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearc
h.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elast
icsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elast
icsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"do
cker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.722Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_mapping_file]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.serv
er","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elasticsear
ch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elasticsear
ch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docker-c
luster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.761Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_mapping_path]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.serv
er","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elasticsear
ch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elasticsear
ch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docker-c
luster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.824Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_mapping_attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearc
h.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elast
icsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elast
icsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"do
cker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.863Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_mapping_content]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.s
erver","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elastics
earch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elastics
earch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docke
r-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.899Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_mapping_meta]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.serv
er","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elasticsear
ch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elasticsear
ch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docker-c
luster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.937Z", "log.level": "INFO", "message":"adding index template [f
scrawler_poc1_docs] for index patterns [poc1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elas
ticsearch.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"o
rg.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw
","elasticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.n
ame":"docker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:11.972Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_folder_mapping_attributes]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elast
icsearch.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"or
g.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw"
,"elasticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.na
me":"docker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:12.024Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_folder_mapping_file]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsear
ch.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elas
ticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elas
ticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"d
ocker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:12.059Z", "log.level": "INFO", "message":"adding component templat
e [fscrawler_poc1_folder_mapping_path]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsear
ch.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elas
ticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elas
ticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"d
ocker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:12.096Z", "log.level": "INFO", "message":"adding index template [f
scrawler_poc1_folder_folders] for index patterns [poc1_folder]", "ecs.version": "1.2.0","service.name":"ES_ECS","eve
nt.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]
","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"swUCN
mO4Q5iGG_zWpT9ydw","elasticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasti
csearch.cluster.name":"docker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:16.542Z", "log.level": "INFO", "message":"creating index [poc1_fol
der] in project [default], cause [auto(bulk api)], templates [provided in request], shards [1]/[1]", "ecs.version":
"1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[2ab8ceee
82ab][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataCreateIndexService","
elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elasticsearch.node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch
.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docker-cluster"}
[elasticsearch] | {"@timestamp":"2025-11-06T21:42:16.672Z", "log.level": "INFO", "message":"[poc1_folder/rnaAoWJzR2W
xNptX14byQA] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.s
erver","process.thread.name":"elasticsearch[2ab8ceee82ab][masterService#updateTask][T#1]","log.logger":"org.elastics
earch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"swUCNmO4Q5iGG_zWpT9ydw","elasticsearch.
node.id":"ykLQsLBVRs23D0cmLJ0swQ","elasticsearch.node.name":"2ab8ceee82ab","elasticsearch.cluster.name":"docker-clus
ter"}

FScrawler

16:43:25,207 �[32mINFO �[m [f.console] ,----------------------------------------------------------------------------------------------------.
|       ,---,.  .--.--.     ,----..                                     ,--,           2.10-SNAPSHOT |
|     ,'  .' | /  /    '.  /   /   \                                  ,--.'|                         |
|   ,---.'   ||  :  /`. / |   :     :  __  ,-.                   .---.|  | :               __  ,-.   |
|   |   |   .';  |  |--`  .   |  ;. /,' ,'/ /|                  /. ./|:  : '             ,' ,'/ /|   |
|   :   :  :  |  :  ;_    .   ; /--` '  | |' | ,--.--.       .-'-. ' ||  ' |      ,---.  '  | |' |   |
|   :   |  |-, \  \    `. ;   | ;    |  |   ,'/       \     /___/ \: |'  | |     /     \ |  |   ,'   |
|   |   :  ;/|  `----.   \|   : |    '  :  / .--.  .-. | .-'.. '   ' .|  | :    /    /  |'  :  /     |
|   |   |   .'  __ \  \  |.   | '___ |  | '   \__\/: . ./___/ \:     ''  : |__ .    ' / ||  | '      |
|   '   :  '   /  /`--'  /'   ; : .'|;  : |   ," .--.; |.   \  ' .\   |  | '.'|'   ;   /|;  : |      |
|   |   |  |  '--'.     / '   | '/  :|  , ;  /  /  ,.  | \   \   ' \ |;  :    ;'   |  / ||  , ;      |
|   |   :  \    `--'---'  |   :    /  ---'  ;  :   .'   \ \   \  |--" |  ,   / |   :    | ---'       |
|   |   | ,'               \   \ .'         |  ,     .-./  \   \ |     ---`-'   \   \  /             |
|   `----'                  `---`            `--`---'       '---"                `----'              |
+----------------------------------------------------------------------------------------------------+
|                                        You know, for Files!                                        |
|                                     Made from France with Love                                     |
|                           Source: https://github.com/dadoonet/fscrawler/                           |
|                          Documentation: https://fscrawler.readthedocs.io/                          |
`----------------------------------------------------------------------------------------------------'

16:43:25,229 �[32mINFO �[m [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [1.9gb/2gb=97.87%], RAM [29gb/31.9gb=90.87%], Swap [31.8gb/36.7gb=86.77%].
16:43:25,230 �[36mDEBUG�[m [f.p.e.c.f.c.FsCrawlerCli] No job name specified. Using default one [fscrawler]...
16:43:25,230 �[36mDEBUG�[m [f.p.e.c.f.c.FsCrawlerCli] Cleaning existing status for job [fscrawler]...
16:43:25,231 �[36mDEBUG�[m [f.p.e.c.f.c.FsCrawlerCli] Starting job [fscrawler]...
16:43:25,646 �[36mDEBUG�[m [f.p.e.c.f.s.FsSettingsLoader] Successfully loaded settings from classpath [fscrawler-default.properties] and files [REDACTED\config_dir\fscrawler\_settings.yaml]
16:43:25,646 �[32mINFO �[m [f.p.e.c.f.c.FsCrawlerCli] attributes_support is set to true but getting group is not available on [windows server 2019].
16:43:25,678 �[36mDEBUG�[m [f.p.e.c.p.FsCrawlerPluginsManager] Loading plugins
16:43:25,681 �[36mDEBUG�[m [f.p.e.c.p.FsCrawlerPluginsManager] Starting plugins
16:43:25,691 �[36mDEBUG�[m [f.p.e.c.p.FsCrawlerPluginsManager] Found FsCrawlerExtensionFsProvider extension for type [http]
16:43:25,691 �[36mDEBUG�[m [f.p.e.c.p.FsCrawlerPluginsManager] Found FsCrawlerExtensionFsProvider extension for type [local]
16:43:25,691 �[36mDEBUG�[m [f.p.e.c.p.FsCrawlerPluginsManager] Found FsCrawlerExtensionFsProvider extension for type [s3]
16:43:25,700 �[32mINFO �[m [f.p.e.c.f.FsCrawlerImpl] attributes_support is set to true but getting group is not available on [windows server 2019].
16:43:25,702 �[36mDEBUG�[m [f.p.e.c.f.FsParserAbstract] creating fs crawler thread [poc1] for [W:\] every [15m]
16:43:25,704 �[36mDEBUG�[m [f.p.e.c.f.FsParserAbstract] We are going to use [.meta.yml] as meta file if found while crawling dirs
16:43:25,794 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get version
16:43:26,445 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get version returns 9.1.3 and 9 as the major version number
16:43:26,446 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 9.1.3
16:43:26,446 �[33mWARN �[m [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
16:43:26,446 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Semantic search is enabled and we are running on a version of Elasticsearch 9.1.3 which is 8.17 or higher. We will try to use the semantic search features.
16:43:26,446 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get license
16:43:26,453 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get license returns basic
16:43:26,454 �[33mWARN �[m [f.p.e.c.f.c.ElasticsearchClient] Semantic search is enabled but we are running Elasticsearch with a basic license although we need either an enterprise or trial license.We will not be able to use the semantic search features ATM. We might switch later to a vector embeddings generation.
16:43:26,458 �[36mDEBUG�[m [f.p.e.c.f.s.FsCrawlerManagementServiceElasticsearchImpl] Elasticsearch Management Service started
16:43:26,459 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get version
16:43:26,525 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get version returns 9.1.3 and 9 as the major version number
16:43:26,525 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 9.1.3
16:43:26,525 �[33mWARN �[m [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
16:43:26,525 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Semantic search is enabled and we are running on a version of Elasticsearch 9.1.3 which is 8.17 or higher. We will try to use the semantic search features.
16:43:26,525 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get license
16:43:26,530 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] get license returns basic
16:43:26,530 �[33mWARN �[m [f.p.e.c.f.c.ElasticsearchClient] Semantic search is enabled but we are running Elasticsearch with a basic license although we need either an enterprise or trial license.We will not be able to use the semantic search features ATM. We might switch later to a vector embeddings generation.
16:43:26,530 �[36mDEBUG�[m [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Elasticsearch Document Service started
16:43:26,530 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Creating/updating component templates for [poc1]
16:43:26,533 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_alias]
16:43:26,578 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_settings_total_fields]
16:43:26,634 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_mapping_attributes]
16:43:26,669 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_mapping_file]
16:43:26,703 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_mapping_path]
16:43:26,771 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_mapping_attachment]
16:43:26,812 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_mapping_content]
16:43:26,845 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_mapping_meta]
16:43:26,878 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Creating/updating index templates for [poc1]
16:43:26,879 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push index template [fscrawler_poc1_docs]
16:43:26,919 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Creating/updating component templates for [poc1_folder]
16:43:26,919 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_folder_mapping_attributes]
16:43:26,972 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_folder_mapping_file]
16:43:27,005 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push component template [fscrawler_poc1_folder_mapping_path]
16:43:27,039 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Creating/updating index templates for [poc1_folder]
16:43:27,040 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] push index template [fscrawler_poc1_folder_folders]
16:43:27,091 �[32mINFO �[m [f.p.e.c.f.FsCrawlerImpl] FSCrawler is now connected to Elasticsearch version [9.1.3]
16:43:27,091 �[36mDEBUG�[m [f.p.e.c.f.FsCrawlerImpl] Starting FSCrawler for job [poc1]
16:43:27,092 �[32mINFO �[m [f.p.e.c.f.FsCrawlerImpl] FSCrawler started in watch mode. It will run unless you stop it with CTRL+C.
16:43:27,092 �[32mINFO �[m [f.p.e.c.f.FsParserAbstract] FS crawler started for [poc1] for [W:\] every [15m]
16:43:27,092 �[32mINFO �[m [f.p.e.c.f.FsParserAbstract] Run #1: job [poc1]: starting...
16:43:27,146 �[36mDEBUG�[m [f.p.e.c.f.FsParserAbstract] indexing [W:\] content
16:43:27,146 �[36mDEBUG�[m [f.p.e.c.f.c.f.FileAbstractorFile] Listing local files from W:\
REDACTED
16:43:27,277 �[36mDEBUG�[m [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(W:\, W:\REDACTED
16:43:27,277 �[36mDEBUG�[m [f.p.e.c.f.f.FsCrawlerUtil] directory = [true], filename = [REDACTED], includes = [null], excludes = [[*/~*]]
16:43:27,277 �[36mDEBUG�[m [f.p.e.c.f.f.FsCrawlerUtil] filename = [REDACTED], excludes = [[*/~*]]
16:43:27,277 �[36mDEBUG�[m [f.p.e.c.f.f.FsCrawlerUtil] checking exclusion for filename = [REDACTED], matches = [[*/~*]]
16:43:27,278 �[36mDEBUG�[m [f.p.e.c.f.f.FsCrawlerUtil] filename = [REDACTED], includes = [null]
16:43:27,278 �[36mDEBUG�[m [f.p.e.c.f.f.FsCrawlerUtil] filename = [REDACTED], excludes = [[*/~*]]
16:43:27,394 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Error while running GET https://REDACTED:9200/_cat/indices/poc1: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"}],"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"},"status":404}
16:43:27,394 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Index [poc1] not found yet
16:43:27,902 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Error while running GET https://REDACTED:9200/_cat/indices/poc1: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"}],"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"},"status":404}
16:43:27,902 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Index [poc1] not found yet
16:43:28,912 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Error while running GET https://REDACTED:9200/_cat/indices/poc1: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"}],"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"},"status":404}
16:43:28,912 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Index [poc1] not found yet
16:43:30,921 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Error while running GET https://REDACTED:9200/_cat/indices/poc1: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"}],"type":"index_not_found_exception","reason":"no such index [poc1]","resource.type":"index_or_alias","resource.id":"poc1","index_uuid":"_na_","index":"poc1"},"status":404}
16:43:30,921 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchClient] Index [poc1] not found yet
16:43:31,459 �[36mDEBUG�[m [f.p.e.c.f.f.b.FsCrawlerSimpleBulkProcessorListener] Going to execute new bulk composed of 5 actions
16:43:31,474 �[36mDEBUG�[m [f.p.e.c.f.c.ElasticsearchEngine] Sending a bulk request of [5] documents to the Elasticsearch service

Expected behavior

Regardless of if there is a existing status.json, I don't expect to see "Index [poc1] not found yet." Poc1 and poc1_folder should be created. I should see in the elasticsearch logs "creating index [poc1_folder]" "[poc1_folder/uid] update_mapping [_doc]" "creating index [poc1]" "[poc1/uid] update_mapping [_doc]"

Versions:

• OS: Windows Server
• Version: 2019

[dadoonet]

@AlexBlueSteele I started to work on your report.

There have been changes indeed that change the way FSCrawler works now.

Before the changes, we were creating manually an index at start, even if you don't have to index a single document.

Now, the process is at start:

• Create component templates and index templates for doc and folder "indices".
• Wait for the first document to be indexed if any

At least we index the root folder in the folder index and if there is no document to be added, we just don't send data to the doc index. Which triggers the debug message you are seeing.

I don't think it's an issue. Could you tell me why you think this behavior is wrong now?

[AlexBlueSteele]

@AlexBlueSteele I started to work on your report.

There have been changes indeed that change the way FSCrawler works now.

Before the changes, we were creating manually an index at start, even if you don't have to index a single document.

Now, the process is at start:

• Create component templates and index templates for doc and folder "indices".
• Wait for the first document to be indexed if any

At least we index the root folder in the folder index and if there is no document to be added, we just don't send data to the doc index. Which triggers the debug message you are seeing.

I don't think it's an issue. Could you tell me why you think this behavior is wrong now?

Ahh ok this makes since. We found out it was because we deleted the elastic index and we had a existing status.json so it did not want to create the index. I believe it's assumed that if the status.json exists, then it wont create the index. It's not a bug; however, I feel more checks for this case with proper handling could be put in place to at least get better error messages. Let me know if this is something you would like to add and I might be able to work on it.