Increasing RAM use while fscrawling

[ScottCov]

I have indexed about 25,000 PDFs so far and my available RAM continues to decline . When I started fscrawler this process was using only 625MB...and it went up to 3.25GB. Obviously it seem to have something to do with logging. I deleted most of the log files and it is now down to 2.5GB. I am using docker for fscrawler and when I ran it thought I set it up so all log files wouldn't be saved inside the container:

docker run -d --env FS_JAVA_OPTS=-DDOC_LEVEL=debug --name fscrawler -v /home/serveracct/logs/log1:/usr/share/fscrawler/logs -v /home/serveracct/logs/log2:/tmp -v ~/.fscrawler:/root/.fscrawler -v /mnt/cloud/cases:/tmp/es:ro dadoonet/fscrawler fscrawler job_name
Here is a description of the process that is taking up increasing memory:

Any ideas how to stop the excessive use of RAM?

And incidentally it may say 794MB of memory above but its 2.37GB:

Scott

[ScottCov]

OK it looks like its gone down now and is using only 724MB...now can someone explain what is going on?

[dadoonet]

Could it be caused by tesseract? I mean: is this happening the same way if you disable OCR?

[dadoonet]

Another question: if you are using a recent build, could it be caused by semantic text? I just added this recently.
Could you check if anything related to this are in your logs?

[ScottCov]

Hi David-

The problem seems to go away after I reboot so it's been alleviated at least temporarily. If its ok I may want to circle back on this at a later date though. I will go thru my logs though and see if I can spot anything. Previously I couldn't reboot since fscrawler seemed to then want to reindex but I solved this by turning off the option to delete files which is another issue that I know we've discussed in the past and I intend to circle back with you on in the future once I get thru some pdfs that really need to indexed.

Much obliged/merci beaucoup!

Scott

[dadoonet]

Avec plaisir 😉

just let me know what are the very first lines of the logs when you restart. Specifically something about semantic

[ScottCov]

Hi David-Sorry never followed up on this but I would like to figure out the issue:

cat es_dockets.log | grep "semantic"returns the following:

`[2025-12-01T20:10:26,991][INFO ][o.e.f.FeatureService ] [pauli] Registered local node features [cluster.stats.source_modes, data_stream.auto_sharding, data_stream.lifecycle.global_retention, data_stream.rollover.lazy, desired_node.version_deprecated, esql.agg_values, esql.async_query, esql.base64_decode_encode, esql.casting_operator, esql.counter_types, esql.disable_nullable_opts, esql.from_options, esql.metadata_fields, esql.metrics_counter_fields, esql.mv_ordering_sorted_ascending, esql.mv_sort, esql.resolve_fields_api, esql.spatial_points_from_source, esql.spatial_shapes, esql.st_centroid_agg, esql.st_contains_within, esql.st_disjoint, esql.st_intersects, esql.st_x_y, esql.string_literal_auto_casting, esql.string_literal_auto_casting_extended, esql.timespan_abbreviations, features_supported, file_settings, flattened.ignore_above_support, geoip.downloader.database.configuration, get_database_configuration_action.multi_node, health.dsl.info, health.extended_repository_indicator, knn_retriever_supported, license-trial-independent-version, logsdb_telemetry, logsdb_telemetry_stats, mapper.boolean_dimension, mapper.flattened.ignore_above_with_arrays_support, mapper.ignore_above_index_level_setting, mapper.index_sorting_on_nested, mapper.keyword_dimension_ignore_above, mapper.keyword_normalizer_synthetic_source, mapper.pass_through_priority, mapper.query_index_mode, mapper.range.null_values_off_by_one_fix, mapper.segment_level_fields_stats, mapper.source.synthetic_source_copy_to_fix, mapper.source.synthetic_source_copy_to_inside_objects_fix, mapper.source.synthetic_source_fallback, mapper.source.synthetic_source_stored_fields_advance_fix, mapper.source.synthetic_source_with_copy_to_and_doc_values_false, mapper.subobjects_auto, mapper.subobjects_auto_fixes, mapper.synthetic_source_keep, mapper.track_ignored_source, mapper.vectors.bbq, mapper.vectors.bit_vectors, mapper.vectors.int4_quantization, put_database_configuration_action.ipinfo, query_rule_list_types, query_rule_retriever_supported, query_rules.test, random_reranker_retriever_supported, repositories.supports_usage_stats, rest.capabilities_action, rest.local_only_capabilities, retrievers_supported, routing.boolean_routing_path, routing.multi_value_routing_path, rrf_retriever_composition_supported, rrf_retriever_supported, script.hamming, script.term_stats, search.vectors.k_param_supported, security.migration_framework, security.role_mapping_cleanup, security.roles_metadata_flattened, semantic_text.default_elser_2, semantic_text.search_inference_id, simulate.component.template.substitutions, simulate.index.template.substitutions, simulate.mapping.addition, simulate.mapping.validation, simulate.mapping.validation.templates, simulate.support.non.template.mapping, slm.interval_schedule, snapshot.repository_verify_integrity, standard_retriever_supported, stats.include_disk_thresholds, text_similarity_reranker_retriever_composition_supported, text_similarity_reranker_retriever_supported, tsdb.ts_routing_hash_doc_value_parse_byte_ref, unified_highlighter_matched_fields, usage.data_tiers.precalculate_stats]

`