Crash at insert with auto embeddings and html_strip

[sanikolaev]

Bug Description:

drop table if exists t;
CREATE TABLE IF NOT EXISTS t (title text, embedding_vector FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='cosine' MODEL_NAME='sentence-transformers/all-MiniLM-L6-v2' FROM='title') html_strip='1';
INSERT INTO t (title) VALUES ('Hello.  I.  Myâ¦');

causes:

Trying boost backtrace:
 0# sphBacktrace(int, bool) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
 1# HandleCrash(int) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
 2# _sigtramp in /usr/lib/system/libsystem_platform.dylib
 3# pthread_kill in /usr/lib/system/libsystem_pthread.dylib
 4# abort in /usr/lib/system/libsystem_c.dylib
 5# _RNvCsiGVaDesi5rv_7___rustc26___rust_alloc_error_handler in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
 6# core::cell::once::OnceCell$LT$T$GT$::try_init::h5ae72910d44ba69b in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
 7# std::panicking::panic_count::is_zero_slow_path::h0e49bc980a448302 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
 8# std::panicking::panic_with_hook::h93c775fc227522dd in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
 9# std::panicking::panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h8561e58c9391724a in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
10# std::panicking::panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h8561e58c9391724a in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
11# std::sys::backtrace::__rust_end_short_backtrace::h55629ed76aaa9d26 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
12# core::panicking::panic::h4a3c64c65dc98033 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
13# alloc::raw_vec::RawVecInner$LT$A$GT$::reserve::do_reserve_and_handle::h076dffaa2fb6fbdc in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
14# alloc::raw_vec::handle_error::ha045b2c2db1a4ffe in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
15# core::panicking::assert_failed::h4b7885290ae7fea7 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
16# _$LT$tokenizers..normalizers..NormalizerWrapper$u20$as$u20$tokenizers..tokenizer..Normalizer$GT$::normalize::h2c05c1a59a04c0aa in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
17# tokenizers::tokenizer::TokenizerImpl$LT$M$C$N$C$PT$C$PP$C$D$GT$::encode_single_sequence::_$u7b$$u7b$closure$u7d$$u7d$::h4495171d93cc7de2 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
18# tokenizers::tokenizer::TokenizerImpl$LT$M$C$N$C$PT$C$PP$C$D$GT$::encode_single_sequence::hd0c7e3d5319f73b9 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
19# _$LT$manticore_knn_embeddings..model..Model$u20$as$u20$manticore_knn_embeddings..model..TextModel$GT$::predict::hd42ecbec4431f294 in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
20# manticore_knn_embeddings::model::text_model_wrapper::TextModelWrapper::make_vect_embeddings::h37a555da2ec00bee in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn_embeddings.so
21# knn::TextToEmbeddings_c::Convert(std::__1::vector<std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::allocator<std::__1::basic_string_view<char, std::__1::char_traits<char>>>> const&, std::__1::vector<std::__1::vector<float, std::__1::allocator<float>>, std::__1::allocator<std::__1::vector<float, std::__1::allocator<float>>>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) const in /opt/homebrew/Cellar/manticore-columnar-lib-dev/9.0.2B25122309-5b73f8d1/share/manticore/modules/lib_manticore_knn.so
22# RtAccum_t::GenerateEmbeddings(int, int, sph::Vector_T<AttrWithModel_t, sph::DefaultCopy_T<AttrWithModel_t>, sph::DefaultRelimit, sph::DefaultStorage_T<AttrWithModel_t>> const&, std::__1::vector<std::__1::vector<std::__1::vector<float, std::__1::allocator<float>>, std::__1::allocator<std::__1::vector<float, std::__1::allocator<float>>>>, std::__1::allocator<std::__1::vector<std::__1::vector<float, std::__1::allocator<float>>, std::__1::allocator<std::__1::vector<float, std::__1::allocator<float>>>>>>&, columnar::Iterator_i*, CSphString&) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
23# RtAccum_t::FetchEmbeddings(TableEmbeddings_c*, sph::Vector_T<AttrWithModel_t, sph::DefaultCopy_T<AttrWithModel_t>, sph::DefaultRelimit, sph::DefaultStorage_T<AttrWithModel_t>> const&, CSphString&) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
24# RtIndex_c::Commit(int*, RtAccum_t*, CSphString*) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
25# CommitMonitor_c::CommitNonEmptyCmds(RtIndex_i*, ReplicationCommand_t const&, bool) const in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
26# CommitMonitor_c::Commit() in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
27# HandleCmdReplicateImpl(RtAccum_t&, int*, CSphString*, int*) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
28# sphHandleMysqlInsert(StmtErrorReporter_i&, SqlStmt_t const&) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
29# ClientSession_c::Execute(std::__1::pair<char const*, int>, RowBuffer_i&) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
30# SqlServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c>>) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
31# MultiServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c>>, std::__1::pair<int, unsigned char>, Proto_e) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
32# std::__1::__function::__func<NetActionAccept_c::Impl_c::ProcessAccept()::$_3, std::__1::allocator<NetActionAccept_c::Impl_c::ProcessAccept()::$_3>, void ()>::operator()() in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd
33# Threads::CoRoutine_c::CreateContext(std::__1::function<void ()>, std::__1::pair<boost::context::stack_context, Threads::StackFlavour_E>)::'lambda'(boost::context::detail::transfer_t)::__invoke(boost::context::detail::transfer_t) in /opt/homebrew/Cellar/manticoresearch-dev/16.3.2-26010809-ec8034be1/bin/searchd

Manticore Search Version:

Manticore 16.3.2 ec8034be1@26010809 dev (columnar 9.0.2 5b73f8d@25122308) (secondary 9.0.2 5b73f8d@25122308) (knn 9.0.2 5b73f8d@25122308) (embeddings 1.0.1)

Operating System Version:

macos

Have you tried the latest development version?

No

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[sanikolaev]

Waiting for @donhardman to review #126

[github-actions]

Thanks for the update! However, the internal checklist is not yet complete. A Manticore team member will close this issue once everything is checked off. 🛠️