When using code to get PRF:
prf_rank = pytcolbert.prf(rerank=False,fb_docs=10,fb_embs=10)
df=prf_rank.search(query).head(1)
query_toks=df.iloc[0]['query_toks']
The resulting query_token often contains noise, such as:
['coating', 'smuggling', 'radioactive', 'kazakhstan', 'legislation', '##b', '12', 'minister', '"', '"']
query_token has unrelated symbols or numbers instead of words.
Is this due to the use of the robust2004 dataset? Is the noise caused by selecting the document during the PRF process and extracting the contents of the document set other than ?
In the indexing phase, I used the code from the pull request. That is, when using the robust2004 document set, the contents of line["text"] are extracted. Is this also required in the PRF phase?
When using code to get PRF:
The resulting query_token often contains noise, such as:
['coating', 'smuggling', 'radioactive', 'kazakhstan', 'legislation', '##b', '12', 'minister', '"', '"']query_token has unrelated symbols or numbers instead of words.
Is this due to the use of the robust2004 dataset? Is the noise caused by selecting the document during the PRF process and extracting the contents of the document set other than ?
In the indexing phase, I used the code from the pull request. That is, when using the robust2004 document set, the contents of line["text"] are extracted. Is this also required in the PRF phase?