Extra Space After Force Token

[hodgesz]

Hello,

I modified LLMLingua2.ipynb from examples to work with our data. Since running on a Mac the follow is the PromptCompressor code.

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(device_map="mps")

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
    device_map="mps"
)

Since there are important tokens we want to keep after compression, we list them in 'force_tokens'

compressed_prompt = llm_lingua.compress_prompt(
    concatenated_string,
    rate=0.40,
    force_tokens=["!", ".", "?", "\n", "{{", "}}", "#", "##", "mediaItems", "Image:", "Image Caption:", "Image Alt Text:"],
    drop_consecutive=True,
)

print(compressed_prompt)
print(len(compressed_prompt))

However, we noticed after keeping the force_tokens, it adds a blank space after the token in the compressed prompt.

original string - {{ mediaItems.fig-ch05_02_01 }}
after compression - {{mediaItems. fig-ch05_01}}

Is there a way to avoid this blank space?

[iofu728]

Hi @hodgesz, thanks for your feedback.

We'll fix this issue soon.

[hodgesz]

Really appreciate your efforts open sourcing this project for others. Thanks for your reply!

[centminmod]

Has this been fixed yet as testing LLMLingua-2 with forced token$, I am also getting space between $ variablename for original $variablename