BLUF: Including summarization in the top system message causes full cache invalidation unnecessarily following a summarization
Solution:
Move summarization in between the preamble section and the chat section. This allows the potentially large system messages and system preambles, which are unaffected by summarization, to remain cached.
This could be a new "Summary Preamble" along with the current System/User preamble messages or a different type of message as long as the message type doesn't cause it to be picked up during summarization.
BLUF: Including summarization in the top system message causes full cache invalidation unnecessarily following a summarization
Solution:
Move summarization in between the preamble section and the chat section. This allows the potentially large system messages and system preambles, which are unaffected by summarization, to remain cached.
This could be a new "Summary Preamble" along with the current System/User preamble messages or a different type of message as long as the message type doesn't cause it to be picked up during summarization.