Objective
As I understand, current implementation of conversation history in context includes the past 6 messages in context with the next prompt if they fit, and trims off more if they don't fit.
I suggest we improve this by using a summary of conversation history. Because Neon's current models have smaller context windows, I set up the AskJerry version with this logic:
If the current conversation history is less than 4096 tokens total, include it all. Otherwise, generate and include an LLM-generated summary of the conversation so far.
Initial Implementation Requirements
Other Considerations
- Context is also important for successfully continuing conversations that include tool use, such as when asking for available courses for summer session you get a result and then could ask a follow-up query such as "what about fall?" and it should use context to infer that it needs to continue using the course query tool not shift to panel answers or anything else.
Objective
As I understand, current implementation of conversation history in context includes the past 6 messages in context with the next prompt if they fit, and trims off more if they don't fit.
I suggest we improve this by using a summary of conversation history. Because Neon's current models have smaller context windows, I set up the AskJerry version with this logic:
If the current conversation history is less than 4096 tokens total, include it all. Otherwise, generate and include an LLM-generated summary of the conversation so far.
Initial Implementation Requirements
Other Considerations