Add strategy to summarize ai message history #630

MarceloRGonc · 2025-05-16T15:30:06Z

Fixes OPS-1787.

Additional notes:

We are only summarizing after an error.

We are summarizing the chat in the following situations:

In selectRelevantTools when there is an error related tokens, we ask for a summary and we retry the request
On the MCP controller
- In the event of a general error, we request a summary in the background. And we append to the error message 'Please try again'
- On an error streaming to the user if the error is related to tokens, we ask for a summary, and we retry the request
- Upon finishing streaming, if the finish reason is 'length', we request a summary. And we append to the message 'The message was truncated because the maximum tokens for the context window was reached. Please try again.'

Summarize strategy

We ask the LLM to do the summary.
If the last message is from the user, we remove it, and we append the message to the end
If the request to the LLM fails, we truncate the chat history based on user interactions. I added an environment variable

linear · 2025-05-16T15:30:10Z

OPS-1787 Summarize MCP conversation history

packages/openops/src/lib/ai/providers/openai.ts

packages/server/shared/src/lib/system/system.ts

Copilot

Pull Request Overview

This PR adds a flexible summarization strategy for AI message history by introducing new system properties to cap history by tokens or message count and wiring summarization logic into the chat flow.

Defines new system props (MAX_LLM_CALLS_WITHOUT_INTERACTION, MAX_TOKENS_IN_LLM_HISTORY, MAX_MESSAGES_IN_LLM_HISTORY) with defaults.
Implements summarizeMessages, integrates it into the controller, and persists both full and summarized histories.
Adds unit tests for summarizeMessages.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/server/shared/src/lib/system/system.ts	Added default values for new LLM history props
packages/server/shared/src/lib/system/system-prop.ts	Expanded `AppSystemProp` enum with new caps
packages/server/api/test/unit/ai/ai-message-history-summarizer.test.ts	New tests covering summarization behavior
packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts	Implemented history summarizer utilities
packages/server/api/src/app/ai/chat/ai-mcp-chat.controller.ts	Integrated summarizer into chat controller and streaming
packages/server/api/src/app/ai/chat/ai-chat.service.ts	Added storage and append/delete for summarized history
packages/openops/src/lib/ai/providers/openai.ts	Added `compatibility: 'strict'` flag

Comments suppressed due to low confidence (1)

packages/server/api/src/app/ai/chat/ai-chat.service.ts:122

Newly added functions (getSummarizedChatHistory, appendMessagesToSummarizedChatHistory, deleteSummarizedChatHistory) lack unit tests; consider adding tests to validate summarized-history storage and retrieval behavior.

export const getSummarizedChatHistory = async (

packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts

packages/server/api/src/app/ai/chat/ai-chat.service.ts

packages/server/api/src/app/ai/chat/ai-mcp-chat.controller.ts

packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts

rita-gorokhod · 2025-06-10T20:36:21Z

packages/server/api/src/app/ai/chat/ai-mcp-chat.controller.ts

@@ -132,38 +158,50 @@ export const aiMCPChatController: FastifyPluginAsyncTypebox = async (app) => {
      isTablesLoaded,
    });

+    const streamMessagesParams: StreamMessagesParams = {


extract everything please from this point and the helper functions streamMessages. it's very difficult to review, especially with the error handling. There shouldn't be this much logic in a controller.

rita-gorokhod · 2025-06-10T20:40:50Z

packages/server/shared/src/lib/system/system.ts

@@ -88,6 +88,9 @@ const systemPropDefaultValues: Partial<Record<SystemProp, string>> = {
  [AppSystemProp.SUPERSET_MCP_SERVER_PATH]: '/root/.mcp/superset',
  [AppSystemProp.DOCS_MCP_SERVER_PATH]: '/root/.mcp/docs.openops.com',
  [AppSystemProp.LOAD_EXPERIMENTAL_MCP_TOOLS]: 'false',
+  [AppSystemProp.MAX_LLM_CALLS_WITHOUT_INTERACTION]: '10',
+  [AppSystemProp.MAX_TOKENS_FOR_HISTORY_SUMMARY]: '2000',


How does it work with our huge prompt and small models?

I don't think there are any models with less than ~4000 tokens. So, I'm defining half of that as default. But the prompt is not included in the summary.

rita-gorokhod · 2025-06-10T20:52:59Z

packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts

+    chatId,
+    [],
+    async (existingMessages) => {
+      let reAdd = false;


I think removing and readding is hard to read.

async function summarizeChatHistory(messages) { const lastMessage = messages[messages.length - 1]; const isLastMessageFromUser = lastMessage && lastMessage.role === 'user' && messages.length > 1; const messagesForSummary = isLastMessageFromUser ? messages.slice(0, -1) : messages; const summarizedHistory = await requestToGenerateSummary(languageModel, messagesForSummary, aiConfig); return isLastMessageFromUser ? [...summarizedHistory, lastMessage] : summarizedHistory; }

wdyt?

rita-gorokhod · 2025-06-10T20:53:41Z

packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts

+      model: languageModel,
+      system: systemPrompt,
+      ...aiConfig.modelSettings,
+      maxTokens: getHistoryMaxTokens(aiConfig),


how does the maxTokens work in this case?

WDYM? The maxTokens is the maximum number of tokens to generate. So it will be the value from the env var (2000 by default) or the value defined in the model settings.

yes but does it mean that the history will be max 2000 tokens? what if it needs more?

rita-gorokhod · 2025-06-10T20:57:13Z

packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts

+  languageModel: LanguageModel,
+  messages: CoreMessage[],
+  aiConfig: AiConfig,
+  attemptIndex = 0,


I think the recursion here is confusing with the attemptIndex .
Why not instead call an inner func?
this method is supposed to be called only twice, no? for full input, and for last 2 interactions -- if those fail we don't try anymore

I think these retries (2) are enough. However, I'm not sure if you want to configure it. Can I always assume 2 retries?

even if it's configurable, you can do it without recursion.

rita-gorokhod · 2025-06-10T21:06:53Z

packages/server/api/src/app/ai/chat/tools.service.ts

+    const errorMessage = error instanceof Error ? error.message : String(error);
+
+    attemptIndex = attemptIndex + 1;
+    if (!shouldTryToSummarize(errorMessage, attemptIndex)) {


I don't think this should be here, this gives you the tools selection, it should already receive a summarized data if summarization was needed.
All the error handling and summarization should be done on the caller level.

We are calling the LLM to select the tools and we are using the entire history. It may be necessary to summarize since we are only summarizing on error.

Sure but then you can do it in the caller.
Call tools with everything --> failed on max? --> summarize --> call tools with summarized

otherwise this logic is spread in multiple places

rita-gorokhod

See comments, this needs some refactoring to simplify the code, it's very difficult to read

sonarqubecloud · 2025-06-12T08:35:04Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Summarize chat history

af62cc4

MarceloRGonc commented May 16, 2025

View reviewed changes

packages/openops/src/lib/ai/providers/openai.ts Outdated Show resolved Hide resolved

MarceloRGonc added 2 commits May 19, 2025 11:14

Add unit tests

89903ae

Move generate message id to file

2bea861

MarceloRGonc changed the base branch from main to mg/OPS-1787-2 May 19, 2025 10:18

Merge branch 'mg/OPS-1787-2' into mg/OPS-1787

2af71dd

MarceloRGonc changed the title ~~Summarize chat history~~ Add strategy to summarize ai message history May 19, 2025

MarceloRGonc marked this pull request as ready for review May 19, 2025 10:22

MarceloRGonc requested a review from Copilot May 19, 2025 10:22

Base automatically changed from mg/OPS-1787-2 to main May 19, 2025 10:22

Merge branch 'main' into mg/OPS-1787

c10cec9

MarceloRGonc commented May 19, 2025

View reviewed changes

packages/server/shared/src/lib/system/system.ts Outdated Show resolved Hide resolved

MarceloRGonc requested review from bigfluffycookie and rita-gorokhod May 19, 2025 10:24

Copilot AI reviewed May 19, 2025

View reviewed changes

packages/server/api/src/app/ai/chat/ai-message-history-summarizer.ts Outdated Show resolved Hide resolved

packages/server/api/src/app/ai/chat/ai-chat.service.ts Outdated Show resolved Hide resolved

packages/server/api/src/app/ai/chat/ai-chat.service.ts Outdated Show resolved Hide resolved

MarceloRGonc added 2 commits May 19, 2025 11:47

Fix unused variable

d6e26c4

Merge branch 'main' into mg/OPS-1787

c6c0b0e

bigfluffycookie reviewed May 19, 2025

View reviewed changes

Merge branch 'main' into mg/OPS-1787

5741cf8

bigfluffycookie approved these changes May 20, 2025

View reviewed changes

MarceloRGonc added 5 commits May 20, 2025 10:35

Merge branch 'main' into mg/OPS-1787

b3f2475

Merge

6b6073d

Merge branch 'main' into mg/OPS-1787

e9f81f1

Merge branch 'main' into mg/OPS-1787

c850240

Restore retries

42dc083

MarceloRGonc changed the base branch from main to mg/simplify-recursion May 21, 2025 14:08

Base automatically changed from mg/simplify-recursion to main May 21, 2025 17:57

MarceloRGonc added 2 commits May 21, 2025 19:23

Merge branch 'main' into mg/OPS-1787

c8ebf26

WIP

5d6a37b

MarceloRGonc added 3 commits May 30, 2025 15:45

Merge branch 'mg/OPS-1787-3' into mg/OPS-1787

3ff98e7

Merge branch 'main' into mg/OPS-1787-3

d846136

Merge branch 'mg/OPS-1787-3' into mg/OPS-1787

57d7e95

MarceloRGonc requested a review from rita-gorokhod May 30, 2025 15:24

Base automatically changed from mg/OPS-1787-3 to main June 2, 2025 07:35

MarceloRGonc added 8 commits June 2, 2025 08:36

Merge branch 'main' into mg/OPS-1787

f61707e

Merge branch 'main' into mg/OPS-1787

83394c9

Merge branch 'main' into mg/OPS-1787

2312728

Merge branch 'main' into mg/OPS-1787

f7f2c34

Fix lint

35dfda3

Merge branch 'main' into mg/OPS-1787

8b0d316

Fix error

888060f

Merge

0a6b28f

rita-gorokhod reviewed Jun 10, 2025

View reviewed changes

rita-gorokhod requested changes Jun 10, 2025

View reviewed changes

MarceloRGonc added 7 commits June 11, 2025 08:36

Merge branch 'main' into mg/OPS-1787

e6d430e

Merge branch 'main' into mg/OPS-1787

f565abe

Refactor

d56079b

Add unit tests

6477bfd

Add unit tests

8275a48

Merge branch 'main' into mg/OPS-1787

34db101

Fix test

9b2ed84

MarceloRGonc requested a review from rita-gorokhod June 11, 2025 19:25

Merge branch 'main' into mg/OPS-1787

7635ddc

Add strategy to summarize ai message history #630

Are you sure you want to change the base?

Add strategy to summarize ai message history #630

Conversation

MarceloRGonc commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Additional notes:

We are summarizing the chat in the following situations:

Summarize strategy

Uh oh!

linear bot commented May 16, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rita-gorokhod Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rita-gorokhod Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rita-gorokhod Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rita-gorokhod Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rita-gorokhod left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jun 12, 2025

Quality Gate passed

Uh oh!

Uh oh!

MarceloRGonc commented May 16, 2025 •

edited

Loading

rita-gorokhod Jun 10, 2025 •

edited

Loading

rita-gorokhod Jun 12, 2025 •

edited

Loading

rita-gorokhod Jun 10, 2025 •

edited

Loading

rita-gorokhod Jun 12, 2025 •

edited

Loading