Fix tool remembering to avoid spamming it

[vria]

Currently, there is an annoying feature to remember available tools in message history. In a typical task that requires extensive use of tools, this remembering leads to excessively repeated "You ONLY have access to the following tools..." messages. This clutters the input and distract the model from the main task.

In this PR, I propose a solution with minimal disruption to the existing behavior. In particular, once tool remembering is needed (tool usage count > 3), the recall message will always appear only once as the last message.

N.B. This PR is intended to start a discussion. I will refine it and add/correct tests once we agree on the solution.

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment for PR #3073

Overview

This pull request primarily modifies the tool remembering functionality within the CrewAI framework, enhancing how tools are tracked and remembered during agent execution. This change is crucial as it improves user interactions with the system by making the cognitive load lighter and interactions more informative.

Positive Changes

• The addition of a tool remembering logic is a welcome improvement, enabling better tracking of tool usage, which ultimately enhances user experience.
• The implementation of the _remember_format_message method is clean and dedicated, showcasing a good separation of concerns for message handling related to tool usage.

Areas for Improvement

While the improvements in functionality and maintainability are noted, there are several key areas where the proposed changes could be enhanced:

1. Magic Number Usage:

   • Issue: The use of the value 3 directly in the code for determining when to trigger the reminder can make the code less readable and maintainable.
   • Suggestion: Replace the magic number with a named constant like this:

     TOOL_REMINDER_THRESHOLD = 3
     if self.task.used_tools > TOOL_REMINDER_THRESHOLD:

2. Clarity in Variable Naming:

   • Issue: The assignment of messages could be clearer.
   • Suggestion: Use more descriptive naming to reflect purpose:

     final_messages = self.messages.copy()
     if self.task.used_tools > TOOL_REMINDER_THRESHOLD:
         final_messages.append(self._remember_format_message())

3. Missing Type Hints:

   • Issue: The _remember_format_message method lacks type hints, which are beneficial for readability and maintainability.
   • Suggestion: Explicitly add type hints like this:

     from typing import Dict
     def _remember_format_message(self) -> Dict[str, str]:
         ...

4. Documentation Update:

   • Issue: Removal of methods like _should_remember_format means documentation for existing methods needs to be updated to accurately reflect the current functionality.
   • Suggestion: Ensure that docstrings for methods like _format_result clearly describe the new logic and parameter requirements post-refactoring.

5. Error Handling:

   • Issue: Code may not robustly handle scenarios where the task or its attributes are not defined.
   • Suggestion: Introduce error handling to prevent potential crashes:

     def _format_result(self, result: Any) -> str:
         try:
             if self.task:
                 self.task.used_tools += 1
         except AttributeError:
             logger.warning("Could not update tool usage counter")
         return str(result)

6. Configuration of Threshold:

   • Suggestion: Making the TOOL_REMINDER_THRESHOLD configurable in the constructor of CrewAgentExecutor can provide flexibility for different use cases:

     class CrewAgentExecutor:
         def __init__(self, 
                      tool_reminder_threshold: int = 3,
                      *args, **kwargs):
             self.tool_reminder_threshold = tool_reminder_threshold

Conclusion

The changes in this PR are a solid step forward for the CrewAI framework. Addressing the highlighted issues—such as magic numbers, documentation, type hints, and error handling—will further enhance the quality and maintainability of the code, ensuring a robust tool usage feature. Once these suggestions are implemented, I believe the PR will be ready for approval.

[lucasgomide]

Hey @vria, this looks really interesting! I do have a few code concerns, but let's first focus on the issue you're trying to solve.

Honestly, I hadn't noticed this behavior before - probably because I rarely use more than 3 tasks per agent (i'm not sure), anyway.. Could you share a bit more detail? Maybe include the LLM message logs, before and after your suggested changes. That would really help clarify the issue and give us some concrete artifacts to work with.

[vria]

Hi @lucasgomide,
here is a concrete launches before and after a fix:

• before.log
• after.log

This is a dummy task to create 10 files. You can observe that:

• before.log contains 5 "You ONLY have access to the following tools..." across 14 messages.
• after.log contains only 2 of them across 13 messages: one in the first system message and the second one in the last message.

We can refactor it further and exclude "You ONLY have access to the following tools..." from the first system message when we decide to put the reminder in the last message.

Yes, as I have mentioned this is not very elaborate PR because I'd like to validate the approach before doing it properly.

[lucasgomide]

@vria just Just so you know, I'm talking to the internal team to get their feedback on this, since they might have more context than I do.

I will back with them soon!