Fix CLI documentation to reflect actual provider count and two-step process

[devin-ai-integration]

Fix CLI documentation to reflect actual provider count and two-step process

Description

Fixes issue #3054 where the CLI documentation incorrectly stated that crewai create crew shows "top 5 most common LLM providers" when it actually shows 12 providers plus additional options in a two-step process.

Problem

The CLI documentation in docs/concepts/cli.mdx contained outdated information:

• Claimed to show "top 5 most common LLM providers"
• Did not mention the two-step process (select provider, then select model)
• Provider list was incomplete and inaccurate

Solution

1. Updated CLI documentation (docs/concepts/cli.mdx):

   • Removed outdated "top 5 most common LLM providers" reference
   • Updated to accurately describe the current behavior showing 12 providers
   • Added description of the two-step process: select provider, then select model
   • Listed all 12 current providers: OpenAI, Anthropic, Google Gemini, NVIDIA NIM, Groq, Hugging Face, Ollama, Watson, AWS Bedrock, Azure, Cerebras, SambaNova

2. Added comprehensive test (tests/test_cli_documentation_sync.py):

   • Validates that documentation doesn't contain outdated "top 5" references
   • Ensures key providers are mentioned in documentation
   • Checks that the actual PROVIDERS list matches expected providers
   • Prevents future documentation drift by automatically detecting mismatches

Testing

• ✅ Validated that documentation no longer contains "top 5" or "5 most common" references
• ✅ Confirmed all 12 providers from src/crewai/cli/constants.py are properly documented
• ✅ Verified the CLI behavior matches the updated documentation
• ✅ Added automated test to prevent regression

Changes Made

• docs/concepts/cli.mdx: Updated provider count and process description
• tests/test_cli_documentation_sync.py: New test file to prevent documentation drift

Verification

The fix was validated by:

1. Examining the actual CLI implementation in src/crewai/cli/constants.py (12 providers)
2. Testing the CLI command crewai create crew to confirm two-step process
3. Running validation script to confirm documentation updates
4. Creating automated test to prevent future issues

Closes #3054

---

Link to Devin run: https://app.devin.ai/sessions/e1ad16a696ab4522a2da6a16a08e4879

Requested by: João (joao@crewai.com)

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment for PR #3055

Overview

This PR effectively updates the CLI documentation and adds new tests to ensure that the documentation remains aligned with the actual implementation of the provider. The changes cover two main files: docs/concepts/cli.mdx and tests/test_cli_documentation_sync.py.

Documentation Changes Analysis (cli.mdx)

Positive Aspects

1. Removed Outdated References: The outdated references to "top 5 providers" were successfully eliminated, reflecting the current landscape accurately.
2. Accurate Provider Count: The fact that the documentation now states there are 12 supported providers enhances user awareness.
3. Clarified Selection Process: The explanation of the two-step selection process is clearer, contributing to a better user experience.
4. Consistent Format: Maintains formatting and structure, aiding readability.

Suggestions for Improvement

1. Provider List Formatting: The current bullet list format could be improved with a table format for better clarity.

   | Provider | Description |
   |----------|-------------|
   | OpenAI   | Standard OpenAI API integration |
   | Anthropic| Claude and Claude 2 models |

2. Version Information Insight: Including the relevant version information would help users with compatibility.

   > Note: As of version X.Y.Z, CrewAI supports 12 primary providers plus additional LiteLLM-supported options.

3. Provider-Specific Details: Adding specific requirements for each provider will guide users more effectively.

   #### Provider-Specific Requirements
   - OpenAI: Requires API key and optional organization ID.
   - Anthropic: Requires API key only.
   - Google Gemini: Requires API key and project configuration.

Test Implementation Analysis (test_cli_documentation_sync.py)

Positive Aspects

1. Separation of Test Cases: Tests are well-structured, promoting maintenance and readability.
2. Comprehensive Provider Verification: All current providers are genuinely verified against documentation.
3. Clear Assertions: Assertions are well-named, which assists in diagnosing problems during tests.

Suggestions for Improvement

1. Enhanced Documentation Strings: Augmenting the docstrings would make the purpose of the tests clearer.

   def test_cli_documentation_matches_providers():
       """
       Ensures CLI documentation accurately reflects available providers.
       
       Tests:
       - Verifies no outdated "top 5" references
       - Checks all currently supported providers are listed
       - Ensures key providers are included
       
       Raises:
           AssertionError: If documentation does not sync with actual providers
       """

2. Define Error Message Constants: Using constants for error messages would increase clarity and maintainability.

   ERROR_MESSAGES = {
       "TOP_5": "Documentation should not mention 'top 5' providers",
       "PROVIDER_COUNT": "Expected {expected} providers, but found {actual}",
       "MISSING_PROVIDER": "Key provider {provider} should be mentioned in documentation"
   }

3. Provider Validation Helper: A helper function could streamline provider existence checks in documentation.

   def validate_provider_existence(content: str, provider: str) -> bool:
       """Validates if a provider exists in the documentation."""
       return provider in content

Security Considerations

1. File Path Validation: Ensure that file reading operations are safe by validating paths:

   def safe_read_docs_file(base_path: Path, file_name: str) -> str:
       """Safely read documentation file with path validation."""
       file_path = base_path / file_name
       if not file_path.is_relative_to(base_path):
           raise ValueError("Invalid file path")
       return file_path.read_text()

Overall Recommendations

1. Documentation:

   • Consider including provider-specific configuration examples.
   • Develop a troubleshooting section for common API key issues.
   • Create a version compatibility matrix for clarity.

2. Tests:

   • Expand coverage with parameterized tests for each provider.
   • Include negative test cases to validate robustness.
   • Check documentation format in tests for comprehensive validation.

3. Future Considerations:

   • Implement automated documentation verification in the CI pipeline.
   • Create a provider capability matrix to assist users further.
   • Explore provider-specific configuration validators to enhance user handling.

The changes proposed in this pull request demonstrate a commitment to accurate and reliable documentation. Ensuring that the automated tests align with the documentation further solidifies the foundation for long-term maintenance. Implementing the suggested improvements would offer even greater benefits in user experience and code maintainability.

[mplachta]

Disclaimer: This review was made by a crew of AI Agents.

---

Code Review for PR #3055: CLI Documentation Update and Documentation-Implementation Sync Test

Summary of Key Findings

• Documentation (docs/concepts/cli.mdx):
  The documentation has been accurately updated to reflect the actual 12 LLM providers plus an "other" option, moving away from the outdated "top 5" references. It now clearly describes the two-step process in the CLI: first choosing a provider, then selecting a model before entering API keys. This update removes confusion and aligns user-facing docs with actual CLI behavior.

• Test Addition (tests/test_cli_documentation_sync.py):
  A valuable test suite was introduced to prevent future divergence between the documentation and CLI provider constants. The tests check for absent outdated text, presence of key providers in the docs, and verify the provider list constant matches the expected providers exactly. This is an excellent practice to maintain documentation accuracy over time.

---

Specific Code Improvements and Suggestions

1. Documentation (cli.mdx):

   • Provider List Consistency and Formatting:
     The hardcoded list mixes styles and casing (e.g., "Google Gemini" vs. plain "OpenAI"). For better readability and professionalism, standardize capitalization and formatting. For example:

     * OpenAI
     * Anthropic
     * Gemini
     * NVIDIA NIM
     * Groq
     * Hugging Face
     * Ollama
     * Watson
     * AWS Bedrock
     * Azure
     * Cerebras
     * SambaNova
     

     Alternatively, use a simple markdown table for better extensibility:

     Provider	Description
     OpenAI	Industry-leading LLM provider
     Anthropic	Safety-focused LLMs
     Gemini	Google’s LLM offering
     NVIDIA NIM	NVIDIA's inference models
     ...	...

   • Clarify Model Selection Step:
     Currently, the doc states: "the CLI will then show you available models for that provider and prompt you to enter your API key." It could be clearer to mention the possibility of no available models or error handling:

     When you select a provider, the CLI presents available models. If no models are available, please check your configuration or API keys.

   • Explain “Other” Option:
     For non-technical users, briefly explain LiteLLM in the context of the "other" option:

     Selecting "other" allows choosing from LiteLLM-supported providers, a compatibility layer enabling additional LLM services.

   • Add Example Workflow Section:
     Enhance user guidance by illustrating a typical CLI interaction:

     #### Example workflow
     1. Run `crewai create crew`
     2. Select provider "OpenAI"
     3. Select model "gpt-4"
     4. Enter your OpenAI API key
     

   • Consider Automation for Provider List:
     To avoid future manual sync issues, consider generating provider lists directly from code constants or embedding markers in docs that can be auto-updated. While outside the scope of this PR, it’s a worthwhile process improvement.

2. Tests (test_cli_documentation_sync.py):

   • Expand Provider Name Checks:
     Instead of checking only three key providers ("OpenAI", "Anthropic", "Gemini") in the documentation, iterate over all providers from PROVIDERS for better coverage:

     for provider in expected_providers:
         formatted_name = provider.replace("_", " ").title().replace("Nim", "NIM").replace("Aws Bedrock", "AWS Bedrock")
         assert formatted_name in docs_content, f"Provider {formatted_name} should be mentioned in documentation"

   • Improve Assert Messages:
     Include actual vs expected values for easier debugging when tests fail:

     assert PROVIDERS == expected_providers, (
         f"PROVIDERS list has changed.\nExpected: {expected_providers}\nFound: {PROVIDERS}"
     )

   • Refactor Path Handling:
     Use a repository root resolver to construct the docs path more robustly:

     repo_root = Path(__file__).parents[1]
     docs_path = repo_root / "docs" / "concepts" / "cli.mdx"

   • Consider Test Structuring:
     Combine similar checks or parameterize tests to simplify maintenance and scale coverage as the provider list grows.

   • Use Flexible Matching:
     If docs formatting changes in the future, using regex or semantic search could reduce false negatives caused by minor wording changes.

3. General Recommendations:

   • Maintain PROVIDERS as the single source of truth in code.
   • Align documentation updates with test changes strictly to prevent regression or drift.
   • Consider scripting or tooling to auto-generate provider lists in docs for high accuracy.
   • Maintain consistency in names and formatting between docs, CLI outputs, and test code.

---

Links to Historical Context and Related Learnings

• This PR fixes issue [BUG] CLI documentation not up to date #3054, which previously highlighted outdated and misleading CLI documentation around provider selection.
• Similar documentation drift issues frequently occur when lists like providers or supported services evolve rapidly without automated syncing.
• The addition of this test is a direct response to that prior problem and embodies a best practice: coupling code and documentation validation through tests.
• The co-authorship in commits reflects collaborative ownership and shared responsibility for quality, a key team dynamic to preserve.

---

Overall Conclusion

This PR significantly improves user clarity by updating CLI documentation to reflect current behavior and expanding the provider list accurately. The accompanying test addition is a strong safeguard against documentation drift—a common pain point in evolving projects.

The implementation and test code are well-structured and clear, though could be improved by expanding provider list coverage in tests, enhancing error messages, and improving formatting consistency in docs. Considering automation for provider list generation for documentation would be a highly beneficial future enhancement.

Great progress overall. Implementing the suggested refinements will further enhance maintainability, usability, and robustness.

Thank you for addressing this important usability issue with both documentation and automated testing.

---

[lucasgomide]

Drop this file test

[lucasgomide]

sync the branch with main and resolve conflicts