feat: Implement AI Wizard for experiment generation and enhance exper… #93
Conversation
There was a problem hiding this comment.
Thanks @yashpatil641. Left a few comments.
I think this repo is going to become obsolete very soon. It'd be good to have you work on the evidential repos instead.
Also, big PRs are very hard to review. Think about how you can break them up into smaller pieces (e.g. backend only, front end but not connected, integration etc.) to make it easier for the reviewer. You can also use feature branches to make it easier.
| from typing import Any, Optional | ||
|
|
||
| from fastapi import APIRouter, HTTPException | ||
| from google import genai |
There was a problem hiding this comment.
We use Litellm wherever possible to remain foundation model agnostic
| router = APIRouter(prefix="/ai_helpers", tags=["AI Helpers"]) | ||
|
|
||
|
|
||
| class ExperimentAIGenerateRequest(BaseModel): |
There was a problem hiding this comment.
You should move these to a schemas.py file. See other routers
|
|
||
| goal: str | ||
| outcome: str | ||
| num_variants: int |
|
|
||
| name: str | ||
| description: str | ||
| methodType: str |
There was a problem hiding this comment.
Maybe an enumeration for this?
|
|
||
| api_key = os.environ.get("GEMINI_API_KEY") | ||
| if not api_key: | ||
| raise HTTPException(status_code=500, detail="Gemini API key not set") |
There was a problem hiding this comment.
This code with run at load time before fastapi app is running so should be a different exception - not HTTPException.
| ) -> list[dict[str, Any]]: | ||
| """Suggest arms for Bayesian A/B test.""" | ||
| # System instruction for arms suggestion | ||
| system_instruction = ( |
There was a problem hiding this comment.
See https://github.com/IDinsight/ask-a-question for how to move this our to a different file where we can manage prompts. Also, can i suggest using """ for multiline strings and textwrap.dedent?
| "No explanation, no markdown." | ||
| ) | ||
|
|
||
| user_prompt = ( |
There was a problem hiding this comment.
Why not use JSON? and then use pydantic to dump JSON
| ), | ||
| ) | ||
| return json.loads(response.text) | ||
| except Exception as e: |
There was a problem hiding this comment.
Don't catch base exceptions! What exception are you expecting here? Let's just catch those
|
|
||
| component: AddNotifications | ||
| }, | ||
| { |
There was a problem hiding this comment.
Do you use prettier to format typescript code? Can i suggest using it?
Reviewer: @poornimaramesh @Jayprakash-SE @sidravi1
Estimate:
Ticket
Implement AI Wizard for experiment generation
Description
Goal
This pull request introduces the AI Experiment Wizard, a new feature designed to simplify the experiment creation process. The goal is to empower non-technical platform owners to design and launch robust experiments by answering a few high-level questions, aligning with our core design principle of making the Experiments Engine simple and accessible.
Changes
/experiments/add/ai-wizardwhere users can input their experimentgoal,outcome, and the desirednumber of variants.backend/app/ai_helpers/routers.pythat use the Gemini API to generate a complete experiment configuration, including a suggested name, description, experiment type, arms, and contexts.useExperimentStoreto handle the state of the AI Wizard. The generated experiment details are used to pre-populate the standard experiment creation form for user review./frontend/src/app/(protected)/experiments/add/components/summary.tsx) now displays the AI-generated goal and outcome, providing better context for the experiment's purpose.Future Tasks (optional)
How has this been tested?
To-do before merge (optional)
Checklist
Fill with
xfor completed.Demo Video:
rec2.mp4