Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Non LLM based context precision #1264

Merged
merged 25 commits into from
Sep 12, 2024
Merged

feat: Non LLM based context precision #1264

merged 25 commits into from
Sep 12, 2024

Conversation

shahules786
Copy link
Member

@shahules786 shahules786 commented Sep 10, 2024 •
edited
Loading

  1. Add Non LLM based context precision
  2. Rename and introduce naming convention to metrics - <llm_or_non_llm><metric_name><with_or_without_reference>
from ragas import SingleTurnSample
from ragas.metrics._context_precision import NonLLMContextPrecisionWithReference

context_precision = NonLLMContextPrecisionWithReference()

sample = SingleTurnSample(
    retrieved_contexts=["The Eiffel Tower is located in Paris."], 
    reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."]
)

await context_precision.single_turn_ascore(sample)

shahules786 and others added 17 commits September 9, 2024 16:27
@shahules786
add bleu score
1fce3f7
@shahules786
non llm string metrics
c4ae7d6
@shahules786
add rogue score
0f0e88a
@shahules786
modify optional dependencies
546d149
@shahules786
move rogue score
625c7b8
@shahules786
Add support to MultiTurn Metrics (explodinggradients#1249)
3475154 
```python
from ragas import evaluate
from ragas.metrics import rubrics_score_without_reference
from ragas.dataset_schema import EvaluationDataset, MultiTurnSample
from ragas.messages import HumanMessage

sample1 = MultiTurnSample(user_input=[HumanMessage(content="What is X")])
sample2 = MultiTurnSample(user_input=[HumanMessage(content="What is X")])
ds = EvaluationDataset(samples=[sample1, sample2])
evaluate(ds,metrics=[rubrics_score_without_reference])
```
@shahules786
change required_columns type
aa7330b
@shahules786
Merge branch 'main' into non-llm-metrics
4db0b8d
@shahules786
add rapidfzz
7329a8e
@shahules786
add context precision non llm based
2f1dcac
@shahules786
normalise distances
634d587
@shahules786
round scores
d2c628d
@shahules786
Merge branch 'main' into metrics#1
854136f
@shahules786
name old metrics
ba6d1f9
@shahules786
add docs for v2
51b63a9
@shahules786
Non LLM based metrics (explodinggradients#1260)
93c2be5 
Added support for
1. BLEU SCORE
2. ROGUE SCORE
3. STRING MATCH
4. EDIT DISTANCE
@shahules786 @jjmachan
feat: set and get prompts for metrics (explodinggradients#1259)
2d1e540 
```python
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental, LongFormAnswerPrompt

faithfulness = FaithfulnessExperimental() 
faithfulness.get_prompts()

#{'long_form_answer_prompt': <ragas.experimental.metrics._faithfulness.LongFormAnswerPrompt at 0x7fd7baa8efb0>,
#'nli_statement_prompt': <ragas.experimental.metrics._faithfulness.NLIStatementPrompt at 0x7fd7baa8f010>}

long_form_prompt = LongFormAnswerPrompt()
long_form_prompt.instruction = "my new instruction"

prompts = {"long_form_answer_prompt":long_form_prompt}
faithfulness.set_prompts(**prompts)
```

---------

Co-authored-by: Jithin James <jamesjithin97@gmail.com>
@shahules786 shahules786 marked this pull request as ready for review September 10, 2024 14:12
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Sep 10, 2024
jjmachan
jjmachan approved these changes Sep 11, 2024
View reviewed changes
Copy link
Member

@jjmachan jjmachan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just the suggestion for doc change

docs/concepts/metrics/context_precision_v2.md Outdated Show resolved Hide resolved
docs/concepts/metrics/context_precision_v2.md Outdated Show resolved Hide resolved
docs/concepts/metrics/context_precision_v2.md Outdated Show resolved Hide resolved
shahules786 and others added 3 commits September 12, 2024 09:21
@shahules786 @jjmachan
Update docs/concepts/metrics/context_precision_v2.md
260cbf1 
Co-authored-by: Jithin James <jamesjithin97@gmail.com>
@shahules786 @jjmachan
Update docs/concepts/metrics/context_precision_v2.md
2244a9e 
Co-authored-by: Jithin James <jamesjithin97@gmail.com>
@shahules786 @jjmachan
Update docs/concepts/metrics/context_precision_v2.md
8401cc6 
Co-authored-by: Jithin James <jamesjithin97@gmail.com>
@shahules786 shahules786 merged commit 196127e into explodinggradients:main Sep 12, 2024
15 checks passed
shahules786 added a commit to shahules786/ragas that referenced this pull request Sep 12, 2024
@shahules786 @jjmachan
feat: Non LLM based context precision (explodinggradients#1264)
f8d8f2f 
1. Add Non LLM based context precision
2. Rename and introduce naming convention to metrics -
<llm_or_non_llm><metric_name><with_or_without_reference>

```python

from ragas import SingleTurnSample
from ragas.metrics._context_precision import NonLLMContextPrecisionWithReference

context_precision = NonLLMContextPrecisionWithReference()

sample = SingleTurnSample(
    retrieved_contexts=["The Eiffel Tower is located in Paris."], 
    reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."]
)

await context_precision.single_turn_ascore(sample)
```

---------

Co-authored-by: Jithin James <jamesjithin97@gmail.com>
shahules786 added a commit to shahules786/ragas that referenced this pull request Sep 12, 2024
@shahules786 @jjmachan
feat: Non LLM based context precision (explodinggradients#1264)
80c4fb4 
1. Add Non LLM based context precision
2. Rename and introduce naming convention to metrics -
<llm_or_non_llm><metric_name><with_or_without_reference>

```python

from ragas import SingleTurnSample
from ragas.metrics._context_precision import NonLLMContextPrecisionWithReference

context_precision = NonLLMContextPrecisionWithReference()

sample = SingleTurnSample(
    retrieved_contexts=["The Eiffel Tower is located in Paris."], 
    reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."]
)

await context_precision.single_turn_ascore(sample)
```

---------

Co-authored-by: Jithin James <jamesjithin97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jjmachan jjmachan jjmachan approved these changes

Assignees
No one assigned
Labels
size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shahules786 @jjmachan