Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

554 improve tokenization function #556

Closed
wants to merge 66 commits into from
Closed

554 improve tokenization function #556

wants to merge 66 commits into from

Conversation

VinciGit00
Copy link
Collaborator

No description provided.

VinciGit00 and others added 30 commits August 9, 2024 17:35
@VinciGit00
fix: broken node
1272273
@semantic-release-bot
ci(release): 1.13.0-beta.8 [skip ci]
b470d97 
## [1.13.0-beta.8](v1.13.0-beta.7...v1.13.0-beta.8) (2024-08-09)

### Bug Fixes

* broken node ([1272273](1272273))
@VinciGit00
feat: add refactoring of default temperature
6c3b37a
@VinciGit00
Update abstract_graph.py
c3f1520
@f-aguzzi
Merge pull request #529 from ScrapeGraphAI/523-set-default-temperatur…
a17e068 
…e-to-0

feat: add refactoring of default temperature
@semantic-release-bot
ci(release): 1.13.0-beta.9 [skip ci]
d4c1a1c 
## [1.13.0-beta.9](v1.13.0-beta.8...v1.13.0-beta.9) (2024-08-10)

### Features

* add refactoring of default temperature ([6c3b37a](6c3b37a))
@VinciGit00 @vedovati-matteo
refactoring of the code
8b2c266 
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
@VinciGit00
Merge branch 'pre/beta' into temp
0cda020
@VinciGit00
Merge pull request #532 from ScrapeGraphAI/temp
8423f10 
allignment
@vedovati-matteo
remove promps form helpers folders
4ca606c
@vedovati-matteo
Added new folder for prompts
3b5b24d
@vedovati-matteo
Changed prompts import from helpers to prompts
9fa3df2
@vedovati-matteo
fix: merge_anwser prompt import
f17cef9
@vedovati-matteo
Merge branch 'pre/beta' of https://github.com/vedovati-matteo/Scrapeg…
7c62100 
…raph-ai into pre/beta
@vedovati-matteo
fixed versioning errors
9814b6d
@vedovati-matteo
robot_node prompt moved
f455fcb
@vedovati-matteo
search_internet_node prompt moved
86a4903
@vedovati-matteo
search_link_node prompt moved
ef96652
@vedovati-matteo
search_link_node prompts moved
98779d1
@vedovati-matteo
Template name refactoring
734b740
@vedovati-matteo
quick fix typo
a3b7181
@VinciGit00
Merge pull request #535 from vedovati-matteo/pre/beta
a165fea 
Refactoring of prompts
@semantic-release-bot
ci(release): 1.14.0-beta.1 [skip ci]
40043f3 
## [1.14.0-beta.1](v1.13.3...v1.14.0-beta.1) (2024-08-11)

### Features

* add refactoring of default temperature ([6c3b37a](6c3b37a))

### Bug Fixes

* broken node ([1272273](1272273))
* merge_anwser prompt import ([f17cef9](f17cef9))

### CI

* **release:** 1.13.0-beta.8 [skip ci] ([b470d97](b470d97))
* **release:** 1.13.0-beta.9 [skip ci] ([d4c1a1c](d4c1a1c))
@VinciGit00 @vedovati-matteo
add new tests
cec5537 
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
@VinciGit00 @vedovati-matteo
refactoring pyproject.toml
de1ec25 
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
@VinciGit00 @vedovati-matteo
feat: update abstract graph
c77231c 
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
@VinciGit00 @vedovati-matteo
Update abstract_graph.py
cef2fdb 
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
@VinciGit00
Merge branch 'pre/beta' into refactoring-of-abstract-graph
8cece1d
@VinciGit00 @vedovati-matteo
fixing import bug
9e1d0f6 
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
@f-aguzzi @VinciGit00
fix(AbstractGraph): pass kwargs to Ernie and Nvidia models
e6bedb6 
Co-Authored-By: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com>
VinciGit00 and others added 26 commits August 12, 2024 18:32
@VinciGit00
Update abstract_graph.py
c105c26
@VinciGit00
Merge branch 'pre/beta' into refactoring-of-abstract-graph
795c932
@VinciGit00
Update abstract_graph.py
9be4474
@f-aguzzi
chore(examples): update provider names to match tokens dictionary
ee078cb
@f-aguzzi
chore(examples): add vertex examples, rename genai examples
1aa9c6e
@VinciGit00
Merge pull request #541 from ScrapeGraphAI/496-not-match-gemini-embed…
c5aebd7 
…dings-model
@semantic-release-bot
ci(release): 1.14.0-beta.3 [skip ci]
3bf9c3c 
## [1.14.0-beta.3](v1.14.0-beta.2...v1.14.0-beta.3) (2024-08-13)

### Bug Fixes

* **models_tokens:** incorrect provider names ([cb6b353](cb6b353))

### chore

* **examples:** add vertex examples, rename genai examples ([1aa9c6e](1aa9c6e))
* **examples:** update provider names to match tokens dictionary ([ee078cb](ee078cb))
@VinciGit00
fixed doc bugs and removed unused rag node
597f2ac
@VinciGit00
removed unused imports
203ee2c
@sandeepchittilla
feat: Add new feature to support gpt-4o variant models with different…
8551448 
… pricing
@f-aguzzi
Merge pull request #536 from ScrapeGraphAI/refactoring-of-abstract-graph
ea72d76 
Refactoring of abstract graph
@semantic-release-bot
ci(release): 1.14.0-beta.4 [skip ci]
7af1e45 
## [1.14.0-beta.4](v1.14.0-beta.3...v1.14.0-beta.4) (2024-08-15)

### Features

* update abstract graph ([c77231c](c77231c))
@VinciGit00
Merge pull request #552 from sandeepchittilla/550-add-support-gpt4o-v…
333f09a 
…ariants

Add new feature to support gpt-4o variant models with different pricing
@VinciGit00
refactoring of code and documentation
b115f96
@VinciGit00
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
5100368 
…ph-ai into pre/beta
@semantic-release-bot
ci(release): 1.14.0-beta.5 [skip ci]
db3494d 
## [1.14.0-beta.5](v1.14.0-beta.4...v1.14.0-beta.5) (2024-08-16)

### Features

* Add new feature to support gpt-4o variant models with different pricing ([8551448](8551448))
@VinciGit00
feat: add integration for new module of gpt4o
982150e
@VinciGit00
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
c722b2d 
…ph-ai into pre/beta
@semantic-release-bot
ci(release): 1.14.0-beta.6 [skip ci]
6730797 
## [1.14.0-beta.6](v1.14.0-beta.5...v1.14.0-beta.6) (2024-08-16)

### Features

* add integration for new module of gpt4o ([982150e](982150e))
@VinciGit00
fix: model count
faef318
@semantic-release-bot
ci(release): 1.14.0-beta.7 [skip ci]
a6fcc1e 
## [1.14.0-beta.7](v1.14.0-beta.6...v1.14.0-beta.7) (2024-08-16)

### Bug Fixes

* model count ([faef318](faef318))
@VinciGit00
add first idea
73fb797
@VinciGit00
add tiktoken tokenization
c7c97c1
@VinciGit00
add tokenization function for openai
1b8b438
@VinciGit00
feat: add tokenization for google
114032a
@VinciGit00
add dictionary for translating
8a92383
@VinciGit00 VinciGit00 linked an issue Aug 17, 2024 that may be closed by this pull request
improve tokenization function #554
Closed
@github-actions GitHub Actions
Copy link

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

PackageVersionScoreDetails
pip/google-generativeai 0.7.2 UnknownUnknown
pip/tiktoken 0.7.0 🟢 4.6
Details
CheckScoreReason
Code-Review🟢 3Found 10/30 approved changesets -- score normalized to 3
Maintained🟢 53 commit(s) and 4 issue activity found in the last 90 days -- score normalized to 5
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Packaging⚠️ -1packaging workflow not detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration
Security-Policy⚠️ 0security policy file not detected
Fuzzing⚠️ 0project is not fuzzed
Vulnerabilities🟢 100 existing vulnerabilities detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0

Scanned Manifest Files

pyproject.toml
  • google-generativeai@0.7.2
  • tiktoken@0.7.0
  • langchain-community@>= 0.2.9
  • tiktoken@>= 0.7
requirements.txt
  • langchain-community@>= 0.2.9

@VinciGit00 VinciGit00 closed this Aug 17, 2024
@f-aguzzi f-aguzzi mentioned this pull request Sep 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

improve tokenization function
6 participants
@VinciGit00 @semantic-release-bot @f-aguzzi @vedovati-matteo @AmazeQiu @sandeepchittilla