Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve: VectorSearch enabled SQLChain? #7454

Closed
wants to merge 57 commits into from

Conversation

mpskex
Copy link
Contributor

@mpskex mpskex commented Jul 10, 2023

Hello from MyScale AI team! 😊👋

We have been working on features to fill up the gap among SQL, vector search and LLM applications. Some inspiring works like self-query retrievers for VectorStores (for example Weaviate and others) really turn those vector search databases into a powerful knowledge base! 🚀🚀

We are thinking if we can merge all in one, like SQL and vector search and LLMChains, making this SQL vector database memory as the only source of your data. Here are some benefits we can think of for now, maybe you have more 👀:

With ALL data you have: since you store all your pasta in the database, you don't need to worry about the foreign keys or links between names from other data source.
Flexible data structure: Even if you have changed your schema, for example added a table, the LLM will know how to JOIN those tables and use those as filters.
SQL compatibility: We found that vector databases that supports SQL in the marketplace have similar interfaces, which means you can change your backend with no pain, just change the name of the distance function in your DB solution and you are ready to go!

Issue resolved:

Change made in this PR:

  • An improved schema handling that ignore types.NullType columns
  • A SQL output Parser interface in SQLDatabaseChain to enable Vector SQL capability and further more
  • A Retriever based on SQLDatabaseChain to retrieve data from the database for RetrievalQAChains and many others
  • Allow SQLDatabaseChain to retrieve data in python native format
  • Includes PR Myscale/improve string pattern match #6737
  • Vector SQL Output Parser for SQLDatabaseChain and SQLDatabaseChainRetriever
  • Prompts that can implement text to VectorSQL
  • Corresponding unit-tests and notebook

Twitter handle:

  • @MyScaleDB

Tag Maintainer:

Prompts / General: @hwchase17, @baskaryan
DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev

Dependencies:

No dependency added

revert

revised to experimentals

try to resolve conflict

reverse change

try to resolve conflict

revised unittest

reformat & relint

revert

update poetry
@mpskex
Copy link
Contributor Author

mpskex commented Jul 31, 2023

I think we are pretty close now. I moved everything to experimental. Could you please review this? @hwchase17

@chtlp
Copy link

chtlp commented Aug 10, 2023

Some of our customers are requiring this feature, @baskaryan @hwchase17 please let us know how to make this feature get through.

Copy link
Contributor

@hwchase17 hwchase17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for the delayed review

since this touches the sql/base file its going to be a bit harder to review than is probably worth it

the easiest way to move forward is probably to make this a completely standalone module in experimental. eg no changes to the existing files. i dont entirely understand the changes, but either way they should probably be done in a separate pr

an end-to-end example notebook would also be helpful here

@mpskex
Copy link
Contributor Author

mpskex commented Aug 22, 2023

Thanks for the comment! I have already migrated this into experimental. And I should have added a notebook to demonstrate what it is doing. I will be working on it now. Thanks for this again ❤️

@mpskex
Copy link
Contributor Author

mpskex commented Aug 23, 2023

sorry for the delayed review

since this touches the sql/base file its going to be a bit harder to review than is probably worth it

the easiest way to move forward is probably to make this a completely standalone module in experimental. eg no changes to the existing files. i dont entirely understand the changes, but either way they should probably be done in a separate pr

an end-to-end example notebook would also be helpful here

I just added a notebook here: docs/extras/modules/data_connection/retrievers/sql_database/myscale_vector_sql.ipynb

@mpskex
Copy link
Contributor Author

mpskex commented Sep 4, 2023

Close because we squashed commits into new -> PR #10177

@mpskex mpskex closed this Sep 4, 2023
@mpskex mpskex deleted the myscale/sql_self_query branch September 4, 2023 10:50
baskaryan pushed a commit that referenced this pull request Sep 7, 2023
Squashed from #7454 with updated features

We have separated the `SQLDatabseChain` from `VectorSQLDatabseChain` and
put everything into `experimental/`.

Below is the original PR message from #7454.

-------

We have been working on features to fill up the gap among SQL, vector
search and LLM applications. Some inspiring works like self-query
retrievers for VectorStores (for example
[Weaviate](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/weaviate_self_query.html)
and
[others](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query.html))
really turn those vector search databases into a powerful knowledge
base! 🚀🚀

We are thinking if we can merge all in one, like SQL and vector search
and LLMChains, making this SQL vector database memory as the only source
of your data. Here are some benefits we can think of for now, maybe you
have more 👀:

With ALL data you have: since you store all your pasta in the database,
you don't need to worry about the foreign keys or links between names
from other data source.
Flexible data structure: Even if you have changed your schema, for
example added a table, the LLM will know how to JOIN those tables and
use those as filters.
SQL compatibility: We found that vector databases that supports SQL in
the marketplace have similar interfaces, which means you can change your
backend with no pain, just change the name of the distance function in
your DB solution and you are ready to go!

### Issue resolved:
- [Feature Proposal: VectorSearch enabled
SQLChain?](#5122)

### Change made in this PR:
- An improved schema handling that ignore `types.NullType` columns 
- A SQL output Parser interface in `SQLDatabaseChain` to enable Vector
SQL capability and further more
- A Retriever based on `SQLDatabaseChain` to retrieve data from the
database for RetrievalQAChains and many others
- Allow `SQLDatabaseChain` to retrieve data in python native format
- Includes PR #6737 
- Vector SQL Output Parser for `SQLDatabaseChain` and
`SQLDatabaseChainRetriever`
- Prompts that can implement text to VectorSQL
- Corresponding unit-tests and notebook

### Twitter handle: 
- @MyScaleDB

### Tag Maintainer:
Prompts / General: @hwchase17, @baskaryan
DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev

### Dependencies:
No dependency added
rsharath pushed a commit to getjavelin/langchain that referenced this pull request Sep 8, 2023
Squashed from langchain-ai#7454 with updated features

We have separated the `SQLDatabseChain` from `VectorSQLDatabseChain` and
put everything into `experimental/`.

Below is the original PR message from langchain-ai#7454.

-------

We have been working on features to fill up the gap among SQL, vector
search and LLM applications. Some inspiring works like self-query
retrievers for VectorStores (for example
[Weaviate](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/weaviate_self_query.html)
and
[others](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query.html))
really turn those vector search databases into a powerful knowledge
base! 🚀🚀

We are thinking if we can merge all in one, like SQL and vector search
and LLMChains, making this SQL vector database memory as the only source
of your data. Here are some benefits we can think of for now, maybe you
have more 👀:

With ALL data you have: since you store all your pasta in the database,
you don't need to worry about the foreign keys or links between names
from other data source.
Flexible data structure: Even if you have changed your schema, for
example added a table, the LLM will know how to JOIN those tables and
use those as filters.
SQL compatibility: We found that vector databases that supports SQL in
the marketplace have similar interfaces, which means you can change your
backend with no pain, just change the name of the distance function in
your DB solution and you are ready to go!

### Issue resolved:
- [Feature Proposal: VectorSearch enabled
SQLChain?](langchain-ai#5122)

### Change made in this PR:
- An improved schema handling that ignore `types.NullType` columns 
- A SQL output Parser interface in `SQLDatabaseChain` to enable Vector
SQL capability and further more
- A Retriever based on `SQLDatabaseChain` to retrieve data from the
database for RetrievalQAChains and many others
- Allow `SQLDatabaseChain` to retrieve data in python native format
- Includes PR langchain-ai#6737 
- Vector SQL Output Parser for `SQLDatabaseChain` and
`SQLDatabaseChainRetriever`
- Prompts that can implement text to VectorSQL
- Corresponding unit-tests and notebook

### Twitter handle: 
- @MyScaleDB

### Tag Maintainer:
Prompts / General: @hwchase17, @baskaryan
DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev

### Dependencies:
No dependency added
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🤖:docs Changes to documentation and examples, like .md, .rst, .ipynb files. Changes to the docs/ folder 🤖:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants