Flowwise Responsive time for Q&A

[trepont]

How to tune the response generated by the flowwise window? we are using Pinecone, OpenAI,PDF loader when I am trying get the answers from the document it was taking 10 sec

[HenryHengZJ]

When you use Pinecone upsert and ask a question in the chat box for the first time, it will upsert the document to Pinecone and that's why it's taking time. Every subsequent question will be fast since upsert is not done again, only the first time, unless the document has been changed. However it is advised to create another chatflow to load existing index if the upsert has already been done before

[trepont]

Thank you, Hendry, for your assistance. I have successfully utilized Chatflow to load the existing index of PineCone. The remarkable outcome is that I am now receiving responses in less than 3 seconds

[tristinbaileel]

I'm new to this so please correct me if I'm wrong in any areas and apologies if you already know this but the quality/accuracy of my responses were impacted the most when changing parameters like chunk and batch size, here's an overview of everything I've learned
Accuracy: The quality or accuracy of the responses generated by the AI system depends on various factors. When you're using the Load Existing Index method, the AI is retrieving data from a pre-existing index in Pinecone's vector store. The relevance and accuracy of its responses will depend on the quality of the information already stored in the index. If the index contains high-quality, relevant data, then the AI system will likely generate accurate and relevant responses. Conversely, if the index contains outdated or irrelevant information, then the accuracy of the responses may suffer.
In contrast, when you're using the Upsert Document method, you're inputting new information into the system. The quality of the responses in this case depends more directly on the quality of the new data you're inputting. If the document you're inserting contains accurate, detailed, and relevant information, then the AI system can generate high-quality responses based on this information.
In both cases, the accuracy of the responses also depends on the effectiveness of the language model (like GPT-3.5 Turbo) and the quality of the embeddings generated by the model like Ada.
Token Limitations and Rate Limits: The models have token and request limitations, defined as TPM (Tokens Per Minute) and RPM (Requests Per Minute). Ada model has an RPM of 3000 and a TPM of 250,000, whereas the GPT-3.5 Turbo has an RPM of 3,500 and a TPM of 180,000. These limits determine how many requests you can make per minute and how many tokens you can process per minute, respectively.
Batch Size: The batch size refers to the number of data samples (in this case, text inputs) processed together. Larger batch sizes allow for faster processing but may lead to increased latency. The optimal batch size depends on the specific requirements of your use case, the computational resources available, and the model's rate limits.
Chunk Size: When splitting text into chunks, the size of these chunks (measured in tokens) is an important consideration. For the Ada model with a context window of 8192 tokens, the chunk size for the text splitter should ideally be around 8000-8100 tokens. If a chunk exceeds the model's context window size, the context gets truncated, which could lead to loss of information and reduced quality of the embeddings.
Embeddings: The Ada model transforms text into high-dimensional vectors or embeddings, which are then stored in Pinecone's vector store. The quality of these embeddings is crucial for the performance of the retrieval system. The Ada model produces 1536-dimensional embeddings, which are smaller and more efficient than previous versions.
Strip New Lines, Timeout: 'Strip New Lines' is an option in the Ada model which, when turned on, removes new line characters from the input text. 'Timeout' is the maximum time to wait for the model to generate embeddings.

[trunghuynh228]

create another chatflow to load existing index if the upsert has already been done before

Could you please share on how to do this when using a in memory vector store? I upserted the documents before then start chat, the first chat is still upserting again