Issue in BaseChunker.ts

[Jaikant]

createChunks function has an asynchronous operation running within a forEach loop. The async function inside forEach doesn't make the loop wait for the promise to resolve. As a result, the function would return an empty arrays of documents, ids, metadatas, before they are fully populated.

[cachho]

Good point, thank you, but this is eslint restricted syntax. Do you not see the warnings?

[cachho]

How does this look as an alternative?

import { createHash } from 'crypto';
import type { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';

import type { BaseLoader } from '../loaders';
import type { Input, LoaderResult } from '../models';
import type { ChunkResult } from '../models/ChunkResult';

class BaseChunker {
  textSplitter: RecursiveCharacterTextSplitter;

  constructor(textSplitter: RecursiveCharacterTextSplitter) {
    this.textSplitter = textSplitter;
  }

  async createChunks(loader: BaseLoader, url: Input): Promise<ChunkResult> {
    const documents: ChunkResult['documents'] = [];
    const ids: ChunkResult['ids'] = [];
    const datas: LoaderResult = await loader.loadData(url);
    const metadatas: ChunkResult['metadatas'] = [];

    const dataPromises = datas.map(async (data) => {
      const { content, metaData } = data;
      const chunks: string[] = await this.textSplitter.splitText(content);
      chunks.forEach((chunk) => {
        const chunkId = createHash('sha256')
          .update(chunk + metaData.url)
          .digest('hex');
        ids.push(chunkId);
        documents.push(chunk);
        metadatas.push(metaData);
      });
    });

    await Promise.all(dataPromises);

    return {
      documents,
      ids,
      metadatas,
    };
  }
}

export { BaseChunker };

[Jaikant]

Yes this should work too!

[cachho]

Do you want to update it? It's your PR :)

[cachho]

did the changes, merging now. please check on your end if everything works as intended. thanks.