text-splitters: bug fix for CharacterTextSplitter replacing original separator with regex pattern when merging

[wulifu2hao]

• Description:
  bug: currently when using CharacterTextSplitter with regex pattern as separator (e.g. "\s") and keep_separator=False to split a text like "Hello world", it will end up with "Hello\sworld" (assume chunk_size=15 so after the splitting it will need to merge it back)

  changes:

  1. "_split_text" function will use the actual original separators to merge split strings, instead of the regex pattern
  2. "_split_text_with_regex" function will return not just the split strings but also the original separators
  3. "_merge_splits" function of TextSplitter in base.py will pick the right separators to join two split strings instead of always using one separator (which could be the regex pattern)
  4. "_join_docs" function of TextSplitter in base.py, similarly, allows passing in a list of separators the facilitate the joining of strings required in the "_merge_splits" function

Issue: #23394

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Sep 21, 2024 0:08am

[wulifu2hao]

@baskaryan pls do help review when you have time , thanks :)

new tests failing

[efriis]

thoughts on these new tests?