Kotaemon chunking method ?

[dromeuf]

In Ressources -> File we can fix chunk_size and chunk overlap

chunk_size : Number of tokens of each text segment. Set 0 to use developer setting.
chunk_overlap : Number of tokens that consecutive text segments should overlap with each other. Set 0 to use developer setting.

The unit is tokens or characters ? default devel chunks size is 1024 characters or tokens ? Default devel overlaps is 256 tokens or characters ?

Kind regards, David.

I can find many value with this grep 1024, 200, 30, ...

dromeuf@MAIA:~/kotaemon_env$ grep -rni "chunk_size" --include="*.py" ./libs/                                                              
./libs/kotaemon/kotaemon/embeddings/openai.py:20:def split_text_by_chunk_size(text: str, chunk_size: int) -> list[list[int]]:             
./libs/kotaemon/kotaemon/embeddings/openai.py:25:        chunk_size: size of each chunk                                                   
./libs/kotaemon/kotaemon/embeddings/openai.py:33:    while chunk := list(islice(tokens, chunk_size)):                                     
./libs/kotaemon/kotaemon/embeddings/openai.py:97:                chunks = split_text_by_chunk_size(text.text or " ", self.context_length) 
./libs/kotaemon/kotaemon/indices/ingests/files.py:83:        chunk_size=1024,                                                             
./libs/kotaemon/kotaemon/indices/rankings/llm_trulens.py:103:        chunk_size=MAX_CONTEXT_LEN,                                          
./libs/kotaemon/kotaemon/indices/splitters/__init__.py:13:        chunk_size: int = 1024,                                                 
./libs/kotaemon/kotaemon/indices/splitters/__init__.py:19:            chunk_size=chunk_size,                                              
./libs/kotaemon/kotaemon/indices/qa/format_context.py:38:                chunk_size=self.max_context_length,                              
./libs/kotaemon/kotaemon/agents/react/agent.py:155:                chunk_size=self.max_context_length,                                    
./libs/kotaemon/kotaemon/agents/rewoo/agent.py:250:                chunk_size=self.max_context_length,                                    
./libs/kotaemon/tests/test_splitter.py:44:    splitter = TokenSplitter(chunk_size=30, chunk_overlap=10)                                   
./libs/kotaemon/tests/test_ingestor.py:11:        text_splitter=TokenSplitter(chunk_size=200, chunk_overlap=10),                          
./libs/kotaemon/tests/test_reader.py:52:    node_parser = SimpleNodeParser.from_defaults(chunk_size=100, chunk_overlap=20)                
./libs/ktem/ktem/index/file/index.py:419:            "chunk_size": {                                                                      
./libs/ktem/ktem/index/file/index.py:457:        obj.chunk_size = self.config.get("chunk_size", 0)                                        
./libs/ktem/ktem/index/file/pipelines.py:66:    chunk_size = None                                                                         
./libs/ktem/ktem/index/file/pipelines.py:67:    if hasattr(settings, "FILE_INDEX_PIPELINE_SPLITTER_CHUNK_SIZE"):                          
./libs/ktem/ktem/index/file/pipelines.py:68:        chunk_size = settings.FILE_INDEX_PIPELINE_SPLITTER_CHUNK_SIZE                         
./libs/ktem/ktem/index/file/pipelines.py:74:    return file_extractors, chunk_size, chunk_overlap                                         
./libs/ktem/ktem/index/file/pipelines.py:388:        chunk_size = self.chunk_batch_size * 4                                               
./libs/ktem/ktem/index/file/pipelines.py:389:        for start_idx in range(0, len(to_index_chunks), chunk_size):                         
./libs/ktem/ktem/index/file/pipelines.py:390:            chunks = to_index_chunks[start_idx : start_idx + chunk_size]                     
./libs/ktem/ktem/index/file/pipelines.py:401:            chunk_size = self.chunk_batch_size                                               
./libs/ktem/ktem/index/file/pipelines.py:402:            for start_idx in range(0, len(to_index_chunks), chunk_size):                     
./libs/ktem/ktem/index/file/pipelines.py:403:                chunks = to_index_chunks[start_idx : start_idx + chunk_size]                 
./libs/ktem/ktem/index/file/pipelines.py:734:        _, dev_chunk_size, dev_chunk_overlap = dev_settings()                                
./libs/ktem/ktem/index/file/pipelines.py:736:        chunk_size = self.chunk_size or dev_chunk_size                                       
./libs/ktem/ktem/index/file/pipelines.py:752:        print(f"Chunk size: {chunk_size}, chunk overlap: {chunk_overlap}")                   
./libs/ktem/ktem/index/file/pipelines.py:758:                chunk_size=chunk_size or 1024,                                               
./libs/ktem/ktem/index/file/base.py:58:    chunk_size = Param(help="Chunk size for this index")                                           
./libs/ktem/ktem/reasoning/react.py:108:                chunk_size=trim_len,                                                              
./libs/ktem/ktem/reasoning/rewoo.py:146:                chunk_size=trim_len,                                        

dromeuf@MAIA:~/kotaemon_env$ grep -rni "chunk_overlap" --include="*.py" ./libs/
./libs/kotaemon/kotaemon/indices/ingests/files.py:84:        chunk_overlap=256,
./libs/kotaemon/kotaemon/indices/rankings/llm_trulens.py:104:        chunk_overlap=0,
./libs/kotaemon/kotaemon/indices/splitters/__init__.py:14:        chunk_overlap: int = 20,
./libs/kotaemon/kotaemon/indices/splitters/__init__.py:20:            chunk_overlap=chunk_overlap,
./libs/kotaemon/kotaemon/indices/qa/format_context.py:39:                chunk_overlap=0,
./libs/kotaemon/kotaemon/agents/react/agent.py:156:                chunk_overlap=0,
./libs/kotaemon/kotaemon/agents/rewoo/agent.py:251:                chunk_overlap=0,
./libs/kotaemon/tests/test_splitter.py:44:    splitter = TokenSplitter(chunk_size=30, chunk_overlap=10)
./libs/kotaemon/tests/test_ingestor.py:11:        text_splitter=TokenSplitter(chunk_size=200, chunk_overlap=10),
./libs/kotaemon/tests/test_reader.py:52:    node_parser = SimpleNodeParser.from_defaults(chunk_size=100, chunk_overlap=20)
./libs/ktem/ktem/index/file/index.py:428:            "chunk_overlap": {
./libs/ktem/ktem/index/file/index.py:458:        obj.chunk_overlap = self.config.get("chunk_overlap", 0)
./libs/ktem/ktem/index/file/pipelines.py:70:    chunk_overlap = None
./libs/ktem/ktem/index/file/pipelines.py:71:    if hasattr(settings, "FILE_INDEX_PIPELINE_SPLITTER_CHUNK_OVERLAP"):
./libs/ktem/ktem/index/file/pipelines.py:72:        chunk_overlap = settings.FILE_INDEX_PIPELINE_SPLITTER_CHUNK_OVERLAP
./libs/ktem/ktem/index/file/pipelines.py:74:    return file_extractors, chunk_size, chunk_overlap
./libs/ktem/ktem/index/file/pipelines.py:734:        _, dev_chunk_size, dev_chunk_overlap = dev_settings()
./libs/ktem/ktem/index/file/pipelines.py:737:        chunk_overlap = self.chunk_overlap or dev_chunk_overlap
./libs/ktem/ktem/index/file/pipelines.py:752:        print(f"Chunk size: {chunk_size}, chunk overlap: {chunk_overlap}")
./libs/ktem/ktem/index/file/pipelines.py:759:                chunk_overlap=chunk_overlap or 256,
./libs/ktem/ktem/index/file/base.py:59:    chunk_overlap = Param(help="Chunk overlap for this index")
./libs/ktem/ktem/reasoning/react.py:109:                chunk_overlap=0,
./libs/ktem/ktem/reasoning/rewoo.py:147:                chunk_overlap=0,