# DocProcessor

The DocProcessor is responsible for various processing tasks on Document, such as text splitting, keyword extraction. The input and output of DocProcessor are both List[Document], ensuring that multiple DocProcessors can be stacked to form a processing pipeline for the Document.

The Document is defined as follows:
```python
import uuid
from typing import Dict, Any, Optional, List, Set

from pydantic import BaseModel, Field, model_validator


class Document(BaseModel):
    class Config:
        arbitrary_types_allowed = True

    id: str = None
    text: Optional[str] = ""
    metadata: Optional[Dict[str, Any]] = None
    embedding: List[float] = Field(default_factory=list)
    keywords: Set[str] = Field(default_factory=set)

    @model_validator(mode='before')
    def create_id(cls, values):
        text: str = values.get('text', '')
        if not values.get('id'):
            values['id'] = str(uuid.uuid5(uuid.NAMESPACE_URL, text))
        return values
```
- id: A unique identifier for a specific document, generated by default using uuid.
- text: The text content of the document.
- metadata: Metadata information of the document, typically including the original filename, the position in the original file, etc.
- embedding: The vectorized form of the document, which could be a text vector, or in a subclass like ImageDocument, it could be the vectorized result of an image.
- keywords: Keywords extracted from the document, which can also serve as tags for the text.

The DocProcessor is defined as follows:
```python
from abc import abstractmethod
from typing import List, Optional

from agentuniverse.agent.action.knowledge.store.query import Query
from agentuniverse.agent.action.knowledge.store.document import Document
from agentuniverse.base.component.component_base import ComponentEnum
from agentuniverse.base.component.component_base import ComponentBase

class DocProcessor(ComponentBase):
    component_type: ComponentEnum = ComponentEnum.DOC_PROCESSOR
    name: Optional[str] = None
    description: Optional[str] = None

    def process_docs(self, origin_docs: List[Document], query: Query = None) -> \
            List[Document]:
        return self._process_docs(origin_docs, query)

    @abstractmethod
    def _process_docs(self, origin_docs: List[Document],
                      query: Query = None) -> \
            List[Document]:
        pass
```
Users need to mainly override the _process_docs function in their custom DocProcessor to implement the specific logic for processing Document.

After writing the corresponding code, you can refer to the following YAML configuration to register your DocProcessor as an agentUniverse component:
```yaml
name: 'dashscope_reranker'
description: 'reranker use dashscope api'
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.dashscope_reranker'
  class: 'DashscopeReranker'
```
The type in the metadata must be DOC_PROCESSOR.

### Pay Attention to the Package Path of Your Custom DocProcessor
In the config.toml file of the agentUniverse project, you need to configure the package path corresponding to your DocProcessor. Please double-check whether the path of the file you created is under the doc_processor path or its subdirectories in CORE_PACKAGE.

For example, the configuration in the sample project is as follows:
```yaml
[CORE_PACKAGE]
doc_processor = ['sample_standard_app.intelligence.agentic.knowledge.doc_processor']
```


## The following DocProcessors are built into agentUniverse:
### [CharacterTextSplitter](../../../../../../agentuniverse/agent/action/knowledge/doc_processor/character_text_splitter.yaml)
This component splits the original text based on the number of characters.
The component definition file is as follows:
```yaml
name: 'character_text_splitter'
description: 'langchain character text splitter'
chunk_size: 200
chunk_overlap: 20
separators: "/n/n"
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.character_text_splitter'
  class: 'CharacterTextSplitter'
```
- chunk_size: The size of the text chunks after splitting.
- chunk_overlap: The length of the overlapping part between adjacent chunks.
- separators: The specified separator.

### [TokenTextSplitter](../../../../../../agentuniverse/agent/action/knowledge/doc_processor/character_text_splitter.yaml)
This component splits the text based on the specified tokenizer, splitting the text into multiple segments according to the set chunk_size and chunk_overlap, where each segment contains a specified number of tokens.

The component definition file is as follows:

```yaml
name: 'token_text_splitter'
description: 'langchain token text splitter'
chunk_size: 200
chunk_overlap: 20
tokenizer: 'default_tokenizer'
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.token_text_splitter'
  class: 'TokenTextSplitter'
```
- chunk_size: The number of tokens in the text after splitting.
- chunk_overlap: The number of overlapping tokens between adjacent chunks.
- tokenizer: The specified tokenizer used to split the text into tokens.

### [RecursiveCharacterTextSplitter](../../../../../../agentuniverse/agent/action/knowledge/doc_processor/recursive_character_text_splitter.yaml)

This component recursively splits the original text based on the specified separators. It first tries to split the text using the separator with the highest priority. If it fails to meet the chunk_size requirement, it will recursively use the next separator until the text is successfully split.

The component definition file is as follows:
```yaml
name: 'recursive_character_text_splitter'
description: 'langchain recursive character text splitter'
chunk_size: 200
chunk_overlap: 20
separators:
  - "\n\n"
  - "\n"
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.recursive_character_text_splitter'
  class: 'RecursiveCharacterTextSplitter'
```
- chunk_size: The size of the text chunks after splitting.
- chunk_overlap: The length of the overlapping part between adjacent chunks.
- separators: A list of specified separators. The component will try to split using the separators in order. If the first separator cannot meet the conditions, it will recursively use the next separator.

### [JiebaKeywordExtractor](../../../../../../agentuniverse/agent/action/knowledge/doc_processor/jieba_keyword_extractor.yaml)
This component uses the Jieba segmentation library to extract keywords from the text. It can extract the most important keywords based on the set top_k parameter, which can be used later to build an inverted index.
The component definition file is as follows:
```yaml
name: 'jieba_keyword_extractor'
description: 'extract keywords from text'
top_k: 3
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.jieba_keyword_extractor'
  class: 'JiebaKeywordExtractor'
```
- top_k: The number of keywords to extract from the text, meaning the  top_k keywords will be extracted.

### [DashscopeReranker](../../../../../../agentuniverse/agent/action/knowledge/doc_processor/dashscope_reranker.yaml)

This component uses the DashScope API to rerank texts, sorting the content recalled by the Store based on the relevance to the Query content.

The component definition file is as follows:
```yaml
name: 'dashscope_reranker'
description: 'reranker use dashscope api'
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.dashscope_reranker'
  class: 'DashscopeReranker'
```
This component requires configuring DASHSCOPE_API_KEY in the environment variables.

### [HierarchicalRegexTextSplitter](../../../../../../agentuniverse/agent/action/knowledge/doc_processor/hierarchical_regex_text_splitter.py)

This component splits the original text into multiple hierarchical levels using specified regex rules, forming a tree-like document structure. Users need to create a custom definition file, with an example as follows:
```yaml
name: 'hierarchical_regex_text_splitter'
description: 'extract keywords from query'
merge_first: True
hierarchical_index:
  - "reg_exp": "第[零一二三四五六七八九十百千]+章"
    "need_summary": True
  - "reg_exp": "第[零一二三四五六七八九十百千]+条"
    "need_summary": False
summary_agent: "simple_summary_agent"
llm:
  name: qwen_llm
  model_name: qwen-plus
metadata:
  type: 'DOC_PROCESSOR'
  module: 'agentuniverse.agent.action.knowledge.doc_processor.hierarchical_regex_text_splitter'
  class: 'HierarchicalRegexTextSplitter'
```
- merge_first: If set to True, it merges the input List[Document] into a single document before splitting.
- hierarchical_index: Represents the splitting rules for different levels. `reg_exp` is the regular expression, and if `need_summary` is set to True, the summary text will replace the original text at that level.
- summary_agent: The Agent responsible for generating the summary text when `need_summary` is set to True in `hierarchical_index`. The default is simple_summary_agent.
- llm: If specified, it will replace the original LLM in the `summary_agent`.