Dpo Router
Defines a router to route the input into different processing pipelines based on certain criteria.
BaseDPORouter
Bases: ABC
Base class for routing in document processing.
route(*args, **kwargs)
abstractmethod
Routes the input into different processing pipelines based on certain criteria.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*args
|
Any
|
Variable length argument list for routing parameters. |
()
|
**kwargs
|
Any
|
Arbitrary keyword arguments for additional routing configuration. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The result of the routing process. |
ConverterRouter()
Bases: BaseDPORouter
Router that classifies a file path into a ConverterType.
The router only classifies — it does not decide which concrete converter to use. The pipeline maps each ConverterType to a concrete converter via its own converter mapping.
Attributes:
| Name | Type | Description |
|---|---|---|
logger |
Logger instance for this class. |
Initialize the ConverterRouter.
route(source, *args, **kwargs)
Classify the source into a ConverterType.
Maps the file extension (case-insensitive) to a ConverterType. Unknown extensions map to ConverterType.UNCATEGORIZED. Existence on disk is not checked.
Supported extensions: 1. .doc → ConverterType.DOC 2. .xls → ConverterType.XLS 3. .xlsb → ConverterType.XLSB
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
Local file path to classify. |
required |
*args
|
Any
|
Additional positional arguments (unused). |
()
|
**kwargs
|
Any
|
Additional keyword arguments (unused). |
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: A dict with the converter type. Example: {CONVERTER_TYPE_KEY: ConverterType.DOC} |
DownloaderRouter()
Bases: BaseDPORouter
Router that classifies a source into a DownloaderType.
The router only classifies — it does not decide which concrete downloader to use. The pipeline maps each DownloaderType to a concrete downloader via its own downloader_mapping.
Attributes:
| Name | Type | Description |
|---|---|---|
logger |
Logger instance for this class. |
Initialize the DownloaderRouter.
route(source, *args, **kwargs)
Classify the source into a DownloaderType.
Routing order (first match wins): 1. input_type in bypass map → mapped DownloaderType (skips all detection) 2. os.path.isfile(source) → DownloaderType.LOCAL_FILE 3. source starts with http(s) → YouTube, Google Drive, HEAD file probe, else web page 4. (default) → DownloaderType.UNCATEGORIZED
Supported input_type bypass values: "smart_crawl", "youtube_url", "direct_file_url".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
Local file path, URL, or Smart Crawl domain string. |
required |
*args
|
Any
|
Additional positional arguments (unused). |
()
|
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
input_type (str, optional): Bypass automatic detection. Supported values: "smart_crawl" → DownloaderType.SMART_CRAWL, "youtube_url" → DownloaderType.YOUTUBE_URL, "direct_file_url" → DownloaderType.DIRECT_FILE_URL. ca_certs_path (str, optional): Path to CA certificate file for SSL verification in the HEAD request.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: A dict with the downloader type. Example: {DOWNLOADER_TYPE_KEY: DownloaderType.WEB_PAGE_URL} |
LoaderRouter()
Bases: BaseDPORouter
Loader Router class.
This router determines the appropriate loader type based on the input source. Returns a dict with the loader type information.
Initialize the LoaderRouter.
This method initializes the LoaderRouter.
route(source, *args, **kwargs)
Route the input source to the appropriate loader type.
This method determines the appropriate loader type based on the input source. It checks if the source is a file or a YouTube URL. 1. If it is a file, it checks the file extension or content to determine the loader type. 2. If it is a YouTube URL, it returns the audio loader type. 3. If it is not a file or a YouTube URL, it returns the uncategorized loader type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The input source, either a file path or a YouTube URL. |
required |
*args
|
Any
|
Additional arguments. |
()
|
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
loader_type_overrides (dict[str, str], optional): Maps a detected loader type to the loader type to return instead. A detected type that is not a key is returned unchanged. Defaults to None.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: A dictionary containing the loader type information. Example: {LOADER_TYPE_KEY: LoaderType.PDF_LOADER} |
PDFTextLayerLoaderRouter(min_text_page_ratio=DEFAULT_MIN_TEXT_PAGE_RATIO)
Bases: LoaderRouter
Select PyMuPDF or GLAIR Vision OCR for PDF files.
A PDF is routed to PyMuPDF when the proportion of pages with non-whitespace extracted text meets the configured threshold. All other PDFs, including unreadable PDFs, are routed to GLAIR Vision OCR.
Initialize the PDF text-layer router.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_text_page_ratio
|
float
|
Minimum proportion of pages
that must contain extractable text for the PyMuPDF route.
Defaults to |
DEFAULT_MIN_TEXT_PAGE_RATIO
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the threshold is outside the inclusive range |
route(source, *args, **kwargs)
Route a PDF based on the proportion of pages with extractable text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The input source, usually a file path. |
required |
*args
|
Any
|
Additional arguments forwarded to |
()
|
**kwargs
|
Any
|
Additional keyword arguments forwarded to
|
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: The selected loader type. Non-PDF sources retain
the result returned by |
ParserRouter()
Bases: BaseDPORouter
Parser Router class.
This router determines the appropriate parser type based on the input source. Returns a dict with the parser type information.
Initialize the ParserRouter.
This method initializes the ParserRouter.
route(source, *args, **kwargs)
Determine the parser type from the input source.
The input source can be: - A string path to a JSON file containing loaded elements. - A list of loaded element dictionaries (in memory).
This method reads the input, extracts the source_type from the first element’s metadata,
and returns the appropriate parser type. If loading fails or metadata is missing,
the parser type will be set as UNCATEGORIZED.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | list[dict[str, Any]]
|
The input source, which can be either: - str: path to a JSON file containing loaded elements. - list[dict[str, Any]]: loaded elements. |
required |
*args
|
Any
|
Additional arguments. |
()
|
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: A dictionary containing the parser type information. Example: {PARSER_TYPE_KEY: ParserType.PDF_PARSER} |