Parser

Document Processing Orchestrator Parser Package.

Modules:

Name	Description
BaseParser	Abstract base class for document parser.
MediaFilterParser	Module for filtering media from elements by MIME type and file extension.
PipelineParser	Module for parsing document with several parsers using pipeline.

BaseParser

Bases: ABC

Base class for document parser.

This class serves as the base for document parser, which will define the structure for every content of document.

Methods:

Name	Description
parse	Abstract method to parse a document.

parse(loaded_elements, **kwargs) abstractmethod

Parse loaded elements to get element structure.

This method is abstract and must be implemented in subclasses. It defines the process of parsing a document using loaded elements.

Parameters:

Name	Type	Description	Default
loaded_elements	Any	The loaded elements from loader. ideally formatted as List[Dict].	required
**kwargs	Any	Additional keyword arguments for customization.	{}

Returns:

Name	Type	Description
Any	Any	The parsed document, ideally formatted as List[Dict]. Each dictionary within the list are recommended to follows the structure of model 'Element', to ensure consistency and ease of use across Document Processing Orchestrator.

MediaFilterParser(filtered_mime_types, filtered_extensions)

Bases: BaseParser

Parser that filters out media from elements based on configurable criteria.

This parser removes media items from element metadata if they match the configured MIME types or file extensions. It filters both URL-based and base64 media.

A media item is removed when any of these match: 1. Data URI MIME type (e.g., data:image/svg+xml;base64,...), for both URL and base64 media 2. URL path extension (e.g., .svg), for URL media only 3. The url query parameter of a proxy URL (e.g. ~gitbook/image?url=https://.../icon.svg), which is the only nested parameter name recognised

Note: HTTP HEAD requests for Content-Type header checking are not performed to avoid performance issues with HTML pages containing many image URLs.

Attributes:

Name	Type	Description
filtered_mime_types	list[str]	List of MIME types to filter (e.g., ["image/svg+xml"]). Can be empty if filtered_extensions is provided.
filtered_extensions	list[str]	List of file extensions to filter (e.g., [".svg"]). Can be empty if filtered_mime_types is provided.

Initialize the MediaFilterParser.

Parameters:

Name	Type	Description	Default
filtered_mime_types	list[str]	List of MIME types to filter (e.g., ["image/svg+xml"]). Can be empty if filtered_extensions is provided.	required
filtered_extensions	list[str]	List of file extensions to filter (e.g., [".svg"]). Can be empty if filtered_mime_types is provided.	required

Raises:

Type	Description
ValueError	If both filtered_mime_types and filtered_extensions are empty. An empty configuration is not a no-op: the element drop in parse still removes media-only elements, so it is rejected rather than silently applied.

parse(loaded_elements, **kwargs)

Parse and filter out media from elements based on configured criteria.

Parameters:

Name	Type	Description	Default
loaded_elements	list[dict[str, Any]]	A list of elements to parse.	required
**kwargs	Any	Additional keyword arguments (unused).	{}

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: Elements with filtered media removed.

PipelineParser()

Pipeline parser for parsing documents.

This class serves as the pipeline parser for parsing documents. It defines the structure for parsing documents with several parsers using pipeline.

Methods:

Name	Description
add_parser	Add parser to the pipeline parser.
parse	Parse the elements using parsers.

Initialize the pipeline parser.

add_parser(parser)

Add parser to the pipeline parser.

This method defines the process of adding parser to the pipeline parser.

Parameters:

Name	Type	Description	Default
parser	BaseParser	The parser to be added.	required

parse(elements, **kwargs)

Parse the elements using pipeline parser.

This method defines the process of parsing the elements using parsers.

Parameters:

Name	Type	Description	Default
elements	list[dict[str, Any]]	A list of dictionaries containing elements.	required
**kwargs	Any	Additional keyword arguments.	{}