dlt.sources.filesystem.helpers

Helpers for the filesystem resource.

fsspec_from_resource

def fsspec_from_resource(
        filesystem_instance: DltResource) -> AbstractFileSystem

View source on GitHub

Extract authorized fsspec client from a filesystem resource

add_columns

def add_columns(columns: List[str],
                rows: List[List[Any]]) -> List[Dict[str, Any]]

View source on GitHub

Adds column names to the given rows.

Arguments:

• columns List[str] - The column names.
• rows List[List[Any]] - The rows.

Returns:

List[Dict[str, Any]]: The rows with column names.

fetch_arrow

def fetch_arrow(file_data, chunk_size: int) -> Iterable[TDataItem]

View source on GitHub

Fetches data from the given CSV file.

Arguments:

• file_data DuckDBPyRelation - The CSV file data.
• chunk_size int - The number of rows to read at once.

Yields:

• Iterable[TDataItem] - Data items, read from the given CSV file.

fetch_json

def fetch_json(file_data, chunk_size: int) -> List[Dict[str, Any]]

View source on GitHub

Fetches data from the given CSV file.

Arguments:

• file_data DuckDBPyRelation - The CSV file data.
• chunk_size int - The number of rows to read at once.

Yields:

• Iterable[TDataItem] - Data items, read from the given CSV file.