open_stream

open_stream(
    path,
    *,
    chunk_size=5000,
    assay=None,
    layer='counts',
    rows=None,
    where=None,
    axis='obs',
    genes=None,
    readahead_mb=128,
)

Iterate over the matrix X in chunks of chunk_size rows, or of chunk_size genes with axis="var".

Decoding runs on a background thread ahead of the consumer: up to 8 chunks, and no more than readahead_mb of them, so peak memory is bounded by the chunk size and that budget, not the file.

rows and where select rows; with both, a row must satisfy both. On h5ad, Zarr, h5seurat (dgCMatrix) and 10x HDF5 only the selected rows are read (their part of the row pointer, then their slices of data/indices), so filtering costs what the matching rows cost; BPCells and MatrixMarket input is still read through. Chunks never span a gap in the selection, and chunk.row_offset is each chunk’s first row in the full matrix.

axis="var" streams blocks of whole genes over every cell, for per-gene work such as one GLM per gene. It needs X stored by gene: an h5ad with CSC X, as scx convert --gene-major writes. Other files raise rather than reading all of X per block. Each chunk’s to_scipy() is a csc_array of shape (n_obs, chunk.ncols), and chunk.var_offset is its first gene. genes selects genes the way rows selects rows, so jobs can split the genes between them.

Parameters

Name Type Description Default
path Pathish File or directory in any format :func:convert reads. required
chunk_size int Maximum rows (cells) per chunk. 5000
assay str | None Seurat assay and layer to read from Seurat input. assay=None reads "RNA" from H5Seurat and the object’s active assay from a .rds. From MuData .h5mu assay names the modality (default: the only one, else "rna"). None
layer str | None Seurat assay and layer to read from Seurat input. assay=None reads "RNA" from H5Seurat and the object’s active assay from a .rds. From MuData .h5mu assay names the modality (default: the only one, else "rna"). None
rows Rows | None A boolean mask over all rows, a sequence of row indices, or a sequence of (start, stop) half-open ranges. None
where Mapping[str, object] | None {column: value} or {column: [values]} over obs: a row is kept when every column’s value is one of the given ones. Values are compared as text (str(v), booleans as True/False); NA never matches. None
axis str "obs" (default) to stream rows, "var" to stream genes. 'obs'
genes Rows | None With axis="var": like rows, over genes. Default: all genes. None
readahead_mb float Most decoded data, in MB, the background thread holds ahead of the consumer. One chunk is always read ahead, however large. 128

Returns

Name Type Description
Iterator[MatrixChunk] CSR row blocks, in row order.

Examples

>>> for chunk in pk.open_stream("atlas.h5ad", chunk_size=5000):
...     X = chunk.to_scipy()  # csr_array sharing chunk.data / chunk.indices
>>> a549 = pk.open_stream("emt.h5ad", where={"CellLine": "A549", "Time": ["3d", "7d"]})
>>> # job i of 4 over a gene-major copy (scx convert atlas.h5ad atlas_by_gene.h5ad --gene-major)
>>> lo, hi = i * n_vars // 4, (i + 1) * n_vars // 4
>>> for chunk in pk.open_stream("atlas_by_gene.h5ad", axis="var", genes=[(lo, hi)], chunk_size=512):
...     Y = chunk.to_scipy()  # csc_array: cells x genes var_offset .. var_offset + ncols