open_stream
open_stream(
path,
*,
chunk_size=5000,
assay=None,
layer='counts',
rows=None,
where=None,
axis='obs',
genes=None,
readahead_mb=128,
)Iterate over the matrix X in chunks of chunk_size rows, or of chunk_size genes with axis="var".
Decoding runs on a background thread ahead of the consumer: up to 8 chunks, and no more than readahead_mb of them, so peak memory is bounded by the chunk size and that budget, not the file.
rows and where select rows; with both, a row must satisfy both. On h5ad, Zarr, h5seurat (dgCMatrix) and 10x HDF5 only the selected rows are read (their part of the row pointer, then their slices of data/indices), so filtering costs what the matching rows cost; BPCells and MatrixMarket input is still read through. Chunks never span a gap in the selection, and chunk.row_offset is each chunk’s first row in the full matrix.
axis="var" streams blocks of whole genes over every cell, for per-gene work such as one GLM per gene. It needs X stored by gene: an h5ad with CSC X, as scx convert --gene-major writes. Other files raise rather than reading all of X per block. Each chunk’s to_scipy() is a csc_array of shape (n_obs, chunk.ncols), and chunk.var_offset is its first gene. genes selects genes the way rows selects rows, so jobs can split the genes between them.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| path | Pathish | File or directory in any format :func:convert reads. |
required |
| chunk_size | int | Maximum rows (cells) per chunk. | 5000 |
| assay | str | None | Seurat assay and layer to read from Seurat input. assay=None reads "RNA" from H5Seurat and the object’s active assay from a .rds. From MuData .h5mu assay names the modality (default: the only one, else "rna"). |
None |
| layer | str | None | Seurat assay and layer to read from Seurat input. assay=None reads "RNA" from H5Seurat and the object’s active assay from a .rds. From MuData .h5mu assay names the modality (default: the only one, else "rna"). |
None |
| rows | Rows | None | A boolean mask over all rows, a sequence of row indices, or a sequence of (start, stop) half-open ranges. |
None |
| where | Mapping[str, object] | None | {column: value} or {column: [values]} over obs: a row is kept when every column’s value is one of the given ones. Values are compared as text (str(v), booleans as True/False); NA never matches. |
None |
| axis | str | "obs" (default) to stream rows, "var" to stream genes. |
'obs' |
| genes | Rows | None | With axis="var": like rows, over genes. Default: all genes. |
None |
| readahead_mb | float | Most decoded data, in MB, the background thread holds ahead of the consumer. One chunk is always read ahead, however large. | 128 |
Returns
| Name | Type | Description |
|---|---|---|
| Iterator[MatrixChunk] | CSR row blocks, in row order. |
Examples
>>> for chunk in pk.open_stream("atlas.h5ad", chunk_size=5000):
... X = chunk.to_scipy() # csr_array sharing chunk.data / chunk.indices
>>> a549 = pk.open_stream("emt.h5ad", where={"CellLine": "A549", "Time": ["3d", "7d"]})
>>> # job i of 4 over a gene-major copy (scx convert atlas.h5ad atlas_by_gene.h5ad --gene-major)
>>> lo, hi = i * n_vars // 4, (i + 1) * n_vars // 4
>>> for chunk in pk.open_stream("atlas_by_gene.h5ad", axis="var", genes=[(lo, hi)], chunk_size=512):
... Y = chunk.to_scipy() # csc_array: cells x genes var_offset .. var_offset + ncols