import picklerick as pk
for chunk in pk.open_stream("tiny.h5ad", chunk_size=2):
print(chunk.row_offset, chunk.nrows, chunk.dtype, chunk.indptr.tolist())0 2 float32 [0, 2, 2]
2 2 float32 [0, 2, 4]
4 2 float32 [0, 1, 3]
open_stream, and read only the cells you select.
picklerick.open_stream hands you the count matrix as a sequence of CSR row blocks instead of one object in memory. Decoding runs on a background thread a few chunks ahead of your loop, so peak memory is set by the chunk size, not the file. This page is Python-only; the R package reaches the same engine through PicklerickMatrix(), a DelayedArray backend.
The example file, tiny.h5ad, has 6 cells and 5 genes. A chunk size of 2 is used so that the loop has several chunks to show; the default is 5000.
Each chunk is a MatrixChunk: NumPy arrays for the CSR row pointer, column indices and values, plus row_offset, the position of its first row in the full matrix.
chunk.to_scipy() wraps a chunk as a scipy.sparse.csr_array that shares its arrays, so any sparse operation works on it. Here the totals per cell are filled in chunk by chunk; inspect(..., names=True) gives the cell names without opening the file with anndata.
import numpy as np
info = pk.inspect("tiny.h5ad", names=True)
totals = np.zeros(info["n_obs"])
for chunk in pk.open_stream("tiny.h5ad", chunk_size=2):
rows = slice(chunk.row_offset, chunk.row_offset + chunk.nrows)
totals[rows] = chunk.to_scipy().sum(axis=1)
dict(zip(info["obs_names"], totals.tolist())){'cell0': 1.5,
'cell1': 0.0,
'cell2': 3.5,
'cell3': 5.5,
'cell4': 3.5,
'cell5': 8.5}
The same totals from the whole matrix in memory:
A per-gene statistic needs every cell, but not all of them at once: add each chunk’s contribution as it streams by. Counting the cells in which each gene is nonzero gives the usual “detected in at least one cell” filter, which is what rowSums(counts(sce) > 0) > 0 computes on a lazy matrix in R. Stored zeros are left out with chunk.data != 0.
{'gene0': 3, 'gene1': 2, 'gene2': 1, 'gene3': 1, 'gene4': 2}
The same counts from the matrix in memory:
keep can then select genes wherever they are used, e.g. adata[:, keep] once the data does fit, or the var mask of a later write.
where selects cells by their obs values and rows by position. On h5ad, Zarr, h5seurat (dgCMatrix) and 10x HDF5 files only the selected rows are read from disk, so a filter costs what the matching cells cost; BPCells and MatrixMarket input is still read through. A chunk never spans a gap in the selection, and row_offset still refers to the full matrix.
1 1
5 1
[1, 5]
Some work goes gene by gene over every cell: one model per gene, or per-gene statistics too wide to accumulate. Rows of a cell-major matrix hold genes scattered across every chunk, so for this the matrix is first written stored by gene (CSC). convert(..., gene_major=True) does that, in a bounded memory budget (transpose_mem, in GB); anndata reads the result as a csc_matrix.
'var'
open_stream(..., axis="var") then hands out blocks of whole genes. Each chunk’s to_scipy() is a csc_array of every cell by the block’s genes, and var_offset is its first gene.
0 2 (6, 2) [1.17, 1.0]
2 2 (6, 2) [0.5, 0.17]
4 1 (6, 1) [0.92]
genes selects genes the way rows selects cells, as (start, stop) ranges or indices, so independent jobs can each take a slice of the genes:
[(3, 2)]
See open_stream for every option.