Stream a matrix in Python

Process a matrix chunk by chunk with open_stream, and read only the cells you select.

picklerick.open_stream hands you the count matrix as a sequence of CSR row blocks instead of one object in memory. Decoding runs on a background thread a few chunks ahead of your loop, so peak memory is set by the chunk size, not the file. This page is Python-only; the R package reaches the same engine through PicklerickMatrix(), a DelayedArray backend.

The example file, tiny.h5ad, has 6 cells and 5 genes. A chunk size of 2 is used so that the loop has several chunks to show; the default is 5000.

Chunks

Each chunk is a MatrixChunk: NumPy arrays for the CSR row pointer, column indices and values, plus row_offset, the position of its first row in the full matrix.

import picklerick as pk

for chunk in pk.open_stream("tiny.h5ad", chunk_size=2):
    print(chunk.row_offset, chunk.nrows, chunk.dtype, chunk.indptr.tolist())
0 2 float32 [0, 2, 2]
2 2 float32 [0, 2, 4]
4 2 float32 [0, 1, 3]

Per-cell totals without loading the matrix

chunk.to_scipy() wraps a chunk as a scipy.sparse.csr_array that shares its arrays, so any sparse operation works on it. Here the totals per cell are filled in chunk by chunk; inspect(..., names=True) gives the cell names without opening the file with anndata.

import numpy as np

info = pk.inspect("tiny.h5ad", names=True)
totals = np.zeros(info["n_obs"])
for chunk in pk.open_stream("tiny.h5ad", chunk_size=2):
    rows = slice(chunk.row_offset, chunk.row_offset + chunk.nrows)
    totals[rows] = chunk.to_scipy().sum(axis=1)

dict(zip(info["obs_names"], totals.tolist()))
{'cell0': 1.5,
 'cell1': 0.0,
 'cell2': 3.5,
 'cell3': 5.5,
 'cell4': 3.5,
 'cell5': 8.5}

The same totals from the whole matrix in memory:

import anndata as ad

adata = ad.read_h5ad("tiny.h5ad")
np.allclose(totals, np.asarray(adata.X.sum(axis=1)).ravel())
True

Filtering genes in one pass

A per-gene statistic needs every cell, but not all of them at once: add each chunk’s contribution as it streams by. Counting the cells in which each gene is nonzero gives the usual “detected in at least one cell” filter, which is what rowSums(counts(sce) > 0) > 0 computes on a lazy matrix in R. Stored zeros are left out with chunk.data != 0.

n_cells = np.zeros(info["n_vars"], dtype=np.int64)
for chunk in pk.open_stream("tiny.h5ad", chunk_size=2):
    n_cells += np.bincount(chunk.indices[chunk.data != 0], minlength=info["n_vars"])

keep = n_cells > 0
dict(zip(info["var_names"], n_cells.tolist()))
{'gene0': 3, 'gene1': 2, 'gene2': 1, 'gene3': 1, 'gene4': 2}

The same counts from the matrix in memory:

np.array_equal(n_cells, np.asarray((adata.X > 0).sum(axis=0)).ravel())
True

keep can then select genes wherever they are used, e.g. adata[:, keep] once the data does fit, or the var mask of a later write.

Reading only some cells

where selects cells by their obs values and rows by position. On h5ad, Zarr, h5seurat (dgCMatrix) and 10x HDF5 files only the selected rows are read from disk, so a filter costs what the matching cells cost; BPCells and MatrixMarket input is still read through. A chunk never spans a gap in the selection, and row_offset still refers to the full matrix.

for chunk in pk.open_stream("tiny.h5ad", where={"cell_type": "T"}):
    print(chunk.row_offset, chunk.nrows)

np.flatnonzero(adata.obs["cell_type"] == "T").tolist()
1 1
5 1
[1, 5]
for chunk in pk.open_stream("tiny.h5ad", rows=[0, 1, 4]):
    print(chunk.row_offset, chunk.nrows)
0 2
4 1

Streaming by gene

Some work goes gene by gene over every cell: one model per gene, or per-gene statistics too wide to accumulate. Rows of a cell-major matrix hold genes scattered across every chunk, so for this the matrix is first written stored by gene (CSC). convert(..., gene_major=True) does that, in a bounded memory budget (transpose_mem, in GB); anndata reads the result as a csc_matrix.

pk.convert("tiny.h5ad", "tiny_by_gene.h5ad", gene_major=True)
pk.inspect("tiny_by_gene.h5ad")["x_axis"]
'var'

open_stream(..., axis="var") then hands out blocks of whole genes. Each chunk’s to_scipy() is a csc_array of every cell by the block’s genes, and var_offset is its first gene.

for chunk in pk.open_stream("tiny_by_gene.h5ad", axis="var", chunk_size=2):
    Y = chunk.to_scipy()
    means = np.asarray(Y.mean(axis=0), dtype=float).ravel()
    print(chunk.var_offset, chunk.ncols, Y.shape, means.round(2).tolist())
0 2 (6, 2) [1.17, 1.0]
2 2 (6, 2) [0.5, 0.17]
4 1 (6, 1) [0.92]

genes selects genes the way rows selects cells, as (start, stop) ranges or indices, so independent jobs can each take a slice of the genes:

[(c.var_offset, c.ncols) for c in pk.open_stream("tiny_by_gene.h5ad", axis="var", genes=[(3, 5)])]
[(3, 2)]

See open_stream for every option.