Skip to contents

Opens path once and reads its genes a block at a time, as read_genes() reads them. For an .rds, opening inflates the file once; each block is then read in parallel from the index that pass built, so a block costs a fraction of a read_genes() call. Memory holds one block, plus 4 bytes per nonzero of the whole matrix while it is read.

Usage

gene_blocks(path, block_size = 2000L, genes = NULL, chunk_size = 1000L)

Arguments

path

Path to the dataset.

block_size

Genes per block.

genes

Genes to read: 1-based integer positions or gene names, in any order. NULL reads all genes.

chunk_size

Genes per internal read.

Value

A picklerick_gene_blocks list:

n

Number of blocks.

genes

List of each block's gene positions.

read(k)

Block k as a cells x genes dgCMatrix.

next_block()

The next block, or NULL after the last.

Details

Forked workers (e.g. parallel::mclapply) can share one iterator and read different blocks with $read(k).

See also

Examples

if (FALSE) { # \dontrun{
it <- gene_blocks("spe.rds", block_size = 2000)
while (!is.null(m <- it$next_block())) fit(m)
fits <- parallel::mclapply(seq_len(it$n), function(k) fit(it$read(k)))
} # }