Skip to contents

Overview

picklerick reads single-cell files through the SCX engine, a Rust library that streams a count matrix in chunks of cells and never holds a Python or HDF5 object graph in R. One reader function, read_h5ad(), handles every format: the format is detected from the file’s content, not its extension.

Format Produced by Read Written
AnnData .h5ad anndata, scanpy yes yes
AnnData Zarr store (.zarr directory, v2 or v3) adata.write_zarr() yes yes (v3)
MuData .h5mu (one modality, picked with assay =) mudata, muon yes
H5Seurat (.h5seurat, dgCMatrix or BPCells-backed) SeuratDisk, Seurat v5 yes yes
10x Genomics HDF5 (.h5) Cell Ranger yes
10x MatrixMarket directory (matrix.mtx.gz + features + barcodes) Cell Ranger yes
BPCells matrix directory BPCells yes X only

Each file gives you the same choice of R object: a SingleCellExperiment, a Seurat object, a light AnnData-like handle, or the raw pieces as a list. Large matrices can stay on disk as a DelayedMatrix.

This vignette uses a 6-cell by 5-gene dataset shipped with the package in every input format. The second cell has no counts and two genes are each expressed in a single cell, so empty cells and sparse genes are part of every example.

library(picklerick)

extdata <- system.file("extdata", package = "picklerick")
h5ad     <- file.path(extdata, "tiny.h5ad")
tenx_h5  <- file.path(extdata, "tiny_10x.h5")
mtx_dir  <- file.path(extdata, "tiny_mtx")
h5seurat <- file.path(extdata, "tiny_v4.h5seurat")

Looking before reading

inspect() reads only metadata: shape, column names, embedding keys, and per-layer sparsity from the row pointer. Nothing from the count matrix is loaded, whatever the file size.

info <- inspect(h5ad)
str(info[c("format", "n_obs", "n_vars", "obs_cols", "var_cols", "obsm_keys")])
#> List of 6
#>  $ format   : chr "H5AD"
#>  $ n_obs    : int 6
#>  $ n_vars   : int 5
#>  $ obs_cols : chr [1:7] "cell_type" "batch" "flag" "score" ...
#>  $ var_cols : chr "gene_symbol"
#>  $ obsm_keys: chr "X_pca"
info$layers
#> $name
#> [1] "counts"
#> 
#> $nnz
#> [1] 9
#> 
#> $nnz_q1
#> [1] 1
#> 
#> $nnz_med
#> [1] 2
#> 
#> $nnz_q3
#> [1] 2
#> 
#> $nnz_max
#> [1] 2

Reading into a SingleCellExperiment

The default target is a SingleCellExperiment. Features are rows and cells are columns, the Bioconductor convention. obs becomes colData(), var becomes rowData(), obsm entries become reducedDims(), layers become assays of their own name, and obsp graphs become colPairs(). The AnnData X becomes the counts assay, unless a layer is already called counts, as here (scanpy keeps raw counts there next to a normalised X); then X keeps its name.

sce <- read_h5ad(h5ad)
sce
#> class: SingleCellExperiment 
#> dim: 5 6 
#> metadata(2): .uns_path varm
#> assays(2): X counts
#> rownames(5): gene0 gene1 gene2 gene3 gene4
#> rowData names(1): gene_symbol
#> colnames(6): cell0 cell1 ... cell4 cell5
#> colData names(7): cell_type batch ... donor n_counts
#> reducedDimNames(1): X_pca
#> mainExpName: NULL
#> altExpNames(0):
SummarizedExperiment::assay(sce, "X")
#> 5 x 6 sparse Matrix of class "dgCMatrix"
#>       cell0 cell1 cell2 cell3 cell4 cell5
#> gene0   0.5     .   .     2.5   .     4.0
#> gene1   .       .   1.5   .     .     4.5
#> gene2   .       .   .     3.0   .     .  
#> gene3   1.0     .   .     .     .     .  
#> gene4   .       .   2.0   .     3.5   .
SummarizedExperiment::colData(sce)
#> DataFrame with 6 rows and 7 columns
#>       cell_type     batch      flag     score is_doublet    donor  n_counts
#>        <factor> <integer> <logical> <numeric>  <logical> <factor> <integer>
#> cell0        B          1      TRUE      0.50      FALSE       d0         3
#> cell1        T          0     FALSE      1.50      FALSE       d1         0
#> cell2        B          2     FALSE       NaN       TRUE       d0         7
#> cell3        B          2      TRUE     -2.00      FALSE       d1        11
#> cell4        NK         1     FALSE      0.00      FALSE       d0         7
#> cell5        T          0     FALSE      3.25      FALSE       d1        17

Column encodings survive the trip: categoricals become factors, pandas nullable integer and boolean columns keep their NAs.

str(as.data.frame(SummarizedExperiment::colData(sce)))
#> 'data.frame':    6 obs. of  7 variables:
#>  $ cell_type : Factor w/ 3 levels "B","NK","T": 1 3 1 1 2 3
#>  $ batch     : int  1 0 2 2 1 0
#>  $ flag      : logi  TRUE FALSE FALSE TRUE FALSE FALSE
#>  $ score     : num  0.5 1.5 NaN -2 0 3.25
#>  $ is_doublet: logi  FALSE FALSE TRUE FALSE FALSE FALSE
#>  $ donor     : Factor w/ 2 levels "d0","d1": 1 2 1 2 1 2
#>  $ n_counts  : int  3 0 7 11 7 17

Unstructured metadata (uns)

uns can be large and deeply nested, so it is not parsed by default. The uns() accessor reads one key at a time from the file when you need it, or parse_uns = TRUE materialises all of it into metadata().

uns(sce)                 # the keys, without reading the values
#> [1] "colors"  "n_pcs"   "params"  "title"   "weights"
uns(sce, "params")
#> $resolution
#> [1] 0.5
#> 
#> $use_raw
#> [1] FALSE
#> Levels: FALSE TRUE

The same reader for every format

Content detection means the file name does not matter. The four files below hold the same cells and genes.

for (f in c(h5ad, tenx_h5, mtx_dir, h5seurat)) {
  x <- read_h5ad(f)
  cat(sprintf("%-20s %s  %d x %d\n", basename(f), inspect(f)$format, nrow(x), ncol(x)))
}
#> tiny.h5ad            H5AD  5 x 6
#> tiny_10x.h5          10x HDF5  5 x 6
#> tiny_mtx             MatrixMarket  5 x 6
#> tiny_v4.h5seurat     H5Seurat  5 x 6

A 10x file carries only the matrix and its names, so its colData() is empty. An H5Seurat file is read from its RNA assay: the counts layer becomes the counts assay and its other layers come along as assays too.

colnames(SummarizedExperiment::colData(read_h5ad(tenx_h5)))
#> character(0)
read_h5ad(h5seurat)
#> class: SingleCellExperiment 
#> dim: 5 6 
#> metadata(1): .uns_path
#> assays(2): counts data
#> rownames(5): gene0 gene1 gene2 gene3 gene4
#> rowData names(0):
#> colnames(6): cell0 cell1 ... cell4 cell5
#> colData names(7): cell_type batch ... donor n_counts
#> reducedDimNames(1): X_pca
#> mainExpName: NULL
#> altExpNames(0):

AnnData Zarr stores

A Zarr store is a directory. Reading one is the same call; read_zarr() is an alias that makes the intent visible. Here we make one with convert(), which streams one file into another format without building any R object.

zarr <- file.path(tempdir(), "tiny.zarr")
convert(h5ad, zarr)
inspect(zarr)$format
#> [1] "Zarr (AnnData)"
identical(
  SummarizedExperiment::assay(read_zarr(zarr), "X"),
  SummarizedExperiment::assay(sce, "X")
)
#> [1] TRUE

Reading into a Seurat object

as = "Seurat" builds a Seurat v5 object: counts from X, cell metadata from obs, feature metadata on the assay, reductions from obsm. obsp graphs are not attached and are dropped with a warning.

obj <- read_h5ad(h5ad, as = "Seurat")
#> Warning: No columnames present in cell embeddings, setting to 'pca_1:2'
#> Warning in .as_seurat(raw, path = path, parse_uns = parse_uns, constructor =
#> constructor): read_h5ad: obsp 'connectivities' is not yet attached to Seurat
#> output; dropped.
obj
#> An object of class Seurat 
#> 10 features across 6 samples within 2 assays 
#> Active assay: RNA (5 features, 0 variable features)
#>  1 layer present: counts
#>  1 other assay present: counts
#>  1 dimensional reduction calculated: pca
head(obj[[]])
#>          orig.ident nCount_RNA nFeature_RNA cell_type batch  flag score
#> cell0 SeuratProject        1.5            2         B     1  TRUE  0.50
#> cell1 SeuratProject        0.0            0         T     0 FALSE  1.50
#> cell2 SeuratProject        3.5            2         B     2 FALSE   NaN
#> cell3 SeuratProject        5.5            2         B     2  TRUE -2.00
#> cell4 SeuratProject        3.5            1        NK     1 FALSE  0.00
#> cell5 SeuratProject        8.5            2         T     0 FALSE  3.25
#>       is_doublet donor n_counts nCount_counts nFeature_counts
#> cell0      FALSE    d0        3             3               2
#> cell1      FALSE    d1        0             0               0
#> cell2       TRUE    d0        7             7               2
#> cell3      FALSE    d1       11            11               2
#> cell4      FALSE    d0        7             7               1
#> cell5      FALSE    d1       17            17               2

constructor = "fast" assembles the Assay5 directly from the matrix just read rather than through CreateSeuratObject(), which copies it twice. It is used only when the names and metadata need none of the sanitising CreateSeuratObject() does, and falls back otherwise. On 10,000 cells it removes the extra 500 MB and most of the construction time; on six cells it makes no visible difference.

obj_fast <- read_h5ad(h5ad, as = "Seurat", constructor = "fast")
#> Warning: No columnames present in cell embeddings, setting to 'pca_1:2'
#> Warning in .as_seurat(raw, path = path, parse_uns = parse_uns, constructor =
#> constructor): read_h5ad: obsp 'connectivities' is not yet attached to Seurat
#> output; dropped.
identical(SeuratObject::LayerData(obj_fast, "counts"), SeuratObject::LayerData(obj, "counts"))
#> [1] TRUE

Deferring the R object

Building a SingleCellExperiment costs a few seconds of S4 warm-up however small the data. as = "AnnData" returns a light handle instead: a shape, the names, and components that are built on first use and cached. Convert it later with as_sce() or as_seurat().

a <- read_h5ad(h5ad, as = "AnnData")
a
#> <AnnData> 5 genes x 6 cells
#>   obs: 7 columns   var: 1 columns
#>   obsm:   X_pca 
#>   layers: counts 
#>   built: nothing yet
dim(a)
#> [1] 5 6
a$obs_names
#> [1] "cell0" "cell1" "cell2" "cell3" "cell4" "cell5"
a$X                                   # built now, cached
#> 5 x 6 sparse Matrix of class "dgCMatrix"
#>       cell0 cell1 cell2 cell3 cell4 cell5
#> gene0   0.5     .   .     2.5   .     4.0
#> gene1   .       .   1.5   .     .     4.5
#> gene2   .       .   .     3.0   .     .  
#> gene3   1.0     .   .     .     .     .  
#> gene4   .       .   2.0   .     3.5   .
sce_from_handle <- as_sce(a)

as = "list" returns the raw pieces (index vectors, column lists, the CSR triplet) for callers assembling something else.

names(read_h5ad(h5ad, as = "list"))
#>  [1] "format"    "n_obs"     "n_vars"    "obs_index" "var_index" "obs_cols" 
#>  [7] "var_cols"  "x_indptr"  "x_indices" "x_data"    "obsm"      "varm"     
#> [13] "layers"    "obsp"      "uns_json"

Leaving the matrix on disk

lazy = TRUE reads the metadata and wraps the matrix in a DelayedMatrix instead of loading it. For an .h5ad the wrapper is HDF5Array::H5SparseMatrix when that package is installed, the conventional seed for that layout. For every other format, and for .h5ad without HDF5Array, it is PicklerickMatrix(), a seed that reads blocks through SCX.

lazy <- read_zarr(zarr, lazy = TRUE)
x <- SummarizedExperiment::assay(lazy, "X")
class(x)
#> [1] "DelayedMatrix"
#> attr(,"package")
#> [1] "DelayedArray"
DelayedArray::seed(x)
#> <PicklerickMatrixSeed> 5 x 6 (features x cells), chunk_size 5000
#>   blocks: sized by DelayedArray::getAutoBlockSize() 
#>   path:  /tmp/RtmpSsKLxy/tiny.zarr
#>   matrix: X

Selecting cells reads only those cells from the store: the row pointer is loaded, then just the selected rows’ slices of the values and column indices. Block operations read one block of cells at a time, every gene of those cells.

x[, c("cell2", "cell5")]
#> <5 x 2> sparse DelayedMatrix object of type "double":
#>       cell2 cell5
#> gene0   0.0   4.0
#> gene1   1.5   4.5
#> gene2   0.0   0.0
#> gene3   0.0   0.0
#> gene4   2.0   0.0
as.matrix(x[, c("cell2", "cell5")])
#>       cell2 cell5
#> gene0   0.0   4.0
#> gene1   1.5   4.5
#> gene2   0.0   0.0
#> gene3   0.0   0.0
#> gene4   2.0   0.0
DelayedArray::colSums(x)
#> cell0 cell1 cell2 cell3 cell4 cell5 
#>   1.5   0.0   3.5   5.5   3.5   8.5

PicklerickMatrix() can also be built directly, and can expose a layer.

m <- PicklerickMatrix(zarr, layer = "counts")
m
#> <5 x 6> sparse DelayedMatrix object of type "double":
#>       cell0 cell1 cell2 cell3 cell4 cell5
#> gene0     1     0     0     5     0     8
#> gene1     0     0     3     0     0     9
#> gene2     0     0     0     6     0     0
#> gene3     2     0     0     0     0     0
#> gene4     0     0     4     0     7     0

Filtering without loading the matrix

Expressions on a DelayedMatrix are recorded, not computed, and reductions run block by block. A gene filter therefore never holds the whole matrix, and the filtered object stays on disk:

keep <- DelayedArray::rowSums(x > 0) > 0   # genes seen in at least one cell
filtered <- lazy[keep, ]
dim(filtered)
#> [1] 5 6
class(SummarizedExperiment::assay(filtered, "X"))
#> [1] "DelayedMatrix"
#> attr(,"package")
#> [1] "DelayedArray"

Most Bioconductor tools (scuttle’s normalisation, scran, …) work on a DelayedMatrix in the same block-wise way, until something asks for the values in memory.

Writing a subset of cells to a new file

To keep some cells in a new file, filter while converting. convert(where = ) takes a named list of obs columns and the values each may take; a cell is kept when every column holds one of its values (values compare as text, NA never matches). No R object is built, and the matrix streams through in chunks of chunk_size cells, so memory stays bounded however many cells match:

tcells <- file.path(tempdir(), "tcells.h5ad")
h5ad |> convert(tcells, where = list(cell_type = "T"))
colnames(read_h5ad(tcells))
#> [1] "cell1" "cell5"

On a real atlas that reads as "atlas.h5ad" |> convert("tcells.h5ad", where = list(cell_type = "T cell")). The output keeps the matching cells in every cell-axis slot (the matrix, layers, colData(), reduced dimensions, and colPairs() graphs restricted to those cells); genes and uns are copied unchanged. .h5ad, Zarr, dgCMatrix H5Seurat and 10x HDF5 inputs read only the matching cells. BPCells and MatrixMarket inputs are read through, dropping the rest.

The lazy route subsets the object in R and writes that:

lazy_h5ad <- read_h5ad(h5ad, lazy = TRUE)
t_lazy <- lazy_h5ad[, lazy_h5ad$cell_type %in% "T"]
write_h5ad(t_lazy, file.path(tempdir(), "tcells_lazy.h5ad"))

Subsetting is lazy, but write_h5ad() holds the subset’s matrix in memory to write it, so its peak grows with the number of matching cells. Use it when the subset is small or you change the object in R first; use convert(where = ) to cut a large subset out of a file as it is.

Controlling peak memory

Peak memory is set by the block size, not by the size of the data. For an .h5ad or Zarr store, DelayedArray::setAutoBlockSize() sets it, for both PicklerickMatrix() and HDF5Array’s seed (default 100 MB):

old <- DelayedArray::getAutoBlockSize()
# Room for two cells per block (8 bytes per value, every gene): on this tiny
# matrix that makes several blocks of whole cells.
DelayedArray::setAutoBlockSize(8 * nrow(x) * 2)
#> automatic block size set to 80 bytes (was 1e+08)
DelayedArray::defaultAutoGrid(x)
#> 1 x 3  RegularArrayGrid object on a 5 x 6 array:
#>      [,1]    [,2]    [,3]   
#> [1,] [ ,1-2] [ ,3-4] [ ,5-6]
DelayedArray::setAutoBlockSize(old)
#> automatic block size set to 1e+08 bytes (was 80)

On a 100,000-cell subset of the Human Lung Cell Atlas (28,024 genes, 194 million nonzeros), the filter above took, with R’s own ~0.6 GB included:

peak memory time
eager read_h5ad() 5.4 GB 14.5 s
PicklerickMatrix(), 10 MB blocks 0.68 GB 18.8 s
PicklerickMatrix(), 100 MB blocks (default) 0.88 GB 16.4 s
HDF5Array seed, 100 MB blocks 0.97 GB 11.2 s

Smaller blocks lower the peak a little and add per-block overhead; past ~100 MB, larger blocks cost memory without being faster. Formats other than .h5ad and Zarr can’t yet read a range of cells on its own, so their blocks are chunk_size cells wide and each block reads through the matrix.

Writing

write_h5ad() writes a SingleCellExperiment as .h5ad, and convert() streams any readable file into .h5ad, .h5seurat or a Zarr v3 store, chosen by the output extension. Both bound memory by the chunk size, not the dataset.

tenx <- read_h5ad(tenx_h5)
out <- file.path(tempdir(), "roundtrip.h5ad")
write_h5ad(tenx, out)
identical(
  SummarizedExperiment::assay(read_h5ad(out), "counts"),
  SummarizedExperiment::assay(tenx, "counts")
)
#> [1] TRUE
convert(h5ad, file.path(tempdir(), "tiny.h5seurat"))
inspect(file.path(tempdir(), "tiny.h5seurat"))$format
#> [1] "H5Seurat"

Session information

sessionInfo()
#> R version 4.5.3 (2026-03-11)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Europe/Zurich
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] picklerick_0.6.2
#> 
#> loaded via a namespace (and not attached):
#>   [1] RColorBrewer_1.1-3          jsonlite_2.0.0             
#>   [3] magrittr_2.0.5              spatstat.utils_3.2-5       
#>   [5] farver_2.1.2                rmarkdown_2.32             
#>   [7] fs_2.1.0                    ragg_1.5.2                 
#>   [9] vctrs_0.7.3                 ROCR_1.0-12                
#>  [11] spatstat.explore_3.8-3      htmltools_0.5.9            
#>  [13] S4Arrays_1.10.1             Rhdf5lib_2.0.0             
#>  [15] rhdf5_2.54.1                SparseArray_1.10.10        
#>  [17] sass_0.4.10                 sctransform_0.4.3          
#>  [19] parallelly_1.48.0           KernSmooth_2.23-26         
#>  [21] bslib_0.12.0                htmlwidgets_1.6.4          
#>  [23] desc_1.4.3                  ica_1.0-3                  
#>  [25] plyr_1.8.9                  plotly_4.12.1              
#>  [27] zoo_1.9-1                   cachem_1.1.0               
#>  [29] igraph_2.3.4                mime_0.13                  
#>  [31] lifecycle_1.0.5             pkgconfig_2.0.3            
#>  [33] Matrix_1.7-4                R6_2.6.1                   
#>  [35] fastmap_1.2.0               MatrixGenerics_1.22.0      
#>  [37] fitdistrplus_1.2-6          future_1.80.0              
#>  [39] shiny_1.14.0                digest_0.6.39              
#>  [41] patchwork_1.3.2             S4Vectors_0.48.1           
#>  [43] Seurat_5.6.0                tensor_1.5.1               
#>  [45] RSpectra_0.16-2             irlba_2.4.1                
#>  [47] GenomicRanges_1.62.1        textshaping_1.0.5          
#>  [49] progressr_1.1.0             spatstat.sparse_3.2-0      
#>  [51] httr_1.4.9                  polyclip_1.10-7            
#>  [53] abind_1.4-8                 compiler_4.5.3             
#>  [55] S7_0.2.2                    fastDummies_1.7.6          
#>  [57] HDF5Array_1.38.0            MASS_7.3-65                
#>  [59] DelayedArray_0.36.1         tools_4.5.3                
#>  [61] lmtest_0.9-40               otel_0.2.0                 
#>  [63] httpuv_1.6.17               future.apply_1.21.0        
#>  [65] goftest_1.2-3               glue_1.8.1                 
#>  [67] h5mread_1.2.1               rhdf5filters_1.22.0        
#>  [69] nlme_3.1-168                promises_1.5.0             
#>  [71] grid_4.5.3                  Rtsne_0.17                 
#>  [73] cluster_2.1.8.2             reshape2_1.4.5             
#>  [75] generics_0.1.4              gtable_0.3.6               
#>  [77] spatstat.data_3.1-9         tidyr_1.3.2                
#>  [79] data.table_1.18.6.1         sp_2.2-4                   
#>  [81] XVector_0.50.0              BiocGenerics_0.56.0        
#>  [83] spatstat.geom_3.8-3         RcppAnnoy_0.0.23           
#>  [85] ggrepel_0.9.8               RANN_2.6.3                 
#>  [87] pillar_1.11.1               stringr_1.6.0              
#>  [89] spam_2.11-4                 RcppHNSW_0.7.0             
#>  [91] later_1.4.8                 splines_4.5.3              
#>  [93] dplyr_1.2.1                 lattice_0.22-9             
#>  [95] survival_3.8-6              deldir_2.0-4               
#>  [97] tidyselect_1.2.1            SingleCellExperiment_1.32.0
#>  [99] miniUI_0.1.2                pbapply_1.7-5              
#> [101] knitr_1.52                  gridExtra_2.3.1            
#> [103] Seqinfo_1.0.0               IRanges_2.44.0             
#> [105] SummarizedExperiment_1.40.0 scattermore_1.2            
#> [107] stats4_4.5.3                xfun_0.62                  
#> [109] Biobase_2.70.0              matrixStats_1.5.0          
#> [111] stringi_1.8.9               yaml_2.3.12                
#> [113] evaluate_1.0.5              codetools_0.2-20           
#> [115] tibble_3.3.1                cli_3.6.6                  
#> [117] uwot_0.2.5                  xtable_1.8-8               
#> [119] reticulate_1.47.0           systemfonts_1.3.2          
#> [121] jquerylib_0.1.4             Rcpp_1.1.2                 
#> [123] globals_0.20.0              spatstat.random_3.5-2      
#> [125] png_0.1-9                   spatstat.univar_3.2-0      
#> [127] parallel_4.5.3              pkgdown_2.2.1              
#> [129] ggplot2_4.0.3               dotCall64_1.2              
#> [131] listenv_1.1.0               viridisLite_0.4.3          
#> [133] scales_1.4.0                ggridges_0.5.7             
#> [135] SeuratObject_5.4.0          purrr_1.2.2                
#> [137] rlang_1.3.0                 cowplot_1.2.0