Reading single-cell files with picklerick
SCX Authors
2026-11-21
Source:vignettes/picklerick.Rmd
picklerick.RmdOverview
picklerick reads single-cell files through the SCX engine, a Rust
library that streams a count matrix in chunks of cells and never holds a
Python or HDF5 object graph in R. One reader function,
read_h5ad(), handles every format: the format is detected
from the file’s content, not its extension.
| Format | Produced by | Read | Written |
|---|---|---|---|
AnnData .h5ad
|
anndata, scanpy | yes | yes |
AnnData Zarr store (.zarr directory, v2 or v3) |
adata.write_zarr() |
yes | yes (v3) |
MuData .h5mu (one modality, picked with
assay =) |
mudata, muon | yes | |
H5Seurat (.h5seurat, dgCMatrix or BPCells-backed) |
SeuratDisk, Seurat v5 | yes | yes |
10x Genomics HDF5 (.h5) |
Cell Ranger | yes | |
10x MatrixMarket directory (matrix.mtx.gz + features +
barcodes) |
Cell Ranger | yes | |
| BPCells matrix directory | BPCells | yes | X only |
Each file gives you the same choice of R object: a
SingleCellExperiment, a Seurat object, a light
AnnData-like handle, or the raw pieces as a list. Large matrices can
stay on disk as a DelayedMatrix.
This vignette uses a 6-cell by 5-gene dataset shipped with the package in every input format. The second cell has no counts and two genes are each expressed in a single cell, so empty cells and sparse genes are part of every example.
library(picklerick)
extdata <- system.file("extdata", package = "picklerick")
h5ad <- file.path(extdata, "tiny.h5ad")
tenx_h5 <- file.path(extdata, "tiny_10x.h5")
mtx_dir <- file.path(extdata, "tiny_mtx")
h5seurat <- file.path(extdata, "tiny_v4.h5seurat")Looking before reading
inspect() reads only metadata: shape, column names,
embedding keys, and per-layer sparsity from the row pointer. Nothing
from the count matrix is loaded, whatever the file size.
info <- inspect(h5ad)
str(info[c("format", "n_obs", "n_vars", "obs_cols", "var_cols", "obsm_keys")])
#> List of 6
#> $ format : chr "H5AD"
#> $ n_obs : int 6
#> $ n_vars : int 5
#> $ obs_cols : chr [1:7] "cell_type" "batch" "flag" "score" ...
#> $ var_cols : chr "gene_symbol"
#> $ obsm_keys: chr "X_pca"
info$layers
#> $name
#> [1] "counts"
#>
#> $nnz
#> [1] 9
#>
#> $nnz_q1
#> [1] 1
#>
#> $nnz_med
#> [1] 2
#>
#> $nnz_q3
#> [1] 2
#>
#> $nnz_max
#> [1] 2Reading into a SingleCellExperiment
The default target is a SingleCellExperiment. Features
are rows and cells are columns, the Bioconductor convention.
obs becomes colData(), var
becomes rowData(), obsm entries become
reducedDims(), layers become assays of their own name, and
obsp graphs become colPairs(). The AnnData
X becomes the counts assay, unless a layer is
already called counts, as here (scanpy keeps raw counts
there next to a normalised X); then X keeps
its name.
sce <- read_h5ad(h5ad)
sce
#> class: SingleCellExperiment
#> dim: 5 6
#> metadata(2): .uns_path varm
#> assays(2): X counts
#> rownames(5): gene0 gene1 gene2 gene3 gene4
#> rowData names(1): gene_symbol
#> colnames(6): cell0 cell1 ... cell4 cell5
#> colData names(7): cell_type batch ... donor n_counts
#> reducedDimNames(1): X_pca
#> mainExpName: NULL
#> altExpNames(0):
SummarizedExperiment::assay(sce, "X")
#> 5 x 6 sparse Matrix of class "dgCMatrix"
#> cell0 cell1 cell2 cell3 cell4 cell5
#> gene0 0.5 . . 2.5 . 4.0
#> gene1 . . 1.5 . . 4.5
#> gene2 . . . 3.0 . .
#> gene3 1.0 . . . . .
#> gene4 . . 2.0 . 3.5 .
SummarizedExperiment::colData(sce)
#> DataFrame with 6 rows and 7 columns
#> cell_type batch flag score is_doublet donor n_counts
#> <factor> <integer> <logical> <numeric> <logical> <factor> <integer>
#> cell0 B 1 TRUE 0.50 FALSE d0 3
#> cell1 T 0 FALSE 1.50 FALSE d1 0
#> cell2 B 2 FALSE NaN TRUE d0 7
#> cell3 B 2 TRUE -2.00 FALSE d1 11
#> cell4 NK 1 FALSE 0.00 FALSE d0 7
#> cell5 T 0 FALSE 3.25 FALSE d1 17Column encodings survive the trip: categoricals become factors,
pandas nullable integer and boolean columns keep their
NAs.
str(as.data.frame(SummarizedExperiment::colData(sce)))
#> 'data.frame': 6 obs. of 7 variables:
#> $ cell_type : Factor w/ 3 levels "B","NK","T": 1 3 1 1 2 3
#> $ batch : int 1 0 2 2 1 0
#> $ flag : logi TRUE FALSE FALSE TRUE FALSE FALSE
#> $ score : num 0.5 1.5 NaN -2 0 3.25
#> $ is_doublet: logi FALSE FALSE TRUE FALSE FALSE FALSE
#> $ donor : Factor w/ 2 levels "d0","d1": 1 2 1 2 1 2
#> $ n_counts : int 3 0 7 11 7 17Unstructured metadata (uns)
uns can be large and deeply nested, so it is not parsed
by default. The uns() accessor reads one key at a time from
the file when you need it, or parse_uns = TRUE materialises
all of it into metadata().
The same reader for every format
Content detection means the file name does not matter. The four files below hold the same cells and genes.
for (f in c(h5ad, tenx_h5, mtx_dir, h5seurat)) {
x <- read_h5ad(f)
cat(sprintf("%-20s %s %d x %d\n", basename(f), inspect(f)$format, nrow(x), ncol(x)))
}
#> tiny.h5ad H5AD 5 x 6
#> tiny_10x.h5 10x HDF5 5 x 6
#> tiny_mtx MatrixMarket 5 x 6
#> tiny_v4.h5seurat H5Seurat 5 x 6A 10x file carries only the matrix and its names, so its
colData() is empty. An H5Seurat file is read from its
RNA assay: the counts layer becomes the
counts assay and its other layers come along as assays
too.
colnames(SummarizedExperiment::colData(read_h5ad(tenx_h5)))
#> character(0)
read_h5ad(h5seurat)
#> class: SingleCellExperiment
#> dim: 5 6
#> metadata(1): .uns_path
#> assays(2): counts data
#> rownames(5): gene0 gene1 gene2 gene3 gene4
#> rowData names(0):
#> colnames(6): cell0 cell1 ... cell4 cell5
#> colData names(7): cell_type batch ... donor n_counts
#> reducedDimNames(1): X_pca
#> mainExpName: NULL
#> altExpNames(0):AnnData Zarr stores
A Zarr store is a directory. Reading one is the same call;
read_zarr() is an alias that makes the intent visible. Here
we make one with convert(), which streams one file into
another format without building any R object.
Reading into a Seurat object
as = "Seurat" builds a Seurat v5 object:
counts from X, cell metadata from
obs, feature metadata on the assay, reductions from
obsm. obsp graphs are not attached and are
dropped with a warning.
obj <- read_h5ad(h5ad, as = "Seurat")
#> Warning: No columnames present in cell embeddings, setting to 'pca_1:2'
#> Warning in .as_seurat(raw, path = path, parse_uns = parse_uns, constructor =
#> constructor): read_h5ad: obsp 'connectivities' is not yet attached to Seurat
#> output; dropped.
obj
#> An object of class Seurat
#> 10 features across 6 samples within 2 assays
#> Active assay: RNA (5 features, 0 variable features)
#> 1 layer present: counts
#> 1 other assay present: counts
#> 1 dimensional reduction calculated: pca
head(obj[[]])
#> orig.ident nCount_RNA nFeature_RNA cell_type batch flag score
#> cell0 SeuratProject 1.5 2 B 1 TRUE 0.50
#> cell1 SeuratProject 0.0 0 T 0 FALSE 1.50
#> cell2 SeuratProject 3.5 2 B 2 FALSE NaN
#> cell3 SeuratProject 5.5 2 B 2 TRUE -2.00
#> cell4 SeuratProject 3.5 1 NK 1 FALSE 0.00
#> cell5 SeuratProject 8.5 2 T 0 FALSE 3.25
#> is_doublet donor n_counts nCount_counts nFeature_counts
#> cell0 FALSE d0 3 3 2
#> cell1 FALSE d1 0 0 0
#> cell2 TRUE d0 7 7 2
#> cell3 FALSE d1 11 11 2
#> cell4 FALSE d0 7 7 1
#> cell5 FALSE d1 17 17 2constructor = "fast" assembles the Assay5
directly from the matrix just read rather than through
CreateSeuratObject(), which copies it twice. It is used
only when the names and metadata need none of the sanitising
CreateSeuratObject() does, and falls back otherwise. On
10,000 cells it removes the extra 500 MB and most of the construction
time; on six cells it makes no visible difference.
obj_fast <- read_h5ad(h5ad, as = "Seurat", constructor = "fast")
#> Warning: No columnames present in cell embeddings, setting to 'pca_1:2'
#> Warning in .as_seurat(raw, path = path, parse_uns = parse_uns, constructor =
#> constructor): read_h5ad: obsp 'connectivities' is not yet attached to Seurat
#> output; dropped.
identical(SeuratObject::LayerData(obj_fast, "counts"), SeuratObject::LayerData(obj, "counts"))
#> [1] TRUEDeferring the R object
Building a SingleCellExperiment costs a few seconds of
S4 warm-up however small the data. as = "AnnData" returns a
light handle instead: a shape, the names, and components that are built
on first use and cached. Convert it later with as_sce() or
as_seurat().
a <- read_h5ad(h5ad, as = "AnnData")
a
#> <AnnData> 5 genes x 6 cells
#> obs: 7 columns var: 1 columns
#> obsm: X_pca
#> layers: counts
#> built: nothing yet
dim(a)
#> [1] 5 6
a$obs_names
#> [1] "cell0" "cell1" "cell2" "cell3" "cell4" "cell5"
a$X # built now, cached
#> 5 x 6 sparse Matrix of class "dgCMatrix"
#> cell0 cell1 cell2 cell3 cell4 cell5
#> gene0 0.5 . . 2.5 . 4.0
#> gene1 . . 1.5 . . 4.5
#> gene2 . . . 3.0 . .
#> gene3 1.0 . . . . .
#> gene4 . . 2.0 . 3.5 .
sce_from_handle <- as_sce(a)as = "list" returns the raw pieces (index vectors,
column lists, the CSR triplet) for callers assembling something
else.
Leaving the matrix on disk
lazy = TRUE reads the metadata and wraps the matrix in a
DelayedMatrix instead of loading it. For an
.h5ad the wrapper is HDF5Array::H5SparseMatrix
when that package is installed, the conventional seed for that layout.
For every other format, and for .h5ad without HDF5Array, it
is PicklerickMatrix(), a seed that reads blocks through
SCX.
lazy <- read_zarr(zarr, lazy = TRUE)
x <- SummarizedExperiment::assay(lazy, "X")
class(x)
#> [1] "DelayedMatrix"
#> attr(,"package")
#> [1] "DelayedArray"
DelayedArray::seed(x)
#> <PicklerickMatrixSeed> 5 x 6 (features x cells), chunk_size 5000
#> blocks: sized by DelayedArray::getAutoBlockSize()
#> path: /tmp/RtmpSsKLxy/tiny.zarr
#> matrix: XSelecting cells reads only those cells from the store: the row pointer is loaded, then just the selected rows’ slices of the values and column indices. Block operations read one block of cells at a time, every gene of those cells.
x[, c("cell2", "cell5")]
#> <5 x 2> sparse DelayedMatrix object of type "double":
#> cell2 cell5
#> gene0 0.0 4.0
#> gene1 1.5 4.5
#> gene2 0.0 0.0
#> gene3 0.0 0.0
#> gene4 2.0 0.0
as.matrix(x[, c("cell2", "cell5")])
#> cell2 cell5
#> gene0 0.0 4.0
#> gene1 1.5 4.5
#> gene2 0.0 0.0
#> gene3 0.0 0.0
#> gene4 2.0 0.0
DelayedArray::colSums(x)
#> cell0 cell1 cell2 cell3 cell4 cell5
#> 1.5 0.0 3.5 5.5 3.5 8.5PicklerickMatrix() can also be built directly, and can
expose a layer.
m <- PicklerickMatrix(zarr, layer = "counts")
m
#> <5 x 6> sparse DelayedMatrix object of type "double":
#> cell0 cell1 cell2 cell3 cell4 cell5
#> gene0 1 0 0 5 0 8
#> gene1 0 0 3 0 0 9
#> gene2 0 0 0 6 0 0
#> gene3 2 0 0 0 0 0
#> gene4 0 0 4 0 7 0Filtering without loading the matrix
Expressions on a DelayedMatrix are recorded, not
computed, and reductions run block by block. A gene filter therefore
never holds the whole matrix, and the filtered object stays on disk:
keep <- DelayedArray::rowSums(x > 0) > 0 # genes seen in at least one cell
filtered <- lazy[keep, ]
dim(filtered)
#> [1] 5 6
class(SummarizedExperiment::assay(filtered, "X"))
#> [1] "DelayedMatrix"
#> attr(,"package")
#> [1] "DelayedArray"Most Bioconductor tools (scuttle’s normalisation, scran, …) work on a
DelayedMatrix in the same block-wise way, until something
asks for the values in memory.
Writing a subset of cells to a new file
To keep some cells in a new file, filter while converting.
convert(where = ) takes a named list of obs
columns and the values each may take; a cell is kept when every column
holds one of its values (values compare as text, NA never
matches). No R object is built, and the matrix streams through in chunks
of chunk_size cells, so memory stays bounded however many
cells match:
tcells <- file.path(tempdir(), "tcells.h5ad")
h5ad |> convert(tcells, where = list(cell_type = "T"))
colnames(read_h5ad(tcells))
#> [1] "cell1" "cell5"On a real atlas that reads as
"atlas.h5ad" |> convert("tcells.h5ad", where = list(cell_type = "T cell")).
The output keeps the matching cells in every cell-axis slot (the matrix,
layers, colData(), reduced dimensions, and
colPairs() graphs restricted to those cells); genes and
uns are copied unchanged. .h5ad, Zarr,
dgCMatrix H5Seurat and 10x HDF5 inputs read only the matching cells.
BPCells and MatrixMarket inputs are read through, dropping the rest.
The lazy route subsets the object in R and writes that:
lazy_h5ad <- read_h5ad(h5ad, lazy = TRUE)
t_lazy <- lazy_h5ad[, lazy_h5ad$cell_type %in% "T"]
write_h5ad(t_lazy, file.path(tempdir(), "tcells_lazy.h5ad"))Subsetting is lazy, but write_h5ad() holds the subset’s
matrix in memory to write it, so its peak grows with the number of
matching cells. Use it when the subset is small or you change the object
in R first; use convert(where = ) to cut a large subset out
of a file as it is.
Controlling peak memory
Peak memory is set by the block size, not by the size of the data.
For an .h5ad or Zarr store,
DelayedArray::setAutoBlockSize() sets it, for both
PicklerickMatrix() and HDF5Array’s seed (default 100
MB):
old <- DelayedArray::getAutoBlockSize()
# Room for two cells per block (8 bytes per value, every gene): on this tiny
# matrix that makes several blocks of whole cells.
DelayedArray::setAutoBlockSize(8 * nrow(x) * 2)
#> automatic block size set to 80 bytes (was 1e+08)
DelayedArray::defaultAutoGrid(x)
#> 1 x 3 RegularArrayGrid object on a 5 x 6 array:
#> [,1] [,2] [,3]
#> [1,] [ ,1-2] [ ,3-4] [ ,5-6]
DelayedArray::setAutoBlockSize(old)
#> automatic block size set to 1e+08 bytes (was 80)On a 100,000-cell subset of the Human Lung Cell Atlas (28,024 genes, 194 million nonzeros), the filter above took, with R’s own ~0.6 GB included:
| peak memory | time | |
|---|---|---|
eager read_h5ad()
|
5.4 GB | 14.5 s |
PicklerickMatrix(), 10 MB blocks |
0.68 GB | 18.8 s |
PicklerickMatrix(), 100 MB blocks (default) |
0.88 GB | 16.4 s |
| HDF5Array seed, 100 MB blocks | 0.97 GB | 11.2 s |
Smaller blocks lower the peak a little and add per-block overhead;
past ~100 MB, larger blocks cost memory without being faster. Formats
other than .h5ad and Zarr can’t yet read a range of cells
on its own, so their blocks are chunk_size cells wide and
each block reads through the matrix.
Writing
write_h5ad() writes a SingleCellExperiment
as .h5ad, and convert() streams any readable
file into .h5ad, .h5seurat or a Zarr v3 store,
chosen by the output extension. Both bound memory by the chunk size, not
the dataset.
tenx <- read_h5ad(tenx_h5)
out <- file.path(tempdir(), "roundtrip.h5ad")
write_h5ad(tenx, out)
identical(
SummarizedExperiment::assay(read_h5ad(out), "counts"),
SummarizedExperiment::assay(tenx, "counts")
)
#> [1] TRUE
convert(h5ad, file.path(tempdir(), "tiny.h5seurat"))
inspect(file.path(tempdir(), "tiny.h5seurat"))$format
#> [1] "H5Seurat"Session information
sessionInfo()
#> R version 4.5.3 (2026-03-11)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Europe/Zurich
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] picklerick_0.6.2
#>
#> loaded via a namespace (and not attached):
#> [1] RColorBrewer_1.1-3 jsonlite_2.0.0
#> [3] magrittr_2.0.5 spatstat.utils_3.2-5
#> [5] farver_2.1.2 rmarkdown_2.32
#> [7] fs_2.1.0 ragg_1.5.2
#> [9] vctrs_0.7.3 ROCR_1.0-12
#> [11] spatstat.explore_3.8-3 htmltools_0.5.9
#> [13] S4Arrays_1.10.1 Rhdf5lib_2.0.0
#> [15] rhdf5_2.54.1 SparseArray_1.10.10
#> [17] sass_0.4.10 sctransform_0.4.3
#> [19] parallelly_1.48.0 KernSmooth_2.23-26
#> [21] bslib_0.12.0 htmlwidgets_1.6.4
#> [23] desc_1.4.3 ica_1.0-3
#> [25] plyr_1.8.9 plotly_4.12.1
#> [27] zoo_1.9-1 cachem_1.1.0
#> [29] igraph_2.3.4 mime_0.13
#> [31] lifecycle_1.0.5 pkgconfig_2.0.3
#> [33] Matrix_1.7-4 R6_2.6.1
#> [35] fastmap_1.2.0 MatrixGenerics_1.22.0
#> [37] fitdistrplus_1.2-6 future_1.80.0
#> [39] shiny_1.14.0 digest_0.6.39
#> [41] patchwork_1.3.2 S4Vectors_0.48.1
#> [43] Seurat_5.6.0 tensor_1.5.1
#> [45] RSpectra_0.16-2 irlba_2.4.1
#> [47] GenomicRanges_1.62.1 textshaping_1.0.5
#> [49] progressr_1.1.0 spatstat.sparse_3.2-0
#> [51] httr_1.4.9 polyclip_1.10-7
#> [53] abind_1.4-8 compiler_4.5.3
#> [55] S7_0.2.2 fastDummies_1.7.6
#> [57] HDF5Array_1.38.0 MASS_7.3-65
#> [59] DelayedArray_0.36.1 tools_4.5.3
#> [61] lmtest_0.9-40 otel_0.2.0
#> [63] httpuv_1.6.17 future.apply_1.21.0
#> [65] goftest_1.2-3 glue_1.8.1
#> [67] h5mread_1.2.1 rhdf5filters_1.22.0
#> [69] nlme_3.1-168 promises_1.5.0
#> [71] grid_4.5.3 Rtsne_0.17
#> [73] cluster_2.1.8.2 reshape2_1.4.5
#> [75] generics_0.1.4 gtable_0.3.6
#> [77] spatstat.data_3.1-9 tidyr_1.3.2
#> [79] data.table_1.18.6.1 sp_2.2-4
#> [81] XVector_0.50.0 BiocGenerics_0.56.0
#> [83] spatstat.geom_3.8-3 RcppAnnoy_0.0.23
#> [85] ggrepel_0.9.8 RANN_2.6.3
#> [87] pillar_1.11.1 stringr_1.6.0
#> [89] spam_2.11-4 RcppHNSW_0.7.0
#> [91] later_1.4.8 splines_4.5.3
#> [93] dplyr_1.2.1 lattice_0.22-9
#> [95] survival_3.8-6 deldir_2.0-4
#> [97] tidyselect_1.2.1 SingleCellExperiment_1.32.0
#> [99] miniUI_0.1.2 pbapply_1.7-5
#> [101] knitr_1.52 gridExtra_2.3.1
#> [103] Seqinfo_1.0.0 IRanges_2.44.0
#> [105] SummarizedExperiment_1.40.0 scattermore_1.2
#> [107] stats4_4.5.3 xfun_0.62
#> [109] Biobase_2.70.0 matrixStats_1.5.0
#> [111] stringi_1.8.9 yaml_2.3.12
#> [113] evaluate_1.0.5 codetools_0.2-20
#> [115] tibble_3.3.1 cli_3.6.6
#> [117] uwot_0.2.5 xtable_1.8-8
#> [119] reticulate_1.47.0 systemfonts_1.3.2
#> [121] jquerylib_0.1.4 Rcpp_1.1.2
#> [123] globals_0.20.0 spatstat.random_3.5-2
#> [125] png_0.1-9 spatstat.univar_3.2-0
#> [127] parallel_4.5.3 pkgdown_2.2.1
#> [129] ggplot2_4.0.3 dotCall64_1.2
#> [131] listenv_1.1.0 viridisLite_0.4.3
#> [133] scales_1.4.0 ggridges_0.5.7
#> [135] SeuratObject_5.4.0 purrr_1.2.2
#> [137] rlang_1.3.0 cowplot_1.2.0