dexp | light-sheet Dataset EXploration and Processing

dexp is a napari, CuPy, Zarr, and DASK based library for managing, processing and visualizing light-sheet microscopy datasets. It consists in light-sheet specialised image processing functions (equalisation, denoising, dehazing, registration, fusion, stabilization, deskewing, deconvolution), visualization functions (napari-based viewing, 2D/3D rendering, video compositing and resizing, mp4 generation), as well as dataset management functions (copy, crop, concatenation, tiff conversion). Almost all functions are GPU accelerated via CuPy but also have a numpy-based fallback option for testing on small datasets. In addition to a functional API, DEXP offers a command line interface that makes it very easy for non-coders to pipeline large processing jobs all the way from a large multi-terabyte raw dataset to fully processed and rendered video in MP4 format.

How to install dexp

Prerequisites:

dexp works on OSX and Windows, but it is recomended to use the latest version of Ubuntu. We recommend a machine with a top-of-the-line NVIDIA graphics card (min 12G to be confortable).

First, make sure to have a working python installation. Second, make sure to have a compatible and functional CUDA installation

Once these prerequisites are satified, you can install dexp.

Installation:

dexp can simply be installed with:

To installs dexp with GPU support (CUDA 11.2), colored console output, and napari support do:

pip install dexp[color,cuda112,napari]

Other available CUDA versions (from CuPy) are: cuda111, cuda110, cuda102, cuda101, cuda100. We recommend using the most recent CUDA version that your system supports, and avoiding versions below 10.0

If instead you do not wish to add CUDA support, you can instead do:

pip install dexp

For OSX users: Before installating dexp, you will first need to install cairo:

brew install cairo

Quick environment setup and installation:

The following commands delete any existing dexp environment, recreate it, and install dexp with support for CUDA 11.2:

conda deactivate
conda env remove --name dexp
conda create -y --name dexp python=3.9
conda activate dexp
pip install dexp[color,cuda112]
pip install napari[pyqt5]

If you are having problems with the cuda/cuda-toolkit the best is to use conda to install the correct version of the cudatoolkit:

conda install -c conda-forge cudatoolkit==11.2.2

You can check here for the best matching version.

Notes:

• You might get some error messages recommending you install missing libraries such as CUDNN, CuTensor, nccl, etc... These messages often come with instructions on what to do.
• Adjust your driver version (here 11.2) to your card(s) and drivers.
• Windows users should call conda install -c conda-forge pyopencl before running the second to last step.

Leveraging extra CUDA libraries for faster processing:

If you want you dexp CUDA-based processing to be even faster, you can install additional libraries such as CUDNN and CUTENSOR with the following command:

install cudalibs 11.2

Change the CUDA version accordingly... This is not needed when installing cupy from conda-forge

Alternative instalation that pulls cupy from conda-forge:

In some cases cupy installed with pip might not recognise your CUDA drivers. If that's the case, you can instead insall dexp with the following commands:

conda deactivate
conda env remove --name dexp
conda create -y --name dexp python=3.9
conda activate dexp
conda install -y -c conda-forge cupy cudatoolkit=11.2
conda install -y -c conda-forge cudnn cutensor nccl
pip install dexp[color]
pip install napari[pyqt5]

The commands above will install dexp on a fresh environment, with cupy, napari, cudnn cutensor and nccl.

dexp Zarr dataset structure

The zarr datasets injested and written by dexp are organized as below:

/ (root)
 └── channel1 (group)
     ├── channel1 (array)
     ├── channel1_projection_0 (optional)
     ├── channel1_projection_1 (optional)
     └── channel1_projection_2 (optional)
  └── channel2 (group)
     ├── channel2 (array)
     ├── channel2_projection_0 (optional)
     ├── channel2_projection_1 (optional)
     └── channel2_projection_2 (optional)
  └── more channels ...

Channels (zarr group) could be of a particular emission color (e.g. DAPI, GFP, etc), or/and of a particular imaging views (e.g. view1 and view2 in a dual view acquisition). Under each channel group, there could be multiple zarr array. The array that has the same name as the group name is typically a n-dimentional stack (e.g. time-z-y-x). The projections of that nd array are optional (useful for quick exploration of the nd stack). When writting output datasets dexp automatically generates these projections. Future versions of dexp might add more such convenience arrays, high in the list is of course downscaled version sof the stacks for faster visualisation and browsing...

Note: Our goal is to eventually transition to a ome-zarr and/or ngff storage by defaut for both reading and writting. For reading we have also support for specific dataset produced by our light-sheet microscopes, see here for supported microscopes and formats. This is currently limited but contributions are very welcome!

DaXi

DEXP supports processing datasets acquired on the DaXi microscope (paper). You can test processing of DaXi data using an example dataset

Versions

The list of released versions can be found here. The version format is: YYYY.MM.DD.M where YYYY is the year, MM the month, dd the day, and M is the number of elapsed minutes of the day. Git tags are automatically set to link pipy versions to github tagged versions so that the corresponding code can be inspected on github, probably the most important feature. This is a very simple and semantically clear versionning scheme that accomodates for a rapid rate of updates.

How to use dexp ?

In depth documentation can be found here for both command line commands and for the python API.

Contributors:

Jordao Bragantini, Ahmet Can Solak, Bin Yang, and Loic A Royer