Jupyter Notebook Binder

Project flow#

LaminDB allows tracking data flow on the entire project level.

Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.

A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-γ production.

These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.

More specifically: Why should I care about data flow?

Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.

While tracking data flow is easier when it’s governed by deterministic pipelines, it becomes hard when it’s governed by interactive human-driven analyses.

LaminDB interfaces workflow mangers for the former and embraces the latter.

Setup#

Init a test instance:

!lamin init --storage ./mydata
Hide code cell output
✅ saved: User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2023-12-05 17:28:45 UTC)
✅ saved: Storage(uid='EgNDmwkd', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata', type='local', updated_at=2023-12-05 17:28:45 UTC, created_by_id=1)
💡 loaded instance: testuser1/mydata
💡 did not register local instance on hub

Import lamindb:

import lamindb as ln
from IPython.display import Image, display
💡 lamindb instance: testuser1/mydata

Steps#

In the following, we walk through exemplified steps covering different types of transforms (Transform).

Note

The full notebooks are in this repository.

App upload of phenotypic data #

Register data through app upload from wetlab by testuser1:

# This function mimics the upload of files via the UI
# In reality, you simply drag and drop files into the UI
def run_upload_crispra_result_app():
    ln.setup.login("testuser1")
    transform = ln.Transform(name="Upload GWS CRISPRa result", type="app")
    ln.track(transform)
    output_path = ln.dev.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage)
    output_file = ln.File(output_path, description="Raw data of schmidt22 crispra GWS")
    output_file.save()


run_upload_crispra_result_app()
Hide code cell output
💡 saved: Transform(uid='sWweDfcVz8cv6C', name='Upload GWS CRISPRa result', type='app', updated_at=2023-12-05 17:28:48 UTC, created_by_id=1)
💡 saved: Run(uid='Zz3C6QlOiPji6QndDmGb', run_at=2023-12-05 17:28:48 UTC, transform_id=1, created_by_id=1)

Hit identification in notebook #

Access, transform & register data in drylab by testuser2:

def run_hit_identification_notebook():
    # log in as another user
    ln.setup.login("testuser2")

    # create a new transform to mimic a new notebook (in reality you just run ln.track() in a notebook)
    transform = ln.Transform(name="GWS CRIPSRa analysis", type="notebook")
    ln.track(transform)

    # access the upload file
    input_file = ln.File.filter(key="schmidt22-crispra-gws-IFNG.csv").one()

    # identify hits
    input_df = input_file.load().set_index("id")
    output_df = input_df[input_df["pos|fdr"] < 0.01].copy()

    # register hits in output file
    ln.File(output_df, description="hits from schmidt22 crispra GWS").save()


run_hit_identification_notebook()
Hide code cell output
💡 saved: Transform(uid='ZUYJkioFyll0jc', name='GWS CRIPSRa analysis', type='notebook', updated_at=2023-12-05 17:28:49 UTC, created_by_id=1)
💡 saved: Run(uid='ejH3AYIOYaBecRq9FKD4', run_at=2023-12-05 17:28:49 UTC, transform_id=2, created_by_id=1)

Inspect data flow:

file = ln.File.filter(description="hits from schmidt22 crispra GWS").one()
file.view_flow()
_images/2f250f85856788f7bdd7386ff8c5f836a9b1820350668edb00837c04692ab2ad.svg

Sequencer upload #

Upload files from sequencer:

def run_upload_from_sequencer_pipeline():
    ln.setup.login("testuser1")

    # create a pipeline transform
    ln.track(ln.Transform(name="Chromium 10x upload", type="pipeline"))
    # register output files of the sequencer
    upload_dir = ln.dev.datasets.dir_scrnaseq_cellranger(
        "perturbseq", basedir=ln.settings.storage, output_only=False
    )
    ln.File(upload_dir.parent / "fastq/perturbseq_R1_001.fastq.gz").save()
    ln.File(upload_dir.parent / "fastq/perturbseq_R2_001.fastq.gz").save()


run_upload_from_sequencer_pipeline()
Hide code cell output
💡 saved: Transform(uid='i0TK1bgLlW60NK', name='Chromium 10x upload', type='pipeline', updated_at=2023-12-05 17:28:50 UTC, created_by_id=1)
💡 saved: Run(uid='COBDlYX4MQ9UkEKBsStE', run_at=2023-12-05 17:28:50 UTC, transform_id=3, created_by_id=1)

scRNA-seq bioinformatics pipeline #

Process uploaded files using a script or workflow manager: Pipelines and obtain 3 output files in a directory filtered_feature_bc_matrix/:

def run_scrna_analysis_pipeline():
    ln.setup.login("testuser2")
    transform = ln.Transform(name="Cell Ranger", version="7.2.0", type="pipeline")
    ln.track(transform)
    # access uploaded files as inputs for the pipeline
    input_files = ln.File.filter(key__startswith="fastq/perturbseq").all()
    input_paths = [file.stage() for file in input_files]
    # register output files
    output_files = ln.File.from_dir("./mydata/perturbseq/filtered_feature_bc_matrix/")
    ln.save(output_files)

    # Post-process these 3 files
    transform = ln.Transform(
        name="Postprocess Cell Ranger", version="2.0", type="pipeline"
    )
    ln.track(transform)
    input_files = [f.stage() for f in output_files]
    output_path = ln.dev.datasets.schmidt22_perturbseq(basedir=ln.settings.storage)
    output_file = ln.File(output_path, description="perturbseq counts")
    output_file.save()


run_scrna_analysis_pipeline()
Hide code cell output
💡 saved: Transform(uid='8StE1JJoOxjRNE', name='Cell Ranger', version='7.2.0', type='pipeline', updated_at=2023-12-05 17:28:52 UTC, created_by_id=1)
💡 saved: Run(uid='WNv1HOUPAabeRlj8Inqv', run_at=2023-12-05 17:28:52 UTC, transform_id=4, created_by_id=1)
💡 saved: Transform(uid='M40TF8vwfInfTI', name='Postprocess Cell Ranger', version='2.0', type='pipeline', updated_at=2023-12-05 17:28:52 UTC, created_by_id=1)
💡 saved: Run(uid='uKZFrEiL96jPATDDqwQ9', run_at=2023-12-05 17:28:52 UTC, transform_id=5, created_by_id=1)

Inspect data flow:

output_file = ln.File.filter(description="perturbseq counts").one()
output_file.view_flow()
_images/dd96c05266788ab7047854a7ed22eada5a7e129eeb8a39a6d890e1bd69e1b360.svg

Integrate scRNA-seq & phenotypic data #

Integrate data in a notebook:

def run_integrated_analysis_notebook():
    import scanpy as sc

    # create a new transform to mimic a new notebook (in reality you just run ln.track() in a notebook)
    transform = ln.Transform(
        name="Perform single cell analysis, integrate with CRISPRa screen",
        type="notebook",
    )
    ln.track(transform)

    # access the output files of bfx pipeline and previous analysis
    file_ps = ln.File.filter(description__icontains="perturbseq").one()
    adata = file_ps.load()
    file_hits = ln.File.filter(description="hits from schmidt22 crispra GWS").one()
    screen_hits = file_hits.load()

    # perform analysis and register output plot files
    sc.tl.score_genes(adata, adata.var_names.intersection(screen_hits.index).tolist())
    filesuffix = "_fig1_score-wgs-hits.png"
    sc.pl.umap(adata, color="score", show=False, save=filesuffix)
    filepath = f"figures/umap{filesuffix}"
    file = ln.File(filepath, key=filepath)
    file.save()
    filesuffix = "fig2_score-wgs-hits-per-cluster.png"
    sc.pl.matrixplot(
        adata, groupby="cluster_name", var_names=["score"], show=False, save=filesuffix
    )
    filepath = f"figures/matrixplot_{filesuffix}"
    file = ln.File(filepath, key=filepath)
    file.save()


run_integrated_analysis_notebook()
Hide code cell output
💡 saved: Transform(uid='keqRYRYKfzsUNz', name='Perform single cell analysis, integrate with CRISPRa screen', type='notebook', updated_at=2023-12-05 17:28:54 UTC, created_by_id=1)
💡 saved: Run(uid='cthG0eTFv6m1QYePYeMP', run_at=2023-12-05 17:28:54 UTC, transform_id=6, created_by_id=1)
WARNING: saving figure to file figures/umap_fig1_score-wgs-hits.png
WARNING: saving figure to file figures/matrixplot_fig2_score-wgs-hits-per-cluster.png

Review results#

Let’s load one of the plots:

# track the current notebook as transform
ln.track()
file = ln.File.filter(key__contains="figures/matrixplot").one()
file.stage()
Hide code cell output
💡 notebook imports: ipython==8.18.1 lamindb==0.63.2 scanpy==1.9.6
💡 saved: Transform(uid='1LCd8kco9lZUz8', name='Project flow', short_name='project-flow', version='0', type=notebook, updated_at=2023-12-05 17:28:55 UTC, created_by_id=1)
💡 saved: Run(uid='Rn0TeEHZymvbwP1lPMVU', run_at=2023-12-05 17:28:55 UTC, transform_id=7, created_by_id=1)
PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/jy85ejZ8KjmETKQlB7db.png')
display(Image(filename=file.path))
_images/cc92513294ee37e9961661fdc08e251027317eb7a1f5308c26df308bc57787f5.png

We see that the image file is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:

file.view_flow()
_images/8f8765e2ab092730ecbbd6b74d61c67300bfe900dbd0b7299e305103d06bd5fe.svg

Alternatively, we can also look at the sequence of transforms:

transform = ln.Transform.search("Bird's eye view", return_queryset=True).first()
transform.parents.df()
uid name short_name version type latest_report_id source_file_id reference reference_type initial_version_id updated_at created_by_id
id
4 8StE1JJoOxjRNE Cell Ranger None 7.2.0 pipeline None None None None None 2023-12-05 17:28:52.142844+00:00 1
transform.view_parents()
_images/3bf8eb48e79440ff3cc08f2157aa2b98e2d631b708fa79b74c59d89b5bfafcf1.svg

Understand runs#

We tracked pipeline and notebook runs through run_context, which stores a Transform and a Run record as a global context.

File objects are the inputs and outputs of runs.

What if I don’t want a global context?

Sometimes, we don’t want to create a global run context but manually pass a run when creating a file:

run = ln.Run(transform=transform)
ln.File(filepath, run=run)
When does a file appear as a run input?

When accessing a file via stage(), load() or backed(), two things happen:

  1. The current run gets added to file.input_of

  2. The transform of that file gets added as a parent of the current transform

You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?

You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:

file.load(is_run_input=True)

Query by provenance#

We can query or search for the notebook that created the file:

transform = ln.Transform.search("GWS CRIPSRa analysis", return_queryset=True).first()

And then find all the files created by that notebook:

ln.File.filter(transform=transform).df()
uid storage_id key suffix accessor description version size hash hash_type transform_id run_id initial_version_id visibility key_is_virtual updated_at created_by_id
id
2 eReE2e3ckvZ8R4eLp7GH 1 None .parquet DataFrame hits from schmidt22 crispra GWS None 18368 fjvT1POciz7QFCK6Wzaflw md5 2 2 None 1 True 2023-12-05 17:28:50.008327+00:00 1

Which transform ingested a given file?

file = ln.File.filter().first()
file.transform
Transform(uid='sWweDfcVz8cv6C', name='Upload GWS CRISPRa result', type='app', updated_at=2023-12-05 17:28:48 UTC, created_by_id=1)

And which user?

file.created_by
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2023-12-05 17:28:50 UTC)

Which transforms were created by a given user?

users = ln.User.lookup()
ln.Transform.filter(created_by=users.testuser2).df()
uid name short_name version type reference reference_type updated_at latest_report_id source_file_id initial_version_id created_by_id
id

Which notebooks were created by a given user?

ln.Transform.filter(created_by=users.testuser2, type="notebook").df()
uid name short_name version type reference reference_type updated_at latest_report_id source_file_id initial_version_id created_by_id
id

We can also view all recent additions to the entire database:

ln.view()
Hide code cell output
File
uid storage_id key suffix accessor description version size hash hash_type transform_id run_id initial_version_id visibility key_is_virtual updated_at created_by_id
id
10 jy85ejZ8KjmETKQlB7db 1 figures/matrixplot_fig2_score-wgs-hits-per-clu... .png None None None 28814 ijpft7zAYShlKDXYYAD4hw md5 6 6 None 1 True 2023-12-05 17:28:54.705790+00:00 1
9 rUoEFxbyxBeClkESLup9 1 figures/umap_fig1_score-wgs-hits.png .png None None None 118999 74WuaFnZeoMTvSpY--lbrA md5 6 6 None 1 True 2023-12-05 17:28:54.487354+00:00 1
8 HKVOlOmZsnNIQQRpg7XR 1 schmidt22_perturbseq.h5ad .h5ad AnnData perturbseq counts None 20659936 la7EvqEUMDlug9-rpw-udA md5 5 5 None 1 False 2023-12-05 17:28:53.284090+00:00 1
7 WUxDtWiOKfSKwUOrmK9q 1 perturbseq/filtered_feature_bc_matrix/matrix.m... .mtx.gz None None None 6 2MJ-1unPDBstlYn_0gYjTQ md5 4 4 None 1 False 2023-12-05 17:28:52.170562+00:00 1
6 XHzkBef3uxu1lsHFJrdQ 1 perturbseq/filtered_feature_bc_matrix/barcodes... .tsv.gz None None None 6 oQ0MhnhULY48eBQJMh6y7Q md5 4 4 None 1 False 2023-12-05 17:28:52.170039+00:00 1
5 j7Qljz0eomhT3qW43iFR 1 perturbseq/filtered_feature_bc_matrix/features... .tsv.gz None None None 6 wTJIxZsss1coTLm-uJO-fQ md5 4 4 None 1 False 2023-12-05 17:28:52.169455+00:00 1
4 usonpFNYuzFKx7rhkHDs 1 fastq/perturbseq_R2_001.fastq.gz .fastq.gz None None None 6 K9b0_r0vL-87-7418XyMWQ md5 3 3 None 1 False 2023-12-05 17:28:50.991948+00:00 1
Run
uid transform_id run_at created_by_id report_id is_consecutive reference reference_type
id
1 Zz3C6QlOiPji6QndDmGb 1 2023-12-05 17:28:48.150824+00:00 1 None None None None
2 ejH3AYIOYaBecRq9FKD4 2 2023-12-05 17:28:49.950382+00:00 1 None None None None
3 COBDlYX4MQ9UkEKBsStE 3 2023-12-05 17:28:50.980655+00:00 1 None None None None
4 WNv1HOUPAabeRlj8Inqv 4 2023-12-05 17:28:52.146736+00:00 1 None None None None
5 uKZFrEiL96jPATDDqwQ9 5 2023-12-05 17:28:52.181051+00:00 1 None None None None
6 cthG0eTFv6m1QYePYeMP 6 2023-12-05 17:28:54.251212+00:00 1 None None None None
7 Rn0TeEHZymvbwP1lPMVU 7 2023-12-05 17:28:55.095755+00:00 1 None None None None
Storage
uid root type region updated_at created_by_id
id
1 EgNDmwkd /home/runner/work/lamin-usecases/lamin-usecase... local None 2023-12-05 17:28:45.786262+00:00 1
Transform
uid name short_name version type latest_report_id source_file_id reference reference_type initial_version_id updated_at created_by_id
id
7 1LCd8kco9lZUz8 Project flow project-flow 0 notebook None None None None None 2023-12-05 17:28:55.092823+00:00 1
6 keqRYRYKfzsUNz Perform single cell analysis, integrate with C... None None notebook None None None None None 2023-12-05 17:28:54.245414+00:00 1
5 M40TF8vwfInfTI Postprocess Cell Ranger None 2.0 pipeline None None None None None 2023-12-05 17:28:52.178492+00:00 1
4 8StE1JJoOxjRNE Cell Ranger None 7.2.0 pipeline None None None None None 2023-12-05 17:28:52.142844+00:00 1
3 i0TK1bgLlW60NK Chromium 10x upload None None pipeline None None None None None 2023-12-05 17:28:50.977361+00:00 1
2 ZUYJkioFyll0jc GWS CRIPSRa analysis None None notebook None None None None None 2023-12-05 17:28:49.946493+00:00 1
1 sWweDfcVz8cv6C Upload GWS CRISPRa result None None app None None None None None 2023-12-05 17:28:48.147372+00:00 1
User
uid handle name updated_at
id
2 bKeW4T6E testuser2 Test User2 2023-12-05 17:28:52.134000+00:00
1 DzTjkKse testuser1 Test User1 2023-12-05 17:28:50.968192+00:00
Hide code cell content
!lamin login testuser1
!lamin delete --force mydata
!rm -r ./mydata
✅ logged in with email testuser1@lamin.ai (uid: DzTjkKse)
💡 deleting instance testuser1/mydata
✅     deleted instance settings file: /home/runner/.lamin/instance--testuser1--mydata.env
✅     instance cache deleted
✅     deleted '.lndb' sqlite file
❗     consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/mydata