What is RhyDB?

RhyDB is a high-performance analytical database for sequence alignment data.

Properties

RhyDB (formerly known as SILO) is a database with optimized support for sequence alignment data. Conceptually, it is like other relational databases such as PostgreSQL, MySQL and DuckDB:

  • A database consists of tables, and a table consists of columns.
  • Columns are typed and RhyDB supports typical types for strings, numbers and dates.
  • Queries are formulated in a declarative language which supports, among others, filters, joins, projections and unions.
  • There is a query optimizer that translates the query to an efficient physical execution plan.

What makes RhyDB unique are its capabilities to handle sequencing data and features optimized for genomic epidemiological workflows:

  • RhyDB has special column types for nucleotide and amino acid sequences.
  • Sequence alignments are stored using compressed bitmaps, enabling both efficient storage and querying.
  • The query language offers syntax and functions specific to working with sequence data.
  • It has built-in functionalities for working with lineage hierarchies.

Where it fits

RhyDB is a foundational software designed to be used either directly by data scientists or by other software. It aims to be very unopinionated: it stores and returns data as they are given but does not impose any biological or statistical assumptions; and it does not enforce any data schema beyond being relational (i.e. shaped as tables).

RhyDB is used by various other tools that the team of RhyDB is also working on:

  • Loculus is a comprehensive and modular package for sharing and managing pathogen sequence data. It does not only store data as provided but manages the bioinformatics processing (QC, alignment, lineage calling, etc.), versioning, authentication and much more. Loculus uses internally RhyDB as its query engine: when a user searches sequences on Loculus, RhyDB performs the work.
  • Pathoplexus is one specific instance of Loculus for facilitating global sharing of viral genomic data.
  • LAPIS is a web service focusing on providing a simple web API to query sequencing data. It sits on top of SILO and translates web API calls to RhyDB queries, and it transforms RhyDB’s result to various data formats.
  • CoV-Spectrum and GenSpectrum are web dashboards to discover and analyze variants of various viruses; they query RhyDB heavily.

A short history and use in other projects

  • January 2021: The origin of RhyDB goes back to when the first Variant of Concern (VOC) of SARS-CoV-2 was detected and the global community started to ramp up sequencing. Chaoran Chen, based at the cEvo group of Tanja Stadler at ETH Zurich, started the development of CoV-Spectrum, a dashboard that allowed the interactive search of mutations in the growing dataset. The first version of LAPIS was developed as its backend.
  • July 2021: While LAPIS used PostgreSQL at the beginning to search mutations, this soon became too inefficient with the growing data. LAPIS (written in Java/Kotlin) started to implement its own columnar storage format and query engine for the alignment data. It consumed a lot of RAM but facilitated the querying of several million SARS-CoV-2 sequences.
  • June 2022: Over time, as the number of sequences continued to rise, LAPIS struggled increasingly. In collaboration with Thomas Neumann and the Chair for Database Systems at TU Munich, Alexander Taepper (back-then a student at TUM) wrote a master’s thesis in which he developed the first research prototype of SILO.
  • May 2023: Alex joined the cEvo group to continue his work on SILO.
  • (TODO: when did we start using SILO in LAPIS/CoV-Spectrum? What are other milestones?)
  • August 2024: After a year of development, Pathoplexus and Loculus were launched to facilitate sharing of pathogen sequencing data. SILO and LAPIS served as their query engine.
  • (TODO: something on the W-ASAP project)
  • August 2026: SILO received its new query language and query optimizer, allowing it to be used more flexibly and independently. With these changes, SILO was announced publicly as a standalone project and was renamed RhyDB.

About the name

(TODO)