Skip to main content

linkml-store

Project description

linkml-store

An AI-ready data management and integration platform. LinkML-Store provides an abstraction layer over multiple different backends (including DuckDB, MongoDB, Neo4j, and local filesystems), allowing for common query, index, and storage operations.

For full documentation, see https://linkml.io/linkml-store/

See these slides for a high level overview.

Warning LinkML-Store is still undergoing changes and refactoring, APIs and command line options are subject to change!

Quick Start

Install, add data, query it:

pip install linkml-store[all]
linkml-store -d duckdb:///db/my.db -c persons insert data/*.json
linkml-store -d duckdb:///db/my.db -c persons query -w "occupation: Bricklayer"

Index it, search it:

linkml-store -d duckdb:///db/my.db -c persons index -t llm
linkml-store -d duckdb:///db/my.db -c persons search "all persons employed in construction"

Validate it:

linkml-store -d duckdb:///db/my.db -c persons validate

Basic usage

The CRUDSI pattern

Most database APIs implement the CRUD pattern: Create, Read, Update, Delete. LinkML-Store adds Search and Inference to this pattern, making it CRUDSI.

The notion of "Search" and "Inference" is intended to be flexible and extensible, including:

  • Search
    • Traditional keyword search
    • Search using LLM Vector embeddings (without a dedicated vector database)
    • Pluggable specialized search, e.g. genomic sequence (not yet implemented)
  • Inference (encompassing validation, repair, and inference of missing data)
    • Classic rule-based inference
    • Inference using LLM Retrieval Augmented Generation (RAG)
    • Statistical/ML inference

Features

Multiple Adapters

LinkML-Store is designed to work with multiple backends, giving a common abstraction layer

Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...

The intent is to give a union of all features of each backend. For example, analytic faceted queries are provided for all backends, not just Solr.

Composable indexes

Many backends come with their own indexing and search schemes. Classically this was Lucene-based indexes, now it is semantic search using LLM embeddings.

LinkML store treats indexing as an orthogonal concern - you can compose different indexing schemes with different backends. You don't need to have a vector database to run embedding search!

See How to Use-Semantic-Search

Use with LLMs

TODO - docs

Validation

LinkML-Store is backed by LinkML, which allows for powerful expressive structural and semantic constraints.

See Indexing JSON

and Referential Integrity

Web API

There is a preliminary API following HATEOAS principles implemented using FastAPI.

To start you should first create a config file, e.g. db/conf.yaml:

Then run:

export LINKML_STORE_CONFIG=./db/conf.yaml
make api

The API returns links as well as data objects, it's recommended to use a Chrome plugin for JSON viewing for exploring the API. TODO: add docs here.

The main endpoints are:

  • http://localhost:8000/ - the root of the API
  • http://localhost:8000/pages/ - browse the API via HTML
  • http://localhost:8000/docs - the Swagger UI

Streamlit app

make app

Background

See these slides for more details

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

linkml_store-0.2.2.tar.gz (87.9 kB view details)

Uploaded Source

Built Distribution

linkml_store-0.2.2-py3-none-any.whl (116.2 kB view details)

Uploaded Python 3

File details

Details for the file linkml_store-0.2.2.tar.gz.

File metadata

  • Download URL: linkml_store-0.2.2.tar.gz
  • Upload date:
  • Size: 87.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.5

File hashes

Hashes for linkml_store-0.2.2.tar.gz
Algorithm Hash digest
SHA256 38fd18b7c8186b97df9bd9d490171a8e0417e5edf0e2d4017836bd620be5a532
MD5 120aa9d1be46c81a21e4971c0c021b7e
BLAKE2b-256 db8d60387a56c75c1eea7bdc19dc5f0a3e648f92d6d17b4ba2425ed2f8098e48

See more details on using hashes here.

Provenance

File details

Details for the file linkml_store-0.2.2-py3-none-any.whl.

File metadata

  • Download URL: linkml_store-0.2.2-py3-none-any.whl
  • Upload date:
  • Size: 116.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.5

File hashes

Hashes for linkml_store-0.2.2-py3-none-any.whl
Algorithm Hash digest
SHA256 fe5987c3dbb5c8ec1cad56566da3c77b4320596c92f11375ac1a86126b66ee66
MD5 709aecc1fe95e362f935e7e57692f757
BLAKE2b-256 d5c38f00dc8d219908cd642b335c4a56ebfb1bc32ef53aa6c839412121f12f89

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page