Skip to main content

venomx

Project description

venomx - Vector Embedding Named Object Model indeX

VENOMX is an exchange standard for vector embeddings, layered on existing vector storage standards, with the goal of sharing vector embeddings for named entities (books, genes, publications, anything in schema.org, ...).

One of the goals is to support an Embedding Hub, potentially layered on existing dataset repositories (e.g. huggingface, zenodo, figshare, ...).

Why?

Why have a standard for embeddings? We have CSVs and Parquet and arrow, what else do we need?

Let's say Barb has made vector embeddings of all of Wikidata using OpenAI text-embedding-ada-002, and distributes them (e.g. on HuggingFace), and Alice downloads from here.

Two months later, Alice wants to use this for RAG querying. But she has forgotten what model was used, and also what version of Wikidata was used. She also doesn't remember what exactly was indexed. Was it the page titles? The descriptions? Or the full Wikipedia text?

Multiply this by multiple datasets, versions, indexing strategies, subsets, and chaos ensues.

The goal of venomx is to provide:

  • a simple YAML format for metadata about an embeddings set
  • a super-simple parquet schema for the embeddings themselves

It is intended to compose with existing standards for model cards and dataset descriptions, rather than replace.

It is also intended to be flexible, and support the following scenarios:

  1. distribute embedding metadata alongside embeddings, with the latter in an efficient format like Parquet
  2. distribute both together in YAML (convenience at the expense of some access-time/space efficiency)
  3. Use of other serializations than YAML for the metadata (JSON, TSV, RDF, Avro, GraphQL, ...)
  4. Use of alternate storage formats for the embeddings (Arrow, HDF5, ...)
  5. Easy composition with your favorite array library (numpy, xarr, pyarrow, ...)
  6. Easy composition with your favorite vector database (ChromaDB, etc)
  7. Use in combination with objects stored in databases like Solr, PostgreSQL, ...

Note that current functionality is highly minimal, but in future there may be plugins e.g. for import/export from vectordbs.

Things that are out of scope including actually creating the embedding and computing over them. There already existing many existing great frameworks for this. Venomx is focused purely on making indexes of embeddings FAIR and easy to share.

Example

This example is based around the Human Phenotype Ontology (HPO). Vector embeddings of HPO are useful for searching for phenotypes, for RAG-type LLM applications, and for applications such as variant prioritization (cosine similarity of vector embeddings could replace traditional ontological semantic similarity measures).

The default way to distribute using venomx is a YAML file with metadata, and a Parquet file:

$ ls
hp.yaml
hp.parquet

The contents of hp.yaml:

description: HPO label index
prefixes:
  HP: http://purl.obolibrary.org/obo/HP_
model:
  name: "text-embedding-ada-002"
model_input_method:
  description: Simple pass through of labels only
  fields: [ "rdfs:label" ]
dataset:
  name: HPO-Jan-2024
  url: http://purl.obolibrary.org/obo/hp/releases/2024-01-01/hp.owl
objects:
  - id: "HP:0000001"
    label: "All"
  - id: "HP:0000002"
    label: "Abnormality of body height"
  # <snip>

Running parquet schema hp.parquet gives the schema for hp.parquet:

{
  "type" : "record",
  "name" : "schema",
  "fields" : [ {
    "name" : "id",
    "type" : [ "null", "int" ],
    "default" : null
  }, {
    "name" : "name",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "embedding",
    "type" : [ "null", {
      "type" : "array",
      "items" : {
        "type" : "record",
        "name" : "list",
        "fields" : [ {
          "name" : "element",
          "type" : [ "null", "float" ],
          "default" : null
        } ]
      }
    } ],
    "default" : null
  }, {
    "name" : "metadata",
    "type" : [ "null", {
      "type" : "map",
      "values" : [ "null", "string" ]
    } ],
    "default" : null
  } ]
}

Command line tools

Although the main purpose of this repo is as a proposed standard, we include some simple tools for basic conversion and validation

Conversion

Currently only two formats are supported:

  • parquet: two files, a metadata yaml file and a parquet file
  • yaml: a combined all-in-one yaml file (may be less efficient)

(THIS IS PROBABLY A BIT CONFUSING AND MAY CHANGE)

The test folder includes an all-in-one example, we can convert that to a dual yaml/parquet format:

venomx convert -f yaml tests/input/example.combined.yaml -t parquet -o tests/output/example.yaml

Validation

venomx validate tests/output/example.yaml

Roadmap

  • Use linkml-arrays standard
  • other embeddings formats (xarr, arrow, ...)
  • ...

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

venomx-0.1.1.tar.gz (11.3 kB view details)

Uploaded Source

Built Distribution

venomx-0.1.1-py3-none-any.whl (11.4 kB view details)

Uploaded Python 3

File details

Details for the file venomx-0.1.1.tar.gz.

File metadata

  • Download URL: venomx-0.1.1.tar.gz
  • Upload date:
  • Size: 11.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for venomx-0.1.1.tar.gz
Algorithm Hash digest
SHA256 844dd995cec070a47f8639943d5e8707fc64787406bd00335163f1b3a5a553b8
MD5 5649a8df0fde246f65a0c4f2cd24100a
BLAKE2b-256 35efb92db89f2267157ac3058198fde03bb65312fb904e31aa8ac39d069cbfa4

See more details on using hashes here.

Provenance

File details

Details for the file venomx-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: venomx-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 11.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for venomx-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 323a68d57c2e9ce3171f02fa0296264213a1ac02d19b4e446f2452259dc4234e
MD5 e2b18e5ffc7eee8f126e4e2f3ba65e49
BLAKE2b-256 4a77608ec0ee8c43450e327d3d82d1a1e34fa18862e91d38f5f80027d3d23c34

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page