MicroLlama

The smallest possible LLM API. Build a question and answer interface to your own content in a few minutes. Uses OpenAI embeddings, gpt-3.5 and Faiss, via Langchain.

Usage

1. Combine your source documents into a single JSON file called source.json. It should look like this:

[
    {
        "source": "Reference to the source of your content. Typically a title.",
        "url": "URL for your source. This key is optional.",
        "content": "Your content as a single string. If there's a title or summary, put these first, separated by new lines."
    }, 
    ...
]

See example.source.json for an example.

2. Install dependencies:

pip install langchain faiss-cpu openai fastapi "uvicorn[standard]"

3. Get an OpenAI API key and add it to the environment, e.g. export OPENAI_API_KEY=sk-etc. Note that indexing and querying require OpenAI credits, which aren't free.

4. Run your server with uvicorn serve:app. If the search index doesn't exist, it'll be created and stored.

5. Query your documents at /api/ask?your question or use the simple front-end at /

Deploying your API

On Fly.io

Sign up for a Fly.io account and install flyctl. Then:

fly launch # answer no to Postgres, Redis and deploying now 
fly secrets set OPENAI_API_KEY=sk-etc 
fly deploy

On Google Cloud Run

gcloud run deploy --source . --set-env-vars="OPENAI_API_KEY=sk-etc"

For Cloud Run and other serverless platforms you should probably generate the FAISS index at container build time, to reduce cold starts. See the two commented lines in Dockerfile.

Based on

• Langchain
• Simon Willison's blog post, datasette-openai and datasette-faiss.
• FastAPI
• GPT Index
• Dagster blog post

TODO

• Use splitting which generates more meaningful fragments, e.g. text_splitter = SpacyTextSplitter(chunk_size=700, chunk_overlap=200, separator=" ")