pilosa 0.5.0

Quick overview

Assuming Pilosa server is running at localhost:10101 (the default):

import pilosa

# Create the default client
client = pilosa.Client()

# Retrieve the schema
schema = client.schema()

# Create an Index object
myindex = schema.index("myindex")

# Create a Frame object
myframe = myindex.frame("myframe")

# make sure the index and frame exists on the server
client.sync_schema(schema)

# Send a SetBit query. PilosaError is thrown if execution of the query fails.
client.query(myframe.setbit(5, 42))

# Send a Bitmap query. PilosaError is thrown if execution of the query fails.
response = client.query(myframe.bitmap(5))

# Get the result
result = response.result

# Act on the result
if result:
    bits = result.bitmap.bits
    print("Got bits: ", bits)

# You can batch queries to improve throughput
response = client.query(
    myindex.batch_query(
        myframe.bitmap(5),
        myframe.bitmap(10),
    )
)
for result in response.results:
    # Act on the result
    print(result)

Data Model and Queries

schema = Schema()
repository = schema.index("repository")

repository = schema.index("repository",
    column_label="repo_id", time_quantum=pilosa.TimeQuantum.YEAR_MONTH)

stargazer = repository.frame("stargazer")

stargazer = repository.frame("stargazer",
    row_label="stargazer_id", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY)

bitmap_query = stargazer.bitmap(1, 100)  # corresponds to PQL: Bitmap(frame='stargazer', stargazer_id=1)

query = repository.union(bitmap_query1, bitmap_query2)

query = repository.batch_query(
    stargazer.bitmap(1, 100),
    repository.union(stargazer.bitmap(100, 200), stargazer.bitmap(5, 100))
)

query = repository.raw_query("Bitmap(frame='stargazer', stargazer_id=5)")

Pilosa URI

A Pilosa URI has the ${SCHEME}://${HOST}:${PORT} format: * Scheme: Protocol of the URI. Default: http. * Host: Hostname or ipv4/ipv6 IP address. Default: localhost. * Port: Port number. Default: 10101.

All parts of the URI are optional, but at least one of them must be specified. The following are equivalent:

• http://localhost:10101

• http://localhost

• http://:10101

• localhost:10101

• localhost

• :10101

A Pilosa URI is represented by the pilosa.URI class. Below are a few ways to create URI objects:

# create the default URI: http://localhost:10101
uri1 = pilosa.URI()

# create a URI from string address
uri2 = pilosa.URI.address("db1.pilosa.com:20202")

# create a URI with the given host and port
URI uri3 = pilosa.URI(host="db1.pilosa.com", port=20202);

Pilosa Client

In order to interact with a Pilosa server, an instance of pilosa.Client should be created. The client is thread-safe and uses a pool of connections to the server, so we recommend creating a single instance of the client and share it with other objects when necessary.

If the Pilosa server is running at the default address (http://localhost:10101) you can create the default client with default options using:

client = pilosa.Client()

To use a a custom server address, pass the address in the first argument:

client = pilosa.Client("http://db1.pilosa.com:15000")

If you are running a cluster of Pilosa servers, you can create a pilosa.Cluster object that keeps addresses of those servers:

cluster = pilosa.Cluster(
    pilosa.URI.address(":10101"),
    pilosa.URI.address(":10110"),
    pilosa.URI.address(":10111"),
);

# Create a client with the cluster
client = pilosa.Client(cluster)

It is possible to customize the behaviour of the underlying HTTP client by passing client options to the Client constructor:

client = pilosa.Client(cluster,
    connect_timeout=1000,  # if can't connect in  a second, close the connection
    socket_timeout=10000,  # if no response received in 10 seconds, close the connection
    pool_size_per_route=3,  # number of connections in the pool per host
    rety_count=5,  # number of retries before failing the request
)

Once you create a client, you can create indexes, frames and start sending queries.

Here is how you would create a index and frame:

# materialize repository index instance initialized before
client.create_index(repository)

# materialize stargazer frame instance initialized before
client.create_frame(stargazer)

If the index or frame exists on the server, you will receive a PilosaError. You can use ensure_index and ensure_frame methods to ignore existing indexes and frames.

You can send queries to a Pilosa server using the query method of client objects:

response = client.query(frame.bitmap(5))

query method accepts optional columns argument:

response = client.query(frame.bitmap(5),
    columns=True  # return column data in the response
)

Server Response

When a query is sent to a Pilosa server, the server either fulfills the query or sends an error message. In the case of an error, PilosaError is thrown, otherwise a QueryResponse object is returned.

A QueryResponse object may contain zero or more results of QueryResult type. You can access all results using the results property of QueryResponse (which returns a list of QueryResult objects) or you can use the result property (which returns either the first result or None if there are no results):

response = client.query(frame.bitmap(5))

# check that there's a result and act on it
result = response.result
if result:
    # act on the result
}

# iterate on all results
for result in response.results:
    # act on the result

Similarly, a QueryResponse object may include a number of column objects, if columns=True query option was used:

# check that there's a column object and act on it
column = response.column
if column:
    # act on the column

# iterate on all columns
for column in response.columns:
    # act on the column

QueryResult objects contain:

• bitmap property to retrieve a bitmap result,

• count_items property to retrieve column count per row ID entries returned from topn queries,

• count attribute to retrieve the number of rows per the given row ID returned from count queries.

bitmap = response.bitmap
bits = bitmap.bits
attributes = bitmap.attributes

count_items = response.count_items

count = response.count

Running tests

You can run unit tests with:

make test

And both unit and integration tests with:

make test-all

Check the test coverage:

make cover

Generating protobuf classes

Protobuf classes are already checked in to source control, so this step is only needed when the upstream public.proto changes.

Before running the following step, make sure you have the Protobuf compiler installed:

make generate