lovely.gae 0.5.0a8

Restoring

We restore snapshot 1, which actually means we delete all “Dummy” objects because this is the only kind in s1. The “Yummy” objects are kept.

>>> s1.restore()

>>> run_tasks(recursive=True)
1

>>> Dummy.all().count()
0
>>> Yummy.all().count()
220

Let us restore s2.

>>> s2.restore()
>>> run_tasks(recursive=True)
4
>>> Dummy.all().count()
220
>>> Yummy.all().count()
220

Deleting

Whe delete is called on a snapshot the range backup objects also get deleted.

>>> from lovely.gae.snapshot import models
>>> s2k = s2.key()
>>> s2.delete()
>>> models.RangeBackup.all().filter('snapshot', s2k).count()
0

Creating Snapshots via http

For testing we setup a wsgi application.

>>> from webtest import TestApp
>>> from lovely.gae.snapshot import getApp
>>> app = TestApp(getApp())

We can now create a snapshot via this url:

>>> res = app.get('/lovely.gae/snapshot.CreateSnapshotHandler')
>>> res.status
'200 OK'
>>> print res.body
Created Snapshot 443
Kinds: ['Dummy', 'Yummy']

Let us complete all jobs.

>>> run_tasks(recursive=True)
8

Kinds are specified via query string.

>>> res = app.get('/lovely.gae/snapshot.CreateSnapshotHandler?kinds=Yummy')
>>> res.status
'200 OK'
>>> print res.body
Created Snapshot 444
Kinds: [u'Yummy']

>>> res = app.get('/lovely.gae/snapshot.CreateSnapshotHandler?kinds=Yummy&kinds=Dummy')
>>> res.status
'200 OK'
>>> print res.body
Created Snapshot 445
Kinds: [u'Dummy', u'Yummy']

Let us complete all jobs.

>>> run_tasks(recursive=True)
12

We can also define a batch size for the snapshot backup ranges.

>>> res = app.get('/lovely.gae/snapshot.CreateSnapshotHandler?batchsize=100')
>>> res.status
'200 OK'
>>> print res.body
Created Snapshot 446
Kinds: ['Dummy', 'Yummy']

Let us complete all jobs again, which are now more per type.

>>> run_tasks(recursive=True)
10

Downloading Snapshots

Snapshots can be downloaded as a single file, which then can be used directly as a development datastore file.

Downloads are done with a client script that uses the remote api to fetch the data. We just test the actual method that get’s called here.

>>> from lovely.gae.snapshot import client
>>> import tempfile, shutil, os
>>> tmp = tempfile.mkdtemp()
>>> f446 = client.download_snapshot(tmp)
Downloading snapshot: 446
Downloading RB:446:Dummy:...
Downloading RB:446:Dummy:...
Downloading RB:446:Dummy:...
Downloading RB:446:Yummy:...
Downloading RB:446:Yummy:...
Downloading RB:446:Yummy:...
Saving to /...snapshot.446
>>> os.listdir(tmp)
['snapshot.446']

By default the latest snapshot gets downloaded, but we can also specify the id of the snapshot to fetch.

>>> file_name = client.download_snapshot(tmp, 443)
Downloading snapshot: 443
Downloading RB:443:Dummy:0000000000:...
Downloading RB:443:Dummy:0000000001:...
Downloading RB:443:Yummy:0000000000:...
Downloading RB:443:Yummy:0000000001:...
Saving to .../snapshot.443
>>> os.listdir(tmp)
['snapshot.443', 'snapshot.446']

If the file already exists we get an exception.

>>> ignored = client.download_snapshot(tmp, 443)
Traceback (most recent call last):
...
RuntimeError: Out file exists '.../snapshot.443'

Let us test if the file works.

>>> from google.appengine.api.datastore_file_stub import DatastoreFileStub
>>> dfs = DatastoreFileStub('lovely-gae-testing',
...                         f446, None)
>>> sum(map(len, dfs._DatastoreFileStub__entities.values()))
440

Typed Lists

This property converts model instances to keys and ensures a length.

>>> from lovely.gae.db.property import TypedListProperty
>>> from google.appengine.ext import db

Let us create a model

>>> class Yummy(db.Model): pass
>>> class Bummy(db.Model): pass

We can now reference Yummy instances with our property. Note that we can also use the kind name as an argument of the kind.

>>> class Dummy(db.Model):
...     yummies = TypedListProperty(Yummy)
...     bummies = TypedListProperty('Bummy', length=3)

The kind arguement needs to be a subclass kind name or a db.Model.

>>> TypedListProperty(object)
Traceback (most recent call last):
...
ValueError: Kind needs to be a subclass of db.Model

>>> dummy = Dummy()
>>> dummy_key = dummy.put()
>>> yummy1 = Yummy()
>>> yummy1_key = yummy1.put()
>>> dummy.yummies = [yummy1]

We cannot set any other type.

>>> bummy1 = Bummy()
>>> bummy1_key = bummy1.put()
>>> dummy.yummies = [bummy1]
Traceback (most recent call last):
...
BadValueError: Wrong kind u'Bummy'

The length needs to match if defined (see above).

>>> dummy.bummies = [bummy1]
Traceback (most recent call last):
...
BadValueError: Wrong length need 3 got 1

>>> dummy.bummies = [bummy1, bummy1, bummy1]
>>> dummy_key == dummy.put()
True

Case-Insensitive String Property

This property allows searching for the lowercase prefix in a case-insensitive manner. This is usefull for autocomplete implementations where we do not want to have a seperate property just for searching.

>>> from lovely.gae.db.property import IStringProperty
>>> class Foo(db.Model):
...     title = IStringProperty()

>>> f1 = Foo(title='Foo 1')
>>> kf1 = f1.put()
>>> f2 = Foo(title='Foo 2')
>>> kf2 = f2.put()
>>> f3 = Foo(title='foo 3')
>>> kf3 = f3.put()
>>> f4 = Foo(title=None)
>>> kf4 = f4.put()

The property does not allow the special seperator character which is just one below the highest unicode character,

>>> f3 = Foo(title='Foo 3' + IStringProperty.SEPERATOR)
Traceback (most recent call last):
...
BadValueError: Not all characters in property title

Note that if we want to do an exact search, we have to use a special filter that can be created by the property instance.

>>> [f.title for f in Foo.all().filter('title =', 'foo 1')]
[]

The “equal” filter arguments can be computed with a special method on the property.

>>> ef = Foo.title.equals_filter('Foo 1')
>>> ef
('title =', u'foo 1\xef\xbf\xbcFoo 1')

>>> [f.title for f in Foo.all().filter(*ef)]
[u'Foo 1']

Let us try with inequallity, e.g. prefix search. Prefix search is normally done with two filters using the highest unicode character.

Search for all that starts with “fo” case-insensitive.

>>> q = Foo.all()
>>> q = q.filter('title >=', 'fo')
>>> q = q.filter('title <', 'fo' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'Foo 1', u'Foo 2', u'foo 3']

Search for all that starts with ‘foo 1’

>>> q = Foo.all()
>>> q = q.filter('title >=', 'foo 1')
>>> q = q.filter('title <', 'foo 1' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'Foo 1']

>>> q = Foo.all()
>>> q = q.filter('title >=', 'foo 2')
>>> q = q.filter('title <=', 'foo 2' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'Foo 2']

>>> q = Foo.all()
>>> q = q.filter('title >=', 'foo 3')
>>> q = q.filter('title <=', 'foo 3' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'foo 3']

Pickle Property

A pickle property can hold any pickleable python object.

>>> from lovely.gae.db.property import PickleProperty

>>> class Pickie(db.Model):
...     data = PickleProperty()

>>> pickie = Pickie(data={})
>>> pickie.data
{}
>>> kp = pickie.put()
>>> pickie.data
{}
>>> pickie = db.get(kp)
>>> pickie.data
{}
>>> pickie.data = {'key':501*"x"}
>>> kp = pickie.put()
>>> pickie.data
{'key': 'xx...xx'}

If the value is not pickleable we get a validation error.

>>> pickie = Pickie(data=dict(y=lambda x:x))
Traceback (most recent call last):
BadValueError: Property 'data' must be pickleable:
(Can't pickle <function <lambda> at ...>:
it's not found as __main__.<lambda>)

Safe ReferenceProperty

>>> from lovely.gae.db.property import SafeReferenceProperty

We use a new model with a gae reference and our safe reference.

>>> class Refie(db.Model):
...     ref   = db.ReferenceProperty(Yummy, collection_name='ref_ref')
...     sfref = SafeReferenceProperty(Yummy, collection_name='sfref_ref')

>>> refie = Refie()
>>> refie.sfref is None
True
>>> refie.ref is None
True

An object to be referenced.

>>> refyummy1 = Yummy()
>>> ignore = refyummy1.put()

Set the references to our yummy object.

>>> refie.sfref = refyummy1
>>> refie.sfref
<Yummy object at ...>
>>> refie.ref = refyummy1
>>> refie.ref
<Yummy object at ...>

>>> refieKey = refie.put()

Now we delete the referenced object.

>>> refyummy1.delete()

And reload our referencing object.

>>> refie = db.get(refieKey)

The gae reference raises an exception.

>>> refie.ref
Traceback (most recent call last):
Error: ReferenceProperty failed to be resolved

We catch the logs here.

>>> import logging
>>> from StringIO import StringIO
>>> log = StringIO()
>>> handler = logging.StreamHandler(log)
>>> logger = logging.getLogger('lovely.gae.db')
>>> logger.setLevel(logging.INFO)
>>> logger.addHandler(handler)

Our safe reference returns None.

>>> pos = log.tell()
>>> refie.sfref is None
True

Let’s see what the log contains.

>>> log.seek(pos)
>>> print log.read()
Unresolved Reference for "Refie._sfref" set to None

Accessing the stale property once again we will see it was reset to None:

>>> pos = log.tell()
>>> refie.sfref is None
True

>>> log.seek(pos)
>>> print log.read() == ''
True

The property get set to None if the reference points to a dead object but only if the property is not required:

>>> class Requy(db.Model):
...     sfref = SafeReferenceProperty(Yummy, collection_name='req_sfref_ref',
...                                   required=True)

>>> refyummy1 = Yummy()
>>> ignore = refyummy1.put()

>>> requy = Requy(sfref = refyummy1)
>>> requyKey = requy.put()

>>> requy.sfref
<Yummy object at ...>

>>> refyummy1.delete()

>>> requy = db.get(requyKey)

>>> pos = log.tell()
>>> requy.sfref is None
True

>>> log.seek(pos)
>>> print log.read()
Unresolved Reference for "Requy._sfref" will remain because it is required