Storage

Object properties data is stored into storages. You may define one or multiple storages, each property can be mapped only to the single storage.

E.g. you may store part of the object data into files and some values in RDBMS database.

You may use ready-made storage classes or create your own.

Contents

• Storage
  • Defining storages
  • File-based storage
    • JSON
    • YAML
    • Pickle
    • MessagePack
    • CBOR
  • Database storages
    • RDBMS
    • Key-value
  • Custom storages
  • Loading all objects from storage
  • Auto-generated primary keys
  • Storage cleanup

Defining storages

import smartobject

# define default storage
smartobject.define_storage(smartobject.JSONStorage())

# define another file storage with id "stor2"
storage2 = smartobject.MessagePackStorage()
# raise errors if data file is not found
storage2.allow_empty = False
smartobject.define_storage(storage2, 'stor2')

# define key-value storage with id "redis1"
smartobject.define_storage(smartobject.RedisStorage(), 'redis1')

# get storage object
r = smartobject.get_storage('redis1')

# purge deleted objects in all storages
smartobject.purge()

File-based storage

File-based storages can not handle properties, marked as “external”.

JSON

class smartobject.storage.JSONStorage(pretty=False)

Bases: smartobject.storage.AbstractFileStorage

Stores object data in JSON format

Uses rapidjson module if installed, otherwise fallbacks to default

Parameters:pretty – if True, store pretty-formatted JSON (indent, sort keys), default is False

cleanup(pks, pattern=None)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete stored object

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

list(pattern=None, **kwargs)

List object files in storage

Parameters:pattern – file pattern (default: all files with {self.ext})

load(pk=None, fname=None, allow_empty=None, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(pattern=None, **kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

prepare_pk(pk)

Converts primary key value to file name

By default, replaces “/” with “___”

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

YAML

class smartobject.storage.YAMLStorage(pretty=True)

Bases: smartobject.storage.AbstractFileStorage

Stores object data in YAML format

Requires pyyaml module

Args: pretty: if True, store pretty-formatted YAML, default is True

cleanup(pks, pattern=None)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete stored object

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

list(pattern=None, **kwargs)

List object files in storage

Parameters:pattern – file pattern (default: all files with {self.ext})

load(pk=None, fname=None, allow_empty=None, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(pattern=None, **kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

prepare_pk(pk)

Converts primary key value to file name

By default, replaces “/” with “___”

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

Pickle

class smartobject.storage.PickleStorage

Bases: smartobject.storage.AbstractFileStorage

Stores object data in Python pickle format

cleanup(pks, pattern=None)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete stored object

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

list(pattern=None, **kwargs)

List object files in storage

Parameters:pattern – file pattern (default: all files with {self.ext})

load(pk=None, fname=None, allow_empty=None, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(pattern=None, **kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

prepare_pk(pk)

Converts primary key value to file name

By default, replaces “/” with “___”

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

MessagePack

class smartobject.storage.MessagePackStorage

Bases: smartobject.storage.AbstractFileStorage

Stores object data in MessagePack format

Requires msgpack-python module

cleanup(pks, pattern=None)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete stored object

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

list(pattern=None, **kwargs)

List object files in storage

Parameters:pattern – file pattern (default: all files with {self.ext})

load(pk=None, fname=None, allow_empty=None, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(pattern=None, **kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

prepare_pk(pk)

Converts primary key value to file name

By default, replaces “/” with “___”

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

CBOR

class smartobject.storage.CBORStorage

Bases: smartobject.storage.AbstractFileStorage

Stores object data in CBOR format

Requires cbor module

cleanup(pks, pattern=None)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete stored object

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

list(pattern=None, **kwargs)

List object files in storage

Parameters:pattern – file pattern (default: all files with {self.ext})

load(pk=None, fname=None, allow_empty=None, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(pattern=None, **kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

prepare_pk(pk)

Converts primary key value to file name

By default, replaces “/” with “___”

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

Database storages

RDBMS

SQLAlchemy-based storage.

class smartobject.storage.SQLAStorage(db, table, pk_field='id')

Bases: smartobject.storage.AbstractStorage

RDBMS storage via SQLAlchemy

Implements get_prop/set_prop methods, can be used for external properties

Can save/load objects

The table for objects must be created manually before the class methods can work

Parameters:

• db – either SQLAlchemy instance, which implements “execute” method (engine, connection) or callable (function) which returns such instance on demand
• table – database table
• pk_field – primary key field in table (default: id)

cleanup(pks, **kwargs)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete object data from the storage

Parameters:

• pk – object primary key
• props – list of object properties mapped to store

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

load(pk, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(**kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

Key-value

Redis-based storage.

class smartobject.storage.RedisStorage(host='localhost', port=6379, db=0, prefix='', **kwargs)

Bases: smartobject.storage.AbstractStorage

Redis storage

Implements get_prop/set_prop methods, can be used for external properties

Can not save/load objects

Stores object properties in format {pk}/{prop}

Deletes properties from Redis server when object is deleted

Parameters:

• host – Redis host
• port – Redis port
• db – Redis DB ID
• prefix – record prefix
• **kwargs – passed to Python redis module as-is

cleanup(pks, **kwargs)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete object data from the storage

Parameters:

• pk – object primary key
• props – list of object properties mapped to store

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

load(pk, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(**kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

purge(**kwargs)

Purge deleted objects

save(pk, data, modified, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

Custom storages

You may use the following classes as prototypes for the own storages:

class smartobject.storage.AbstractStorage

Bases: object

Abstract storage class which can be used as storage template

cleanup(pks, **kwargs)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete object data from the storage

Parameters:

• pk – object primary key
• props – list of object properties mapped to store

get_prop(pk, prop, **kwargs)

Get single object property from the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property

load(pk, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(**kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

load_by_prop(key, prop, **kwargs)

Load object data from the storage by secondary key

Parameters:

• key – object key value
• prop – object key name

purge(**kwargs)

Purge deleted objects

save(pk, data, modified, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

set_prop(pk, prop, value, **kwargs)

Save single object property to the storage

Must be implemented if external properties are mapped

Parameters:

• pk – object primary key
• prop – object property
• value – property value

class smartobject.storage.AbstractFileStorage

Bases: smartobject.storage.AbstractStorage

Abstract class for file-based storages

Has the following properties:

allow_empty: if no object data file is found, return empty data (defalt: True)

instant_delete: delete object files instantly (default: True)

File-based storages usually don’t implement get_prop/set_prop methods

File-based storages have additional “fname” property for load() method

cleanup(pks, pattern=None)

Cleanup storage

Deletes from storage all items which are not in provided list of primary keys

Parameters:pks – list of primary keys Returns:number of deleted items

delete(pk, props, **kwargs)

Delete stored object

list(pattern=None, **kwargs)

List object files in storage

Parameters:pattern – file pattern (default: all files with {self.ext})

load(pk=None, fname=None, allow_empty=None, **kwargs)

Load object data from the storage

Parameters:pk – object primary key

load_all(pattern=None, **kwargs)

Load object data for all objecta and return it as list generator

Each object in list is a dictionary with fields:

data: object data info: additional info, e.g. fname the data is loaded from

info is passed to object after_load() method as kwargs

prepare_pk(pk)

Converts primary key value to file name

By default, replaces “/” with “___”

purge(**kwargs)

Purge deleted objects

save(pk=None, data={}, modified={}, **kwargs)

Save object data to the storage

Parameters:

• pk – object primary key
• data – full object data
• modified – modified properties only

Returns:

object primary key

Loading all objects from storage

Sometimes it’s useful to preload all objects of the certain type. To make this possible, storages have load_all() method, which returns data for all objects in the specified storage.

The data is returned as list generator, where each item is a dict with fields “data” - contains object data, and “info” - contains additional object info (e.g. file name the object is loaded from). “info” usually should be passed to object after_load() method as kwargs.

SmartObject factory allows to do this in a few lines of code:

factory = smartobject.SmartObjectFactory(MyObjClass, autosave=True)
# if no storage_id arg is specified, objects are loaded from the default
# storage
factory.load_all()

Auto-generated primary keys

Both RDBMS and file storages support auto-generated primary keys. For RDBMS, keys are generated by SQL server (key is auto-incremented integer field). For file storages, key are generated as UUIDs.

When object with null primary key field is being saved for the first time, SmartObject tries to obtain and set primary key field from storage.

To simplify this, you may create SmartObject factory with autosave=True argument: all objects, created by this factory, will be auto-saved as soon as created.

factory = smartobject.SmartObjectFactory(MyObjClass, autosave=True)
# the new object is automatically saved and its primary key is set from the
# storage
obj = factory.create()

Storage cleanup

Sometimes storage may contain orphaned objects. To clean them, it has cleanup(pks) method, where pks is a list of object primary keys to leave, while all other objects will be removed.

if you use SmartObject factory, you may use its cleanup(storage_id) method as well. The method removes all objects from the specified storage, except the objects in factory.