rom Package¶

Rom - the Redis object mapper for Python

Released under the LGPL license version 2.1 and version 3 (you can choose which you’d like to be bound under).

Sponsorships Available¶

Don’t like LGPL? Sponsor the project and get almost any license you want.

This project has been partly sponsored by structd.com . Historically, rom has been used to help support the delivery of millions of food orders for chownow.com, and has been used as the primary backend and prototype for several startups that have made it all the way to series A.

Thank you to our sponsors and those who have consumed our services.

You are welcome for the good service.

Your company link here.

Documentation¶

Updated documentation can be found: https://josiahcarlson.github.io/rom/

What¶

Rom is a package whose purpose is to offer active-record style data modeling within Redis from Python, similar to the semantics of Django ORM, SQLAlchemy, Google’s Appengine datastore, and others.

Why¶

I was building a personal project, wanted to use Redis to store some of my data, but didn’t want to hack it poorly. I looked at the existing Redis object mappers available in Python, but didn’t like the features and functionality offered.

What is available¶

Data types:

Strings (2.x: str/unicode, 3.3+: str), ints, floats, decimals, booleans
datetime.datetime, datetime.date, datetime.time
Json columns (for nested structures)
OneToMany and ManyToOne columns (for model references)
Non-rom ForeignModel reference support

Indexes:

Numeric range fetches, searches, and ordering
Full-word text search (find me entries with col X having words A and B)
Prefix matching (can be used for prefix-based autocomplete)
Suffix matching (can be used for suffix-based autocomplete)
Pattern matching on string-based columns
All indexing except Geo indexing is available when using Redis 2.6.0 and later
Geo indexing available with Redis 3.2.0 and later

Other features:

Per-thread entity cache (to minimize round-trips, easy saving of all entities)
The ability to cache query results and get the key for any other use (see: Query.cached_result())

Getting started¶

Make sure you have Python 2.6, 2.7, or 3.3+ installed
Make sure that you have Andy McCurdy’s Redis client library installed: https://github.com/andymccurdy/redis-py/ or https://pypi.python.org/pypi/redis
Make sure that you have the Python 2 and 3 compatibility library, ‘six’ installed: https://pypi.python.org/pypi/six
(optional) Make sure that you have the hiredis library installed for Python
Make sure that you have a Redis server installed and available remotely
Update the Redis connection settings for rom via rom.util.set_connection_settings() (other connection update options, including per-model connections, can be read about in the rom.util documentation):
```
import redis
from rom import util

util.set_connection_settings(host='myhost', db=7)
```

Warning

If you forget to update the connection function, rom will attempt to connect to localhost:6379 .

Create a model:

import rom

# All models to be handled by rom must derived from rom.Model
class User(rom.Model):
    email = rom.String(required=True, unique=True, suffix=True)
    salt = rom.String()
    hash = rom.String()
    created_at = rom.Float(default=time.time)

Create an instance of the model and save it:

PASSES = 32768
def gen_hash(password, salt=None):
    salt = salt or os.urandom(16)
    comp = salt + password
    out = sha256(comp).digest()
    for i in xrange(PASSES-1):
        out = sha256(out + comp).digest()
    return salt, out

user = User(email='user@host.com')
user.salt, user.hash = gen_hash(password)
user.save()
# session.commit() or session.flush() works too

Load and use the object later:

user = User.get_by(email='user@host.com')
at_gmail = User.query.endswith(email='@gmail.com').all()

Lua support¶

From version 0.25.0 and on, rom assumes that you are using Redis version 2.6 or later, which supports server-side Lua scripting. This allows for the support of multiple unique column constraints without annoying race conditions and retries. This also allows for the support of prefix, suffix, and pattern matching on certain column types.

If you are using a version of Redis prior to 2.6, you should upgrade Redis. If you are unable or unwilling to upgrade Redis, but you still wish to use rom, you should call rom._disable_lua_writes(), which will prevent you from using features that require Lua scripting support.

Expiring models/TTLs¶

There is a series of feature requests/bug reports/pull requests to add the ability for rom to automatically delete and/or expire entity data stored in Redis. This is a request that has been made (as of January 2016) 6 different times.

Long story short: rom stores a bunch of data in secondary structures to make querying fast. When a model “expires”, that data doesn’t get deleted. To delete that data, you have to run a cleanup function that literally has to scan over every entity in order to determine if the model had been expired. That is a huge waste and is the antithesis of good design.

Instead, if you create a new expire_at float column with index=True, the column can store when the entity is to expire. Then to expire the data, you can use: Model.query.filter(expire_at=(0, time.time())).limit(10) to (for example) get up to the 10 oldest entities that need to be expired.

Now, I know what you are thinking. You are thinking, “but I wish the data would just go away on its own.” And I don’t disagree. But for that to happen, Redis needs to grow Lua-script triggers, or you need to run a separate daemon to periodically clean up left-over data. But … if you need to run a separate daemon to clean up left-over data by scanning all of your rom entities, wouldn’t it just be better/faster in every way to keep an explicit column and do it efficiently? I think so, and you should too.

class rom.Boolean(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Used for boolean columns.

All standard arguments supported. See Column for details on supported arguments.

All values passed in on creation are casted via bool(), with the exception of None (which behaves as though the value was missing), and any existing data in Redis is considered False if empty, and True otherwise.

Used via:

class MyModel(Model):
    col = Boolean()

Queries via MyModel.get_by(...) and MyModel.query.filter(...) work as expected when passed True or False.

Note

these columns are not sortable by default.

rom.CASE_INSENSITIVE(val): Old alias for SIMPLE_CI

class rom.ClassProperty(get, set=None, delete=None)

Bases: object

Borrowed from: https://gist.github.com/josiahcarlson/1561563

class rom.Column(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: _Column

Column objects handle data conversion to/from strings, store metadata about indices, etc. Note that these are “heavy” columns, in that whenever data is read/written, it must go through descriptor processing. This is primarily so that (for example) if you try to write a Decimal to a Float column, you get an error the moment you try to do it, not some time later when you try to save the object (though saving can still cause an error during the conversion process).

Standard Arguments:

required - determines whether this column is required on creation

default - a default value (either a callable or a simple value) when this column is not provided

unique - can be enabled on string, unicode, and integer columns, and allows for required distinct column values (like an email address on a User model)

index - can be enabled on numeric, string, and unicode columns. Will create a ZSET-based numeric index for numeric columns and a “full word”-based search for string/unicode columns. If enabled for other (or custom) columns, remember to provide the keygen argument

keygen - pass a function that takes your column’s value and returns the data that you want to index (see the keygen docs for what kinds of data to return)

keygen2 - pass a function that takes your column name and the dict representing the current entity’s complete data - can be used for creating multi-column indexes

String/Text arguments:

prefix - can be enabled on any column that generates a list of strings as a result of the default or passed keygen function, and will allow the searching of prefix matches (autocomplete) over your data. See Query.startswith() for details.

suffix - can be enabled in the same contexts as prefix and enables suffix matching over your data. Any individual string in the returned data will be reversed (you need to make sure this makes conceptual sense with your data) before being stored or used. See Query.endswith() for details.

Warning

Enabling prefix or suffix matching on a column only makes sense for columns defining a non-numeric keygen function.

Notes:

If you have disabled Lua support, you can only have at most one unique column on each model

Unique and index are not mutually exclusive

The keygen argument determines how index values are generated from column data (with a reasonably sensible default for numeric columns, and 2 convenient options for string/text columns)

If you set required to True, then you must have the column set during object construction: MyModel(col=val)

If index and prefix, or index and suffix are set, the same keygen will be used for both the regular index as well as the prefix and/or suffix searches

If prefix is set, you can perform pattern matches over your data. See documentation on Query.like() for details.

Pattern matching over data is only guaranteed to be valid or correct for ANSI strings that do not include nulls, though we make an effort to support unicode strings and strings with embedded nulls

Prefix, suffix, and pattern matching are performed within a Lua script, so may have substantial execution time if there are a large number of matching prefix or suffix entries

Whenever possible, pattern matching will attempt to use any non-wildcard prefixes on the pattern to limit the items to be scanned. A pattern starting with ?, *, +, or ! will not be able to use any prefix, so will scan the entire index for matches (aka: expensive)

There are 3 types of string indexes that rom currently supports:

SIMPLE/SIMPLE_CI - sorting only with query.order_by('x') - https://josiahcarlson.github.io/rom/util.html#rom.util.SIMPLE

IDENTITY/IDENTITY_CI - equality only with query.filter(x=...) - https://josiahcarlson.github.io/rom/util.html#rom.util.IDENTITY

FULL_TEXT - bag of words inverted index for query.filter(x=...) - https://josiahcarlson.github.io/rom/util.html#rom.util.FULL_TEXT

To each of those 3 index types, you can further add a prefix/suffix index, whose semantics are as follows:

prefix - query.startswith(column=pfix) and query.like(column='stuff?*')

suffix - query.endswith(column=sfix)

SIMPLE/SIMPLE_CI/IDENTITY/IDENTITY_CI - prefix/suffix the whole string case sensitive or insensitive

FULL_TEXT - prefix/suffix on individual words parsed out of the full text

exception rom.ColumnError

Bases: ORMError

Raised when your column definitions are not correct

exception rom.DataRaceError

Bases: InvalidOperation

Raised when more than one writer tries to update the same columns on the same entity

class rom.Date(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A date column.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Date()

class rom.DateTime(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A datetime column.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = DateTime()

Note

tzinfo objects are not stored

class rom.Decimal(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A Decimal-only numeric column (converts ints/longs into Decimals automatically). Attempts to assign Python float will fail.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Decimal()

exception rom.EntityDeletedError

Bases: InvalidOperation

Raised when another writer deleted the entity from Redis; use .save(force=True) to re-save

rom.FULL_TEXT(val): This is a basic full-text index keygen function. Words are lowercased, split by whitespace, and stripped of punctuation from both ends before an inverted index is created for term searching.

class rom.Float(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Numeric column that supports integers and floats (values are turned into floats on load from Redis).

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Float()

class rom.ForeignModel(fmodel, required=False, default=<object object>)

Bases: Column

This column allows for rom models to reference an instance of another model from an unrelated ORM or otherwise.

Note

In order for this mechanism to work, the foreign model must have an id attribute or property represents its primary key, and must have a classmethod or staticmethod named get() that returns the proper database entity.

Used via:

class MyModel(Model):
    col = ForeignModel(DjangoModel)

dm = DjangoModel(col1='foo')
django.db.transaction.commit()

x = MyModel(col=dm)
x.save()

class rom.GeneralIndex(namespace)

Bases: object

This class implements general indexing and search for the rom package.

Warning

You probably don’t want to be calling this directly. Instead, you should rely on the Query object returned from Model.query to handle all of your query pre-processing.

Generally speaking, numeric indices use ZSETs, and text indices use SETs built using an ‘inverted index’.

Say that we have words hello world in a column c on a model with primary key MyModel:1. The member 1 will be added to SETs with keys:

MyModel:c:hello
MyModel:c:world

Text searching performs a sequence of intersections of SETs for the words to be searched for.

Numeric range searching performs a sequence of intersections of ZSETs, removing items outside the requested range after each intersection.

Searches will pre-sort intersections from smallest to largest SET/ZSET prior to performing the search to improve performance.

Prefix, suffix, and pattern matching change this operation. Given a key generated of hello on a column c on a model with primary key MyModel:1, the member hello\01 with score 0 will be added to a ZSET with the key name MyModel:c:pre for the prefix/pattern index. On a suffix index, the member olleh\01 with score 0 will be added to a ZSET with the key name MyModel:c:suf.

Prefix and suffix matches are excuted in Lua with a variant of the autocomplete method described in Redis in Action. These methods ensure a runtime proportional to the number of matched entries.

Pattern matching also uses a Lua script to scan over data in the prefix index, exploiting prefixes in patterns if they exist.

count(conn, filters)

Returns the count of the items that match the provided filters.

For the meaning of what the filters argument means, see the .search() method docs.

search(conn, filters, order_by, offset=None, count=None, timeout=None)

Search for model ids that match the provided filters.

Arguments:

filters - A list of filters that apply to the search of one of the following two forms:

'column:string' - a plain string will match a word in a text search on the column

Note

Read the documentation about the Query object for what is actually passed during text search

('column', min, max) - a numeric column range search, between min and max (inclusive by default)

Note

Read the documentation about the Query object for information about open-ended ranges

['column:string1', 'column:string2'] - will match any of the provided words in a text search on the column

Prefix('column', 'prefix') - will match prefixes of words in a text search on the column

Suffix('column', 'suffix') - will match suffixes of words in a text search on the column

Pattern('column', 'pattern') - will match patterns over words in a text search on the column

order_by - A string that names the numeric column by which to sort the results by. Prefixing with ‘-’ will return results in descending order

Note

While you can technically pass a non-numeric index as an order_by clause, the results will basically be to order the results by string comparison of the ids (10 will come before 2).

Note

If you omit the order_by argument, results will be ordered by the last filter. If the last filter was a text filter, see the previous note. If the last filter was numeric, then results will be ordered by that result.

offset - A numeric starting offset for results

count - The maximum number of results to return from the query

class rom.GeoIndex(name, callback)

Bases: tuple

callback: Alias for field number 1

name: Alias for field number 0

rom.IDENTITY(val)

This is a basic “equality” index keygen, primarily meant to be used for things like:

Model.query.filter(col='value')

Where FULL_TEXT would transform a sentence like “A Simple Sentence” into an inverted index searchable by the words “a”, “simple”, and/or “sentence”, IDENTITY will only be searchable by the original full sentence with the same capitalization - “A Simple Sentence”. See IDENTITY_CI for the same function, only case-insensitive.

rom.IDENTITY_CI(val): Case-insensitive version of IDENTITY

rom.IDENTITY_STRING(val): like IDENTITY, but for String columns

rom.IDENTITY_STRING_CI(val): like IDENTITY_CI, but for String columns

class rom.IndexOnly(column=None, keygen=None, prefix=False, suffix=False, keygen2=None, index=False, unique=False)

Bases: Column

This column doesn’t actually store data, except in indexes for other columns. Say, for example, you have an email text field that you want to be unique, look-up by identity, but also be parsed out for a sort of address-book index. Normally that would suck. But not with IndexOnly() columns! Data set on this attribute will raise an exception, and will be silently ignored (and potentially deleted) if already stored in Redis.

Used via:

import re
def split_email(val):
    if val:
        return set(filter(None, re.split('[^\w]', val.lower())))

class MyModel(Model):
    email = Text(unique=True)
    elookup = IndexOnly('email', keygen=split_email)

MyModel(email='user@host.com').save()
MyModel.get_by(elookup='user')
MyModel.query.filter(elookup=['user', 'host']).all()

Note

I’ve been using a variation of this internally for some of my own work, and I thought I’d release it as a convenience column.

class rom.Integer(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Used for integer numeric columns.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Integer()

exception rom.InvalidColumnValue

Bases: ColumnError

Raised when you attempt to pass a primary key on entity creation or when data assigned to a column is the wrong type

exception rom.InvalidOperation

Bases: ORMError

Raised when trying to delete or modify a column that cannot be deleted or modified

class rom.Json(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Allows for more complicated nested structures as attributes.

All standard arguments supported. The keygen argument must be provided if index is True.

Used via:

class MyModel(Model):
    col = Json()

class rom.ManyToOne(ftable, on_delete=<object object>, required=False, default=<object object>)

Bases: Column

This ManyToOne column allows for one model to reference another model. While a ManyToOne column does not require a reverse OneToMany column (which will return a list of models that reference it via a ManyToOne), it is generally seen as being useful to have both sides of the relationship defined.

Aside from the name of the other model, only the required and default arguments are accepted.

Four arguments are supported:

ftable - the name of the other model (required argument)

on_delete - how to handle foreign key references on delete,
supported options include: ‘no action’, ‘restrict’, ‘cascade’ ‘set default’, and ‘set null’ (required argument)

required - determines whether this column is required on creation

default - a default value (either a callable or a simple value) when this column is not provided

Used via:

class MyModel(Model):
    col = ManyToOne('OtherModelName')

Note

All ManyToOne columns are indexed numerically, which means that you can find entities referencing specific id ranges or even sort by referenced ids.

exception rom.MissingColumn

Bases: ColumnError

Raised when a model has a required column, but it is not provided on construction

class rom.Model(**kwargs)

Bases: object

This is the base class for all models. You subclass from this base Model in order to create a model with columns. As an example:

class User(Model):
    email_address = String(required=True, unique=True)
    salt = String(default='')
    hash = String(default='')
    created_at = Float(default=time.time, index=True)

Which can then be used like:

user = User(email_addrss='user@domain.com')
user.save() # session.commit() or session.flush() works too
user = User.get_by(email_address='user@domain.com')
user = User.get(5)
users = User.get([2, 6, 1, 7])

To perform arbitrary queries on entities involving the indices that you defined (by passing index=True on column creation), you access the .query class property on the model:

query = User.query
query = query.filter(created_at=(time.time()-86400, time.time()))
users = query.execute()

Note

You can perform single or chained queries against any/all columns that were defined with index=True.

Composite/multi-column unique constraints

As of version 0.28.0 and later, rom supports the ability for you to have a unique constraint involving multiple columns. Individual columns can be defined unique by passing the ‘unique=True’ specifier during column definition as always.

The attribute unique_together defines those groups of columns that when taken together must be unique for .save() to complete successfully. This will work almost exactly the same as Django’s unique_together, and is comparable to SQLAlchemy’s UniqueConstraint().

Usage:

class UniquePosition(Model):
    x = Integer()
    y = Integer()

    unique_together = [
        ('x', 'y'),
    ]

Note

If one or more of the column values on an entity that is part of a unique constraint is None in Python, the unique constraint won’t apply. This is the typical behavior of nulls in unique constraints inside both MySQL and Postgres.

copy(): Creates a shallow copy of the given entity (any entities that can be retrieved from a OneToMany relationship will not be copied).

delete(**kwargs): Deletes the entity immediately. Also performs any on_delete operations specified as part of column definitions.

classmethod does_not_endwith(attr, values, blocksize=100)

This iterates over all entities with an attribute that does not end with the provided values. This is only available to models with a suffix index on the given attribute; values must be normalized the same as with the indexed inputs for this to work properly (lowercased, etc.).

Args:

attr - name of the attribute/column on the entity.
values - list of values to exclude.

This method will auto-forget items from the session after yielding them, so if you want to change data, you’ll have to handle saving and deleting outside of the session.

..note: values <= 7 characters long will be fast, values >= 8 characters: will require round trips and will be substantially slower.

classmethod does_not_startwith(attr, values, blocksize=100)

This iterates over all entities with an attribute that does not start with the provided values. This is only available to models with a prefix index on the given attribute; values must be normalized the same as with the indexed inputs for this to work properly (lowercased, etc.).

Args:

attr - name of the attribute/column on the entity.
values - list of values to exclude.

This method will auto-forget items from the session after yielding them, so if you want to change data, you’ll have to handle saving and deleting outside of the session.

..note: values <= 7 characters long will be fast, values >= 8 characters: will require round trips and will be substantially slower.

classmethod get(ids)

Will fetch one or more entities of this type from the session or Redis.

Used like:

MyModel.get(5)
MyModel.get([1, 6, 2, 4])

Passing a list or a tuple will return multiple entities, in the same order that the ids were passed.

classmethod get_by(**kwargs)

This method offers a simple query method for fetching entities of this type via attribute numeric ranges (such columns must be indexed), or via unique columns.

Some examples:

user = User.get_by(email_address='user@domain.com')
# gets up to 25 users created in the last 24 hours
users = User.get_by(
    created_at=(time.time()-86400, time.time()),
    _limit=(0, 25))

Optional keyword-only arguments:

_limit - A 2-tuple of (offset, count) that can be used to paginate or otherwise limit results returned by a numeric range query

_numeric - An optional boolean defaulting to False that forces the use of a numeric index for .get_by(col=val) queries even when col has an existing unique index

If you would like to make queries against multiple columns or with multiple criteria, look into the Model.query class property.

Note

rom will attempt to use a unique index first, then a numeric index if there was no unique index. You can explicitly tell rom to only use the numeric index by using .get_by(..., _numeric=True).

Note

Ranged queries with get_by(col=(start, end)) will only work with columns that use a numeric index.

save(full=False, force=False)

Saves the current entity to Redis. Will only save changed data by default, but you can force a full save by passing full=True.

If the underlying entity was deleted and you want to re-save the entity, you can pass force=True to force a full re-save of the entity.

to_dict(): Returns a copy of all data assigned to columns in this entity. Useful for returning items to JSON-enabled APIs. If you want to copy an entity, you should look at the .copy() method.

transfer(other, attr, value, txn_model, txn_attr, decimal_places=0, refresh_entities=True, refresh_index=True)

Transfer some numeric value from one entity to another.

This can (for example) be used to transfer money as part of an in-game transaction, or other sort of value transfer.

other - the other entity you would like to participate in this transaction (must both have the same db connection)

attr - the name of the attribute to transfer value on

value - the value to transfer (rounded to decimal_places)

txn_model - the entity that represents the value transfer to perform (must have the same db connection as self, and other)

txn_attr - the attribute on the entity that represents if the value has been transferred

decimal_places - the number of decimal places to the right of the decimal to round to inside Redis / Lua; note that for values >0, this will introduce binary/decimal rounding problems; so small epsilon credit may go away, and you will want to explicitly round on the client on read + display. Or better yet; stick to integers.

refresh_entities - will refresh the entity data on transfer if True-ish

refresh_index - will refresh the update any relevant indexes after the transfer, if True-ish; implies refresh_entities

..warning: This doesn’t magically create more bits for you. Values in: Redis are either stored as up-to 64 bit integers (0 decimal places) or 64 bit doubles with 53 bits of precision. For doubles, that means 15-16 decimal digits. For 64 bit integers, that is 19+ digits, but only integers. So if you want to maximize both precision, and your range of values for “gold”, “points”, “experience”, “value”, etc.; use your smallest denomination as your 1, and divmod on the client for display if you need to.

update(*args, **kwargs)

Updates multiple attributes in a model. If args are provided, this method will assign attributes in the order returned by list(self._columns) until one or both are exhausted.

If kwargs are provided, this method will assign attributes to the names provided, after args have been processed.

exception rom.ORMError

Bases: Exception

Base class for all ORM-related errors

class rom.OneToMany(ftable, column=None)

Bases: Column

OneToMany columns do not actually store any information. They rely on a properly defined reverse ManyToOne column on the referenced model in order to be able to fetch a list of referring entities.

Two arguments are supported:

ftable - the name of the other model

column - the attribute on the other model with the reference to
this column, required if the foreign model has multiple columns referencing this model with OneToOne or ManyToOne columns

Used via:

class MyModel(Model):
    col = OneToMany('OtherModelName')
    ocol = OneToMany('ModelName')

class rom.OneToOne(ftable, on_delete=<object object>, required=False, default=<object object>, unique=False)

Bases: ManyToOne

This OneToOne column allows for one model to reference another model. A OneToOne column does not require a reverse OneToOne column, and provides on_delete behavior.

Five arguments are supported:

ftable - the name of the other model (required argument)

on_delete - how to handle foreign key references on delete,
supported options include: ‘no action’, ‘restrict’, ‘cascade’ ‘set default’, and ‘set null’ (required argument)

required - determines whether this column is required on creation

default - a default value (either a callable or a simple value) when this column is not provided

unique - whether or not the referenced entity must be a unique
reference

Used via:

class MyModel(Model):
    col = OneToOne('OtherModelName', 'no action')

Note

All OneToOne columns are indexed numerically, which means that you can find entities referencing specific id ranges or even sort by referenced ids.

class rom.Pattern(attr, pattern)

Bases: tuple

attr: Alias for field number 0

pattern: Alias for field number 1

class rom.Prefix(attr, prefix)

Bases: tuple

attr: Alias for field number 0

prefix: Alias for field number 1

class rom.PrimaryKey(index=False)

Bases: Column

This is a primary key column, used when you want the primary key to be named something other than ‘id’. If you omit a PrimaryKey column on your Model classes, one will be automatically created for you.

Only the index argument will be used. You may want to enable indexing on this column if you want to be able to perform queries and sort the results by primary key.

Used via:

class MyModel(Model):
    id = PrimaryKey()

class rom.Query(model, filters=(), order_by=None, limit=None, select=None)

Bases: object

This is a query object. It behaves a lot like other query objects. Every operation performed on Query objects returns a new Query object. The old Query object does not have any updated filters.

all(): Alias for execute().

cached_result(timeout)

This will execute the query, returning the key where a ZSET of your results will be stored for pagination, further operations, etc.

The timeout must be a positive integer number of seconds for which to set the expiration time on the key (this is to ensure that any cached query results are eventually deleted, unless you make the explicit step to use the PERSIST command).

Note

Limit clauses are ignored and not passed.

Usage:

ukey = User.query.endswith(email='@gmail.com').cached_result(30)
for i in xrange(0, conn.zcard(ukey), 100):
    # refresh the expiration
    conn.expire(ukey, 30)
    users = User.get(conn.zrange(ukey, i, i+99))
    ...

count()

Will return the total count of the objects that match the specified filters.:

# counts the number of users created in the last 24 hours
User.query.filter(created_at=(time.time()-86400, time.time())).count()

delete(blocksize=100)

Will delete the entities that match at the time the query is executed.

Used like:

MyModel.query.filter(email=...).delete()
MyModel.query.endswith(email='@host.com').delete()

Warning

can’t be used on models on either side of a OneToMany, ManyToOne, or OneToOne relationship.

endswith(**kwargs)

When provided with keyword arguments of the form col=suffix, this will limit the entities returned to those that have a word with the provided suffix in the specified column(s). This requires that the suffix=True option was provided during column definition.

Usage:

User.query.endswith(email='@gmail.com').execute()

execute(): Actually executes the query, returning any entities that match the filters, ordered by the specified ordering (if any), limited by any earlier limit calls.

filter(**kwargs)

Only columns/attributes that have been specified as having an index with the index=True option on the column definition can be filtered with this method. Prefix, suffix, and pattern match filters must be provided using the .startswith(), .endswith(), and the .like() methods on the query object, respectively. Geo location queries should be performed using the .near() method.

Filters should be of the form:

# for numeric ranges, use None for open-ended ranges
attribute=(min, max)

# you can also query for equality by passing a single number
attribute=value

# for string searches, passing a plain string will require that
# string to be in the index as a literal
attribute=string

# to perform an 'or' query on strings, you can pass a list of
# strings
attribute=[string1, string2]

As an example, the following will return entities that have both hello and world in the String column scol and has a Numeric column ncol with value between 2 and 10 (including the endpoints):

results = MyModel.query \
    .filter(scol='hello') \
    .filter(scol='world') \
    .filter(ncol=(2, 10)) \
    .all()

If you only want to match a single value as part of your range query, you can pass an integer, float, or Decimal object by itself, similar to the Model.get_by() method:

results = MyModel.query \
    .filter(ncol=5) \
    .execute()

Note

Trying to use a range query attribute=(min, max) on indexed string columns won’t return any results.

Note

This method only filters columns that have been defined with index=True.

first(): Returns only the first result from the query, if any.

iter_result(timeout=30, pagesize=100, no_hscan=False)

Iterate over the results of your query instead of getting them all with .all(). Will only perform a single query. If you expect that your processing will take more than 30 seconds to process 100 items, you should pass timeout and pagesize to reflect an appropriate timeout and page size to fetch at once.

Usage:

for user in User.query.endswith(email='@gmail.com').iter_result():
    # do something with user
    ...

like(**kwargs)

When provided with keyword arguments of the form col=pattern, this will limit the entities returned to those that include the provided pattern. Note that ‘like’ queries require that the prefix=True option must have been provided as part of the column definition.

Patterns allow for 4 wildcard characters, whose semantics are as follows:

? - will match 0 or 1 of any character

* - will match 0 or more of any character

+ - will match 1 or more of any character

! - will match exactly 1 of any character

As an example, imagine that you have enabled the required prefix matching on your User.email column. And lets say that you want to find everyone with an email address that contains the name ‘frank’ before the @ sign. You can use either of the following patterns to discover those users.

*frank*@

*frank*@

Note

Like queries implicitly start at the beginning of strings checked, so if you want to match a pattern that doesn’t start at the beginning of a string, you should prefix it with one of the wildcard characters (like * as we did with the ‘frank’ pattern).

limit(offset, count)

Will limit the number of results returned from a query:

# returns the most recent 25 users
User.query.order_by('-created_at').limit(0, 25).execute()

order_by(column)

When provided with a column name, will sort the results of your query:

# returns all users, ordered by the created_at column in
# descending order
User.query.order_by('-created_at').execute()

replace(**kwargs): Copy the Query object, optionally replacing the filters, order_by, or limit information on the copy. This is mostly an internal detail that you can ignore.

select(*column_names, **kwargs)

Select the provided column names from the model, do not return an entity, do not involve the rom session, just get the raw and/or processed column data from Redis.

Keyword-only arguments:

include_pk=False - whether to include the primary key in the
returned data (we need to get this in some cases, so we fetch it anyway; if you want it, we can return it to you - just be careful with the namedtuple option - see the warning below)

decode=True - whether to take a pass through normal data
decoding in the model (will not return an entity/model)

ff=_dict_data_factory - the type of data to return from the
select after all filters/limits/order_by are applied

Warning

If include_pk = True and if you don’t provide the primary key column, it will be appended to your list of columns.

Note

if you want to provide a new factory function for the returned data, it must be of the form (below is the actual dict factory function)

def _dict_data_factory(columns):
    _dict = dict
    _zip = zip
    def make(data):
        # do whatever you need to turn your tuple of columns plus
        # your list of data into whatever you want:
        return _dict(_zip(columns, data))
    return make

Available factory functions:

``rom.query._dict_data_factory`` - default

``rom.query._list_data_factory`` - lowest overhead, as the data passed in above is a list that you can do anything to

``rom.query._tuple_data_factory`` - when you want tuples instead

``rom.query._namedtuple_data_factory`` - get namedtuples, see see warning below

Warning

If you use the _namedtuple_data_factory, and your columns include underscore prefixes, they will be stripped. If this results in a name collision, you will get an exception. If you want different behavior, write your own 20 line factory function that does exactly what you want, and pass it; they are really easy!

startswith(**kwargs)

When provided with keyword arguments of the form col=prefix, this will limit the entities returned to those that have a word with the provided prefix in the specified column(s). This requires that the prefix=True option was provided during column definition.

Usage:

User.query.startswith(email='user@').execute()

exception rom.QueryError

Bases: InvalidOperation

Raised when arguments to Model.get_by() or Query.filter are not valid

exception rom.RestrictError

Bases: InvalidOperation

Raised when deleting an object referenced by other objects

rom.SIMPLE(val): This is a basic case-sensitive “sorted order” index keygen function for strings. This will return a value that is suitable to be used for ordering by a 7-byte prefix of a string (that is 7 characters from a byte-string, and 1.75-7 characters from a unicode string, depending on character -> encoding length).

Warning

Case sensitivity is based on the (encoded) byte prefixes of the strings/text being indexed, so ordering may be different than a native comparison ordering (especially if an order is different based on characters past the 7th encoded byte).

rom.SIMPLE_CI(val): The same as SIMPLE, only case-insensitive.

class rom.SaferDateTime(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: DateTime

A (safer) datetime column (see Issue #109 or the below note for more information)

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = SaferDateTime()

Note

tzinfo objects are not stored

Note

what makes this “safer” than other datetime objects is that there are exactly two types of values that can be set here: a datetime object, or a str that can represent a float, which is the number of seconds since the unix epoch.

class rom.String(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A plain string column (str in 2.x, bytes in 3.x). Trying to save unicode strings will probably result in an error, if not corrupted data.

All standard arguments and String/Text arguments supported. See Column for details on supported arguments.

This column can be indexed in one of five ways - a sorted index on a 7 byte prefix of the value (keygen=rom.SIMPLE), a sorted index on a lowercased 7 byte prefix of the value (keygen=rom.SIMPLE_CI), a case-insensitive full-text index (keygen=rom.FULL_TEXT), a case-sensitive identity index (keygen=rom.IDENTITY), and a case-insensitive identity index (keygen=rom.IDENTITY_CI).

Used via:

class MyModel(Model):
    col = String()

class rom.Suffix(attr, suffix)

Bases: tuple

attr: Alias for field number 0

suffix: Alias for field number 1

class rom.Text(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A unicode string column. Behavior is more or less identical to the String column type, except that unicode is supported (unicode in 2.x, str in 3.x). UTF-8 is used by default as the encoding to bytes on the wire, which will affect rom.SIMPLE and rom.SIMPLE_CI indexes.

All standard arguments supported. See Column for details on supported arguments.

This column can be indexed in one of five ways - a sorted index on a 7 byte prefix of the value (keygen=rom.SIMPLE), a sorted index on a lowercased 7 byte prefix of the value (keygen=rom.SIMPLE_CI), a case-insensitive full-text index (keygen=rom.FULL_TEXT), a case-sensitive identity index (keygen=rom.IDENTITY), and a case-insensitive identity index (keygen=rom.IDENTITY_CI).

For the 7 byte prefix/suffixes on indexes using the rom.SIMPLE and rom.SIMPLE_CI keygen, because we use UTF-8 to encode text, a single character can turn into 1-3 bytes, so may not be useful in practice.

Used via:

class MyModel(Model):
    col = Text()

class rom.Time(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A time column. Timezones are ignored.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Time()

Note

tzinfo objects are not stored

exception rom.UniqueKeyViolation

Bases: ORMError

Raised when trying to save an entity without a distinct column value

class rom.UnsafeColumn

Bases: _Column

UnsafeColumns are attributes on models that allow for explicit model-related data to be stored in native Redis structures. Data stored in these columns are not synchronized with the underlying race-condition-prevention locking mechanisms used during Model.save() and Model.delete(). Though this data is deleted during Model.delete().

As such, there may be race conditions if you delete an entity from one process, but work on unsafe columns in another. To clean up leftover unwanted attributes, use util.clean_unsafe_cols(*models) to clean the namespace of given models.

By default, UnsafeColumn data is stored: '%s:%s:%s' % (ent._namespace, ent.<primary key>, column_name)

Which is separate from where most normal column data is stored, and can be accessed directly if wanted. Hence why it is “unsafe”.

rom.columns rom.exceptions rom.index rom.model rom.query rom.util

rom Package¶

Sponsorships Available¶

Documentation¶

What¶

Why¶

What is available¶

Getting started¶

Lua support¶

Expiring models/TTLs¶

Table of Contents

Previous topic

Next topic

This Page