columns Module

Rom - the Redis object mapper for Python

Copyright 2013-2016 Josiah Carlson

Released under the LGPL license version 2.1 and version 3 (you can choose which you’d like to be bound under).

class rom.columns.Boolean(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Used for boolean columns.

All standard arguments supported. See Column for details on supported arguments.

All values passed in on creation are casted via bool(), with the exception of None (which behaves as though the value was missing), and any existing data in Redis is considered False if empty, and True otherwise.

Used via:

class MyModel(Model):
    col = Boolean()

Queries via MyModel.get_by(...) and MyModel.query.filter(...) work as expected when passed True or False.

Note

these columns are not sortable by default.

class rom.columns.Column(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: _Column

Column objects handle data conversion to/from strings, store metadata about indices, etc. Note that these are “heavy” columns, in that whenever data is read/written, it must go through descriptor processing. This is primarily so that (for example) if you try to write a Decimal to a Float column, you get an error the moment you try to do it, not some time later when you try to save the object (though saving can still cause an error during the conversion process).

Standard Arguments:

  • required - determines whether this column is required on creation

  • default - a default value (either a callable or a simple value) when this column is not provided

  • unique - can be enabled on string, unicode, and integer columns, and allows for required distinct column values (like an email address on a User model)

  • index - can be enabled on numeric, string, and unicode columns. Will create a ZSET-based numeric index for numeric columns and a “full word”-based search for string/unicode columns. If enabled for other (or custom) columns, remember to provide the keygen argument

  • keygen - pass a function that takes your column’s value and returns the data that you want to index (see the keygen docs for what kinds of data to return)

  • keygen2 - pass a function that takes your column name and the dict representing the current entity’s complete data - can be used for creating multi-column indexes

String/Text arguments:

  • prefix - can be enabled on any column that generates a list of strings as a result of the default or passed keygen function, and will allow the searching of prefix matches (autocomplete) over your data. See Query.startswith() for details.

  • suffix - can be enabled in the same contexts as prefix and enables suffix matching over your data. Any individual string in the returned data will be reversed (you need to make sure this makes conceptual sense with your data) before being stored or used. See Query.endswith() for details.

Warning

Enabling prefix or suffix matching on a column only makes sense for columns defining a non-numeric keygen function.

Notes:

  • If you have disabled Lua support, you can only have at most one unique column on each model

  • Unique and index are not mutually exclusive

  • The keygen argument determines how index values are generated from column data (with a reasonably sensible default for numeric columns, and 2 convenient options for string/text columns)

  • If you set required to True, then you must have the column set during object construction: MyModel(col=val)

  • If index and prefix, or index and suffix are set, the same keygen will be used for both the regular index as well as the prefix and/or suffix searches

  • If prefix is set, you can perform pattern matches over your data. See documentation on Query.like() for details.

  • Pattern matching over data is only guaranteed to be valid or correct for ANSI strings that do not include nulls, though we make an effort to support unicode strings and strings with embedded nulls

  • Prefix, suffix, and pattern matching are performed within a Lua script, so may have substantial execution time if there are a large number of matching prefix or suffix entries

  • Whenever possible, pattern matching will attempt to use any non-wildcard prefixes on the pattern to limit the items to be scanned. A pattern starting with ?, *, +, or ! will not be able to use any prefix, so will scan the entire index for matches (aka: expensive)

There are 3 types of string indexes that rom currently supports:

To each of those 3 index types, you can further add a prefix/suffix index, whose semantics are as follows:

  • prefix - query.startswith(column=pfix) and query.like(column='stuff?*')

  • suffix - query.endswith(column=sfix)

  • SIMPLE/SIMPLE_CI/IDENTITY/IDENTITY_CI - prefix/suffix the whole string case sensitive or insensitive

  • FULL_TEXT - prefix/suffix on individual words parsed out of the full text

class rom.columns.Date(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A date column.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Date()
class rom.columns.DateTime(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A datetime column.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = DateTime()

Note

tzinfo objects are not stored

class rom.columns.Decimal(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A Decimal-only numeric column (converts ints/longs into Decimals automatically). Attempts to assign Python float will fail.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Decimal()
class rom.columns.Float(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Numeric column that supports integers and floats (values are turned into floats on load from Redis).

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Float()
class rom.columns.ForeignModel(fmodel, required=False, default=<object object>)

Bases: Column

This column allows for rom models to reference an instance of another model from an unrelated ORM or otherwise.

Note

In order for this mechanism to work, the foreign model must have an id attribute or property represents its primary key, and must have a classmethod or staticmethod named get() that returns the proper database entity.

Used via:

class MyModel(Model):
    col = ForeignModel(DjangoModel)

dm = DjangoModel(col1='foo')
django.db.transaction.commit()

x = MyModel(col=dm)
x.save()
class rom.columns.IndexOnly(column=None, keygen=None, prefix=False, suffix=False, keygen2=None, index=False, unique=False)

Bases: Column

This column doesn’t actually store data, except in indexes for other columns. Say, for example, you have an email text field that you want to be unique, look-up by identity, but also be parsed out for a sort of address-book index. Normally that would suck. But not with IndexOnly() columns! Data set on this attribute will raise an exception, and will be silently ignored (and potentially deleted) if already stored in Redis.

Used via:

import re
def split_email(val):
    if val:
        return set(filter(None, re.split('[^\w]', val.lower())))

class MyModel(Model):
    email = Text(unique=True)
    elookup = IndexOnly('email', keygen=split_email)

MyModel(email='user@host.com').save()
MyModel.get_by(elookup='user')
MyModel.query.filter(elookup=['user', 'host']).all()

Note

I’ve been using a variation of this internally for some of my own work, and I thought I’d release it as a convenience column.

class rom.columns.Integer(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Used for integer numeric columns.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Integer()
class rom.columns.Json(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

Allows for more complicated nested structures as attributes.

All standard arguments supported. The keygen argument must be provided if index is True.

Used via:

class MyModel(Model):
    col = Json()
class rom.columns.ManyToOne(ftable, on_delete=<object object>, required=False, default=<object object>)

Bases: Column

This ManyToOne column allows for one model to reference another model. While a ManyToOne column does not require a reverse OneToMany column (which will return a list of models that reference it via a ManyToOne), it is generally seen as being useful to have both sides of the relationship defined.

Aside from the name of the other model, only the required and default arguments are accepted.

Four arguments are supported:

  • ftable - the name of the other model (required argument)

  • on_delete - how to handle foreign key references on delete,

    supported options include: ‘no action’, ‘restrict’, ‘cascade’ ‘set default’, and ‘set null’ (required argument)

  • required - determines whether this column is required on creation

  • default - a default value (either a callable or a simple value) when this column is not provided

Used via:

class MyModel(Model):
    col = ManyToOne('OtherModelName')

Note

All ManyToOne columns are indexed numerically, which means that you can find entities referencing specific id ranges or even sort by referenced ids.

class rom.columns.OneToMany(ftable, column=None)

Bases: Column

OneToMany columns do not actually store any information. They rely on a properly defined reverse ManyToOne column on the referenced model in order to be able to fetch a list of referring entities.

Two arguments are supported:

  • ftable - the name of the other model

  • column - the attribute on the other model with the reference to

    this column, required if the foreign model has multiple columns referencing this model with OneToOne or ManyToOne columns

Used via:

class MyModel(Model):
    col = OneToMany('OtherModelName')
    ocol = OneToMany('ModelName')
class rom.columns.OneToOne(ftable, on_delete=<object object>, required=False, default=<object object>, unique=False)

Bases: ManyToOne

This OneToOne column allows for one model to reference another model. A OneToOne column does not require a reverse OneToOne column, and provides on_delete behavior.

Five arguments are supported:

  • ftable - the name of the other model (required argument)

  • on_delete - how to handle foreign key references on delete,

    supported options include: ‘no action’, ‘restrict’, ‘cascade’ ‘set default’, and ‘set null’ (required argument)

  • required - determines whether this column is required on creation

  • default - a default value (either a callable or a simple value) when this column is not provided

  • unique - whether or not the referenced entity must be a unique

    reference

Used via:

class MyModel(Model):
    col = OneToOne('OtherModelName', 'no action')

Note

All OneToOne columns are indexed numerically, which means that you can find entities referencing specific id ranges or even sort by referenced ids.

class rom.columns.PrimaryKey(index=False)

Bases: Column

This is a primary key column, used when you want the primary key to be named something other than ‘id’. If you omit a PrimaryKey column on your Model classes, one will be automatically created for you.

Only the index argument will be used. You may want to enable indexing on this column if you want to be able to perform queries and sort the results by primary key.

Used via:

class MyModel(Model):
    id = PrimaryKey()
class rom.columns.SaferDateTime(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: DateTime

A (safer) datetime column (see Issue #109 or the below note for more information)

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = SaferDateTime()

Note

tzinfo objects are not stored

Note

what makes this “safer” than other datetime objects is that there are exactly two types of values that can be set here: a datetime object, or a str that can represent a float, which is the number of seconds since the unix epoch.

class rom.columns.String(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A plain string column (str in 2.x, bytes in 3.x). Trying to save unicode strings will probably result in an error, if not corrupted data.

All standard arguments and String/Text arguments supported. See Column for details on supported arguments.

This column can be indexed in one of five ways - a sorted index on a 7 byte prefix of the value (keygen=rom.SIMPLE), a sorted index on a lowercased 7 byte prefix of the value (keygen=rom.SIMPLE_CI), a case-insensitive full-text index (keygen=rom.FULL_TEXT), a case-sensitive identity index (keygen=rom.IDENTITY), and a case-insensitive identity index (keygen=rom.IDENTITY_CI).

Used via:

class MyModel(Model):
    col = String()
class rom.columns.Text(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A unicode string column. Behavior is more or less identical to the String column type, except that unicode is supported (unicode in 2.x, str in 3.x). UTF-8 is used by default as the encoding to bytes on the wire, which will affect rom.SIMPLE and rom.SIMPLE_CI indexes.

All standard arguments supported. See Column for details on supported arguments.

This column can be indexed in one of five ways - a sorted index on a 7 byte prefix of the value (keygen=rom.SIMPLE), a sorted index on a lowercased 7 byte prefix of the value (keygen=rom.SIMPLE_CI), a case-insensitive full-text index (keygen=rom.FULL_TEXT), a case-sensitive identity index (keygen=rom.IDENTITY), and a case-insensitive identity index (keygen=rom.IDENTITY_CI).

For the 7 byte prefix/suffixes on indexes using the rom.SIMPLE and rom.SIMPLE_CI keygen, because we use UTF-8 to encode text, a single character can turn into 1-3 bytes, so may not be useful in practice.

Used via:

class MyModel(Model):
    col = Text()
class rom.columns.Time(required=False, default=<object object>, unique=False, index=False, keygen=None, prefix=False, suffix=False, keygen2=None)

Bases: Column

A time column. Timezones are ignored.

All standard arguments supported. See Column for details on supported arguments.

Used via:

class MyModel(Model):
    col = Time()

Note

tzinfo objects are not stored

class rom.columns.UnsafeColumn

Bases: _Column

UnsafeColumns are attributes on models that allow for explicit model-related data to be stored in native Redis structures. Data stored in these columns are not synchronized with the underlying race-condition-prevention locking mechanisms used during Model.save() and Model.delete(). Though this data is deleted during Model.delete().

As such, there may be race conditions if you delete an entity from one process, but work on unsafe columns in another. To clean up leftover unwanted attributes, use util.clean_unsafe_cols(*models) to clean the namespace of given models.

By default, UnsafeColumn data is stored: '%s:%s:%s' % (ent._namespace, ent.<primary key>, column_name)

Which is separate from where most normal column data is stored, and can be accessed directly if wanted. Hence why it is “unsafe”.