Jason Madden

Transaction IDs in ZODB and RelStorage

Jason Madden — Mon, 03 May 2021 11:31:09 GMT

Earlier, we introduced ZODB as a transactional object database. This post takes a look into an implementation detail: transaction IDs. What are they, what are they used for, and why does any of this matter?

Along the way, we'll also look at another type of identifier used in ZODB, object IDs.

Contents

Rather than try to talk about transactions in general, I'll use ZODB-colored lenses and only talk about transactions and transaction IDs as they relate to ZODB and how ZODB uses them.

Implementing ZODB: Motivation for Transaction IDs

To understand how ZODB uses transaction IDs, let's work through an example of a basic use of ZODB, pausing along the way to consider how ZODB is actually implementing things.

We know that ZODB stores persistent objects; we can ask ZODB to fetch a persistent object for us (perhaps much later, or perhaps from a different machine) and it will come back to us in the state we left it. The easiest way to do this is by using normal Python mechanisms to build and walk through an object graph, starting from the ZODB root object.

>>> from ZODB.DB import DB
>>> from ZODB.DemoStorage import DemoStorage
>>> from persistent import Persistent
>>> import transaction
>>> db = DB(DemoStorage())
>>> transaction.begin()
<transaction._transaction.Transaction object at 0x110604c80>
>>> conn = db.open()
>>> conn.root()['obj'] = Persistent()
>>> transaction.commit()
None

We can use a different connection (simulating a different machine, say) and walk through the object graph, starting from the root, to find our object.

>>> transaction.begin()
<transaction._transaction.Transaction object at 0x110a3ff00>
>>> conn2 = db.open()
>>> conn2.root()['obj']
<persistent.Persistent object at 0x110a973d0 oid 0x3e2ee68c1ab9f8d3 in <Connection at 110a3fa00>>

Object IDs

Everything starts from the root. But where does the root object come from? Let's look at the root() method of the Connection:

def root(self):
    """Return the database root object."""
    return RootConvenience(self.get(z64))

That points us to the get() method:

def get(self, oid):
    """Return the persistent object with oid 'oid'."""
    # Simplified,  details removed
    obj = self._cache.get(oid, None)
    if obj is not None:
        return obj
    pickle_data, _ = self._storage.load(oid)
    obj = self._reader.getGhost(pickle_data)
    self._cache.new_ghost(oid, obj)
    return obj

There's a lot going on in these two little snippets, so lets unpack them, starting with get(). What can we deduce from this?

A Connection caches objects, and when objects can't be found in the cache, it asks the "storage" object to load it. (More on the storage in a minute.)
Persistent objects are identified by "OIDs" (short for "object identifier", or just "object ID"). Just as any Python object in memory has a unique identifier—returned by id(obj) and commonly seen in repr output as a hexadecimal number, as in <persistent.Persistent object at 0x110a973d0...>—, any persistent object stored in the ZODB has a unique identifier, its OID. An important difference is that id(obj) is only meaningful within the scope of a single process, but the ZODB OID is usable from any process connected to the same ZODB.
You can think of the OID as (part of) the "address" of a particular persistent object in the same way that id(obj) returns the memory address of an object in CPython. The OID is used to find or store objects in the cache, and also to load the stored data for the object.
The root object has a special, pre-defined OID known as "z64". That's our starting point for traversing the object graph.

You can't see this just from the code, but ZODB defines OIDs to be 8 byte, or 64-bit, quantities. The special value z64 is just 8 zero bytes in a row. ZODB has some utilities to convert between the 8 byte form or a 64-bit number.

>>> from ZODB.Connection import z64
>>> z64
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> conn.root()._p_oid
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> from ZODB.utils import p64, u64
>>> u64(z64)
0
>>> p64(0)
b'\x00\x00\x00\x00\x00\x00\x00\x00'

Each persistent object knows its own OID and keeps track of it in a special attribute known as _p_oid.

>>> conn.get(z64)._p_oid
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> conn.root()['obj']._p_oid
b'>.\xe6\x8c\x1a\xb9\xf8\xd3'
>>> u64(conn.root()['obj']._p_oid)
4480772168698427603

Storage Details and Referencing Objects

The get() method of a ZODB Connection returns persistent objects, but it uses the lower level "storage" interface to get the actual data stored for the object. Each ZODB Connection has a storage object, and all the Connections of a particular ZODB DB have storage objects that are reading and writing to the same shared data (e.g., a single file for FileStorage, or a single SQL database for RelStorage.) Among other things, the storage is responsible for holding the data stored for the persistent objects. This data, or state, is kept as a byte-string produced by pickle. We can use pickletools to peek at this.

>>> pickletools.dis(conn._storage.load(z64)[0])
    0: \x80 PROTO      3
    2: c    GLOBAL     'persistent.mapping PersistentMapping'
   40: q    BINPUT     0
   42: .    STOP
highest protocol among opcodes = 2
None
>>> pickletools.dis(conn._storage.load(z64)[0][43:])
    0: \x80 PROTO      3
    2: }    EMPTY_DICT
    3: q    BINPUT     1
    5: X    BINUNICODE 'data'
   14: q    BINPUT     2
   16: }    EMPTY_DICT
   17: q    BINPUT     3
   19: X    BINUNICODE 'obj'
   27: q    BINPUT     4
   29: C    SHORT_BINBYTES b'>.\xe6\x8c\x1a\xb9\xf8\xd3'
   39: q    BINPUT     5
   41: c    GLOBAL     'persistent Persistent'
   64: q    BINPUT     6
   66: \x86 TUPLE2
   67: q    BINPUT     7
   69: Q    BINPERSID
   70: s    SETITEM
   71: s    SETITEM
   72: .    STOP
   highest protocol among opcodes = 3

The important part here is the line labeled 29. Notice how that byte string matches the OID of our second persistent object. We can see that our root persistent object is referring to our other persistent object via its OID. The opcode labeled 69, BINPERSID is where ZODB hooks in to the unpickling process. By setting an attribute on the unpickler called "persistent_load", ZODB arranges to be called any time one of these OIDs is located in a pickle. ZODB can then go back to the cache or the storage to find the matching object.

Conflict Detection Through Serial Numbers

We've seen how ZODB can address objects, finding them and loading them by OID, automatically re-creating in memory the object graph we built. But that's not all ZODB does. ZODB provides the ability to modify and save objects, and to detect when more than one Connection attempts to do so at the same time, resulting in a conflict. How does this work?

To explore this, lets have both connections make a change to their version of the root object. We'll be using separate transaction managers, one for each connection, in order to do this.

>>> from transaction import TransactionManager
>>> conn1_txm = TransactionManager()
>>> conn2_txm = TransactionManager()
>>> conn.close()
None
>>> conn2.close()
None
>>> conn = db.open(conn1_txm)
>>> conn2 = db.open(conn2_txm)
>>> conn.root()['obj2'] = Persistent()
>>> conn2.root()['obj3'] = Persistent()

If we commit the changes from one transaction, everything goes as planned:

>>> conn1_txm.commit()
None

But trying to commit the other transaction fails because the object has already been modified in an incompatible way:

>>> conn2_txm.commit()
Traceback (most recent call last):
...
   raise ConflictError(oid=oid, serials=(committedSerial, oldSerial),
ZODB.POSException.ConflictError: database conflict error
    (oid 0x00,
    class persistent.mapping.PersistentMapping,
    serial this txn started with 0x03dfd117d4dbf099 2021-05-03 16:23:49.888861,
    serial currently committed 0x03dfd123995e91dd 2021-05-03 16:35:35.945956)

How did ZODB know the object had changed? In addition to keeping track of its OID, each persistent object also keeps track of its serial number in the _p_serial special attribute. When one connection tries to make a change to an object, the serial number of the object being changed is compared with the serial number ZODB has stored for the object, and if they're not equal, there's a conflict. (The Connection.readCurrent() API does the same thing for objects that haven't been modified.)

We can confirm this by matching the _p_serial of each connection's object with the error message reported above.

>>> conn2.root()._p_serial
b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x99'
>>> hex(u64(conn2.root()._p_serial))
'0x3dfd117d4dbf099'
>>> conn.root()._p_serial
b'\x03\xdf\xd1#\x99^\x91\xdd'
>>> hex(u64(conn.root()._p_serial))
'0x3dfd123995e91dd'

That serial number is the transaction ID. This is sometimes abbreviated to TID.

Thus, to uniquely identify a particular revision of a particular object, we need both its OID and TID. (Together, they form the full "address" of the object.) The storage object has methods to load object states as they existed at particular serial numbers:

>>> db.storage.loadSerial(z64, conn2.root()._p_serial)
(b'\x80\x03cpersistent.mapping\nPersistentMapping\nq\x00.\x80\x03}q\x01'
 b'X\x04\x00\x00\x00dataq\x02}q\x03X\x03\x00\x00\x00objq\x04C\x08>.'
 b'\xe6\x8c\x1a\xb9\xf8\xd3q\x05cpersistent\nPersistent\nq\x06\x86q\x07Qss.')
>>> db.storage.loadSerial(z64, conn.root()._p_serial)
(b'\x80\x03cpersistent.mapping\nPersistentMapping\nq\x00.\x80\x03}q\x01'
 b'X\x04\x00\x00\x00dataq\x02}q\x03(X\x03\x00\x00\x00objq\x04C\x08>'
 b'.\xe6\x8c\x1a\xb9\xf8\xd3q\x05cpersistent\nPersistent\nq\x06\x86q\x07QX\x04'
 b'\x00\x00\x00obj2q\x08C\x08>.\xe6\x8c\x1a\xb9\xf8\xd4q\th\x06\x86q\nQus.')

Setting the Serial Number

You might be wondering when the _p_serial of a persistent object is set. There are two times. First, when an object has been added or modified during a transaction, one of the last things ZODB does is update the _p_serial. Here's part of tpc_finish (the last part of committing a transaction) from Connection:

def tpc_finish(self, transaction):
    """Indicate confirmation that the transaction is done.
    """
    serial = self._storage.tpc_finish(transaction)
    assert type(serial) is bytes, repr(serial)
    for oid_iterator in self._modified, self._creating:
        for oid in oid_iterator:
            obj = self._cache.get(oid)
            # Ignore missing objects and don't update ghosts.
            if obj is not None and obj._p_changed is not None:
                obj._p_changed = 0
                obj._p_serial = serial

Notice that it's the underlying storage that's responsible for allocating the transaction ID. Likewise, the underlying storage is responsible for allocating new OIDs when objects are first stored.

Properties of Transaction IDs

From the examples above, we've learned a few things about transaction IDs.

Like OIDs, TIDs are also 8-byte, or 64-bit, numbers (b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x99').
They can be printed as those 8 bytes, but more often they are printed as a number (279171602205896857), usually in hexadecimal (0x3dfd117d4dbf099).
Most interestingly, transaction IDs can also be interpreted as points in UTC time (2021-05-03 16:23:49.888861).

The persistent.timestamp.TimeStamp class assists with parsing and formatting TIDs.

>>> from persistent.timestamp import TimeStamp
>>> ts = TimeStamp(b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x99')
>>> print(ts)
2021-05-03 16:23:49.888861
>>> ts.timeTime()
1620059029.888861

Transaction IDs Are Based On the Current Time

When a transaction commits, ZODB assigns it a transaction ID based on the current time. This is part of tpc_begin (the first part of committing a transaction) from the BaseStorage class, a common base class for many storage implementations:

def tpc_begin(self, transaction, tid=None, status=' '):
    ...
    if tid is None:
        now = time.time()
        t = TimeStamp(*(time.gmtime(now)[:5] + (now % 60,)))
        ...

Transaction IDs Must Increase

Each transaction that commits gets a new transaction ID, based on the current time. Since time moves forward (later is always later than now), that suggests that subsequent transaction IDs will just get bigger and bigger. And normally that would be true. But suppose the computer's clock gets set back, or suppose the database file is moved from one machine whose clock was running way ahead to another machine whose clock is set to the right time. How do we guarantee that a value derived from time.time() is actually going to be larger than the last value?

BaseStorage and the TimeStamp class have us covered. BaseStorage keeps track of the last TID that's been committed, and TimeStamp has a laterThan method that guarantees that the returned value is larger than the previous value:

def tpc_begin(self, transaction, tid=None, status=' '):
    ...
    if tid is None:
        now = time.time()
        t = TimeStamp(*(time.gmtime(now)[:5] + (now % 60,)))
        self._ts = t.laterThan(self._ts)

Different Transaction IDs Can Have the Same Timestamp

But there's a gotcha here. Just because the value is larger doesn't mean that it will actually print as a different time. The conversion from 64-bit number to floating point number of seconds since the epoch to human-readable string is lossy. We can see this if we continue the example we started above.

>>> later = ts.laterThan(ts)
>>> later > ts
True
>>> ts
b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x99'
>>> later
b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x9a'
>>> print(later)
2021-05-03 16:23:49.888861
>>> print(ts)
2021-05-03 16:23:49.888861
>>> later.timeTime()
1620059029.888861
>>> ts.timeTime()
1620059029.888861

Even though the numeric value of later is exactly one bigger than that of ts, they still print the same and have the same timeTime() value. Empirically, up to 16 sequential TID values can have the same timeTime() when using the C implementation of TimeStamp; the Python implementation rounds slightly differently and up to 70 sequential TID values have printed the same timeTime() (both implementations print the date string the same for 70 sequential TIDs). The takeaway? Don't rely too heavily on the timeTime() or printed date string to compare or store TIDs. Prefer instead to use either the raw() 8-byte value or its 64-bit integer representation.

Uses of Transaction IDs

In addition to detecting conflicts, or loading particular revisions of objects, ZODB can make explicit use of TIDs in other ways. Usually, this is through their form as a timestamp. For example, the pack method of IStorage, used to perform garbage collection, takes a "pack time" as its first argument. This number of seconds is converted into a TID through the TimeStamp APIs by most storages. (But, because of the gotcha outlined above, this is slightly approximate or potentially ambiguous.)

Similarly, the open method of ZODB.DB can accept an optional before= or at= parameter to request a historical view of the database. These parameters can be a specific datetime.datetime object representing an approximate timestamp, or they can be an exact 8-byte TID.

Transaction IDs in RelStorage

Earlier, I showed the code that BaseStorage and FileStorage use to create a new TID based on the current time. Since FileStorage can only be used by a single process on a single computer at a time, it's easy to guarantee that the TID is always increasing.

RelStorage is a bit different. The underlying SQL database in RelStorage can be used by many different processes on many different computers at the same time. There's no guarantee that all these computers will have their clocks set exactly the same, so trying to use the local time from each of them to create timestamps can lead to confusion (TIDs might seem to have large gaps or otherwise jump around, or may not even be related to the actual commit time at all!).

To combat this, RelStorage computes the next TID on the database server itself (for supported databases, namely MySQL and PostgreSQL). That way, there's only one clock involved.

RelStorage's client-side pickle cache is highly integrated with TIDs, but that's another post.

RelStorage 3.0

Jason Madden — Tue, 12 Nov 2019 11:18:52 GMT

We're happy to announce the release of RelStorage 3.0, the relational storage engine for ZODB. Compared to RelStorage 2, highlights include a 30% reduction in memory usage, and up to 98% faster performance! (Ok, yes, that's from one specific benchmark and not everything is 98% faster, but improved performance was a major goal.)

RelStorage 3.0 is a major release of RelStorage with a focus on performance and scalability. It's the result of a concentrated development effort spanning six months, with each pre-release being in production usage with large databases.

Read on to find out what's new.

Contents

Overview

Please note that this document is only an overview. For details on the extensive changes since the previous release, RelStorage 2.1, and this one, please view the detailed changelog.

If you're not familiar with ZODB (the native Python object database) and how it uses pluggable storage engines like RelStorage, please take a moment to review this introduction.

This document will cover a few important things to know about RelStorage 3.0, and then go over some of the most important changes in it. Next, we'll show how those changes affect performance. Finally, we'll wrap it up with a whirlwind tour of some of the minor changes.

Backwards Incompatible Changes

Before we get to the good stuff, it's important to highlight the small number of backwards incompatible changes and other things to be aware of when migrating from RelStorage 2 to RelStorage 3.

Schema Changes

In history preserving schemas, the empty column of the transaction table has been renamed to is_empty. In MySQL 8.0.4, empty became a reserved word. The table is altered automatically when first opened with RelStorage 3.0. This makes the schema incompatible with opening under RelStorage 2. [1]

A new table, used during the commit process, is also automatically added when first opened with RelStorage 3.

Under MySQL, any remaining tables that were using the MyISAM engine are converted to InnoDB when the schema is first opened. The only tables remaining that were MyISAM were the pack tables and the new_oid table, all of which should ordinarily be empty, so this conversion shouldn't take long.

Option Changes

The shared-blob-dir default has changed from true to false. If you were using a non-shared blob-dir, meaning that blobs were only stored on the filesystem, you'll need to explicitly set this to true. The previous default could easily lead to accidental data loss, and there is now a performance penalty for a true value. See blob_cache for more information.

The previously deprecated option poll-interval has been removed.

Several of the cache persistence options are now deprecated and ignored. They'll generate a warning on startup if found in the configuration.

Concurrent Deployment With RelStorage 2 Is Not Possible

Caution!

It is not possible for RelStorage 3 to write to a database at the same time that RelStorage 2 is writing to it.

The specifics around locking have changed entirely, and are not compatible between the two versions. If RelStorage 3 and RelStorage 2 are both writing to a database, corruption is the very likely result. For this reason, shutting down all RelStorage 2 instances, or at least placing them into read-only mode, is required.

RelStorage does not take specific steps to prevent this. It is up to you to ensure any RelStorage 2 instances are shutdown or at least read-only before deploying RelStorage 3.

Major Changes

Benchmarking Notes

The benchmark data was collected with zodbshootout 0.8. Recent revisions of zodbshootout have adopted pyperf as the underlying benchamrk engine. This helps ensure much more stable, consistent results. It also allows collecting a richer set of data for later analysis. The data was passed through the seaborn statistical visualization library, which is built using pandas, numpy and matplotlib, to produce the plots shown here.

For the comparisons between RelStorage 3 and RelStorage 2, the RDBMS servers (MySQL 8 and PostgreSQL 11) were run on one computer running Gentoo Linux using SSD storage. The RelStorage client was run on a different computer, and the two were connected with gigabit ethernet. The machines were otherwise idle.

The comparisons used Python 2.7.16 (because 2.7 was the only version of Python that had a native code gevent driver compatible with RelStorage 2). The database drivers were mysqlclient 1.4.4 and psycopg2 2.8.4. The gevent driver used this version of ultramysql, umysqldb 1.0.4.dev2, PyMySQL 0.9.3 and gevent 1.5a2.

Why transaction of size 1, 5 and 20 objects? Review of a database containing 60 million objects showed the average transaction involved 2.6 objects with a standard deviation of 7.4. A database with 30 million objects had an average transaction of 6.1 objects and a standard deviation of 13.

In all the examples that follow, results for PostgreSQL are green while those for MySQL are in blue. The darker shades are RelStorage 3, while the lighter shades are RelStorage 2. The y axis is time, and shorter bars are better (no units or tickmarks are shown because the scale differs between graphs and we're generally focused on deltas not absolute values); the black line that appears in the middle of some bars is the confidence interval. Click for a larger version.

Most examples show a cross section of RDBMS server by concurrency kind (threads or process) by concurrency level (1, 5, or 20 concurrent threads or processes) by object count (1, 5 or 20 objects).

To accomplish its goals of improving performance (especially under high loads distributed across many processes on many machines), reducing memory usage, and being usable in small environments without access to a full RDBMS server (such as containers or test environments), RelStorage features several major internal changes.

Pickle Cache

The shared in-memory pickle cache has been redesigned to be precise and MVCC based; it no longer uses the old checkpoint system. This means that old revisions of objects can proactively be removed from the cache when they are no longer needed. Together, this means that connections within a process are able to share polling information, with the upshot being that there are no longer large, stop-the-world poll queries in order to rebuild checkpoints. Individual poll queries are usually smaller too.

It has also been changed to have substantially less overhead for each cached object value. Previously, it would take almost 400 bytes of overhead to store one cache entry. In examining a database of 60 million objects, it turned out that the average object size was only a little over 200 bytes. Using 400 bytes to store 200 bytes was embarrassing, and because the cache limit computations didn't take the overhead into account it meant that if you configured an in-memory cache size of 200MB, the cache could actually occupy up to 600MB.

Now, storing a cached value needs only a little more than 100 bytes, and the exact amount of overhead is included when enforcing the cache limit (so a 200MB limit means 200MB of memory usage). The Python cache implementation using a CFFI-based Segmented LRU and Python lookup dictionary were replaced with a minimal Cython extension using a C++ boost.intrusive list and map. This also essentially eliminates the cache's impact on the Python garbage collector, which should improve garbage collection times [2].

In concrete terms, one set of production processes that required almost 19GB of memory now requires only about 12GB: a 36% reduction.

Persistent Cache

Along with rearchitecting the in-memory cache, the on-disk persistent cache has been rebuilt on top of SQLite. Its hit rate is much improved and can easily reach 100% if nothing in the database changed. If you haven't deployed the persistent cache before, now would be a great time to give it a try.

If you had used the persistent cache in the past, the new cache should just work. Old cache files will be ignored and you might want to manually remove any that exist to reclaim disk space.

ZODB 5 Parallel Commit

RelStorage 3 now requires ZODB 5, and implements ZODB 5's parallel commit feature. During most of the ZODB commit process, including conflict resolution, only objects being modified are exclusively locked. Objects that were provided to Connection.readCurrent() are locked only in share mode so they may be locked that way by several transactions at the same time. (This fact is particularly important because BTrees call readCurrent() for every node traversed while searching for the correct leaf node to add/remove/update, meaning there can be a surprising amount of contention.)

Only at the very end of the ZODB commit process when it is time to commit to the database server is a database-wide lock taken while the transaction ID is allocated. This should be a very brief time, so transactions that are operating on distinct sets of objects can continue concurrently for much longer (especially if conflicts occur that must be resolved; one thread resolving conflicts no longer prevents other threads from resolving non-overlapping conflicts).

This works on most databases, but it works best on a database that supports NOWAIT share locks, like MySQL 8 or PostgreSQL. SQLite doesn't support object-level locking or parallel commit. Oracle doesn't support shared object-level locks.

ZODB 5 Prefetch

RelStorage 3 implements efficient object prefetching through Connection.prefetch(). This is up to 74% faster than reading objects individually on demand.

RelStorage 2 did not implement prefetch so this benchmark falls back to reading objects individually. RelStorage 3 is able to query the database in a single bulk operation.

Support for SQLite

Note

The SQLite support is relatively new and hasn't received much production-level testing.

On some systems, the underlying sqlite3 module may experience crashes when lots of threads are used (even though a ZODB connection and its RelStorage instance and sqlite connection are not thread safe and must only ever be used from a single thread at a time, the sequential use from multiple threads can still cause issues).

RelStorage 3 can use a local SQLite3 database file. I'll quote the RelStorage FAQ to explain why:

Why does RelStorage support a SQLite backend? Doesn't that defeat the point?

The SQLite backend fills a gap between FileStorage and an external RDBMS server.

FileStorage is fast, requires few resources, and has no external dependencies. This makes it well suited to small applications, embedded applications, or applications where resources are constrained or ease of deployment is important (for example, in containers).

However, a FileStorage can only be opened by one process at a time. Within that process, as soon as a thread begins committing, other threads are locked out of committing.

An external RDBMS server (e.g., PostgreSQL) is fast, flexible and provides lots of options for managing backups and replications and performance. It can be used concurrently by many clients on many machines, any number of which can be committing in parallel. But that flexibility comes with a cost: it must be setup and managed. Sometimes running additional processes complicates deployment scenarios or is undesirable (for example, in containers).

A SQLite database combines the low resource usage and deployment simplicity of FileStorage with the ability for many processes to read from and write to the database concurrently. Plus, it's typically faster than ZEO. The tradeoff: all processes using the database must be on a single machine on order to share memory.

I'll leave the rest to the FAQ, with the exception of these two performance graphs showing SQLite slot in comfortably next to FileStorage and ZEO.

For small to medium sized write transactions, SQLite can actually outperform FileStorage and ZEO when threads are in use. When separate processes are in use, SQLite always beats ZEO.

When reading objects, SQLite is always faster than ZEO, but slower than FileStorage.

Performance Improvements

Much effort was spent on improving RelStorage's performance, in terms of overall speed and memory usage as well as concurrency and scalability. Here, let's compare the performance of RelStorage 2.1.1 with RelStorage 3.0 graphically from a speed perspective.

Writing

Note

RelStorage 2 failed to complete a number of the write benchmarks that used 20 objects and 20 concurrent processes or threads: some tasks would fail to obtain the commit lock using the default 10 second timeout. Those tasks were excluded from the results.

RelStorage 3 did not have this problem.

We'll start with writing to the database:

When simply adding new objects to the database, PostgreSQL is between 29% and 72% faster. MySQL is essentially within the margin of error for the non-concurrent cases, and up to 79% faster for larger, more concurrent tests.

Updating existing objects:

When updating objects that already existed in the database, The difference for MySQL ranges between statistically insignificant up to to 79% faster.. PostgreSQL likewise ranges from statistical insignificance and up to 80% faster.

Handling conflicts received extra attention.

Updating objects that have to resolve conflicts.

Once again, some cases were statistically insignificant for both databases. The cases that were statistically significant show a 15% to 84% improvement for PostgreSQL, with the range being from 11% to 89% for MySQL. Not only is it faster, going by the much tighter confidence intervals, it's also less volatile and more consistent.

Reading

RelStorage, like ZEO, includes a secondary pickle cache that's shared amongst all the connections in a process. Here's what it looks like to read data out of that cache (not hitting the database at all).

Reading from the pickle cache, by concurrency kind and object count.

Ideally the bars for both databases of a particular RelStorage release would be equal heights in any given test case because this test is database independent. That's not quite the case. When they're not, they're within each other's confidence intervals.

Because the bottom row of graphs is separate processes, they don't benefit much from the actual sharing of the cache. The improvements there, 30 – 40%, show the effect of the cache changes in isolation. The top row of graphs shows the improvement in the intended use case, when the cache is shared by multiple threads in a process. In that case, the difference can be up to 90%. [5]

Reading directly from the database, on the other hand, is harder to qualify.

Oh no! It looks like RelStorage 3 actually got slower when reading directly from the database. That's one of its core tasks. How could that be allowed to happen? (Spoiler alert: it didn't.)

Look closely at the pattern. In the top row, when we're testing with threads, RelStorage 3 is always at least as good as RelStorage 2 and frequently better [4]. It's only the bottom row, dealing with processes, that RelStorage 3 looks bad. But as you move further to the right, where more processes are making more queries to load more objects, the gap begins to close. At 20 objects in 5 processes, the gap is essentially gone. (And then we fall off a cliff at 20 processes querying 20 objects. It's not entirely clear exactly what's going on, but that's more CPUs/hardware threads than either the client machine or server machine has so it's not surprising that efficiency begins to fall.)

It turns out the benchmark includes the cost of opening a ZODB connection. For processes, that's a connection using a whole new ZODB instance, so there will be no prior connections open. But for threads, the ZODB instance is shared, so there will be other connections in the pool.

Working with brand new RelStorage connections got a bit slower in RelStorage 3 compared to RelStorage 2. They use more prepared statements (especially on PostgreSQL), and they use more database session state (especially on MySQL). Performing the first poll of the database state may also be a bit more expensive. So when the connection doesn't get used to do much before being closed, these costs outweigh the other speedups in RelStorage 3. But somewhere between making 5 and 20 queries for objects, the upfront costs are essentially amortized away. As always in ZODB, connection pooling with appropriate pool settings is important.

gevent

In RelStorage 2, gevent was supported on Python 2 and Python 3 for MySQL and PostgreSQL when using a pure-Python database driver (typically PyMySQL for the former and pg8000 for the later). There was special gevent support for MySQL using a custom database driver, but only on Python 2. This driver took a hybrid approach, providing some C acceleration of low-level operations, but delegating most operations to PyMySQL.

RelStorage 3 supports gevent-aware, fully native drivers, for both PostgreSQL and MySQL on both Python 2 and Python 3. Moreover, the MySQL driver has special support for RelStorage's two-phase commit protocol, essentially boosting the priority of a greenlet that's committing [6]. This avoid situations where a greenlet takes database-wide locks and then yields control to a different greenlet that starves the event loop, leaving the database locked for an unacceptable amount of time and halting the forward progress of other processes.

Do these things make a difference? We can compare the performance of gevent with MySQL to find out.

Adding objects, by concurrency type.

Adding objects improved across the board for all concurrency types.

Updating objects, by concurrency type.

Updating objects improved across the board for all concurrency types. Updating conflicting objects shows a similar gain.

Reading objects, by concurrency type.

Reading individual objects, by contrast, shows no distinct trend. I suspect that it's essentially unchanged (the transactional and polling parts around it that were changed are not measured here), but we'd need more samples to be able to properly show that.

Minor Changes

This section documents some of the other changes in RelStorage 3.

Supported Versions

Support for PostgreSQL 12 and MySQL 8 were added, as well as support for Python 3.8.

Support for MySQL 5.6 and PostgreSQL 9.5 were removed, as was support for old versions of ZODB. Also, RelStorage no longer depends on ZEO (so it's theoretically possible that Python 2.7.8 and earlier could run RelStorage, but this isn't tested or recommended).

Most tested database drivers were updated to newer versions, and in some cases the minimum supported versions were updated.

mysqlclient must be 1.4, up from 1.3.7.
psycopg2 must be 2.8, up from 1.6.1.
psycopg2cffi must be 2.8.1, up from 2.7.4.
cx_Oracle must be 6.0, up from 5.0.
Support was removed for the Python 2-only driver umysqldb.

This table summarizes the support various databases have for RelStorage 3 features.

Supported Features
	PostgreSQL	MySQL	Oracle	SQLite
Parallel commit	Yes	Yes	Yes	No
Shared readCurrent locks	Yes	Yes	No	No
Non-blocking readCurrent locks	Yes	Native on MySQL 8, emulated on MySQL 5.7	Yes	N/A (there is no distinction in lock type)
Streaming blobs	Yes	No (emulated via chunking)	Yes	No (consider configuring a shared-blob-dir)
Central transaction ID allocation	Yes	Yes	No (could probably be implemented)	N/A (but essentially yes because it only involves one machine)
Atomic lock and commit without Python involvement	Yes (except with PG8000)	Yes	No (could probably be implemented)	No

The Blob Cache

Using a shared-blob-dir (where all blobs are only stored on a filesystem and never in the database) disables much of the parallel commit features. This is because testing whether we can actually store the blob successfully during the "vote" phase of ZODB's two-phase commit requires knowing the transaction ID, and knowing the transaction ID requires taking the database-wide commit lock. This is much sooner than is otherwise required and the lock is held for much longer (e.g., during conflict resolution).

Increasing popularity, and ever-growing databases, make the implementation of the blob cache all the more important. This release focused on blob cache maintenance, specifically the process whereby the blob-cache-size (if any) is ensured.

First, for history free databases, when a new revision of a blob is uploaded to replace an older one, if RelStorage has the old revision cached on disk and can determine that it's not in use, it will be deleted as part of the commit process. This applies whether or not a cache size limit is in place.

If it becomes necessary to prune the blob cache, the process of doing so has been streamlined. It spawns far fewer unnecessary threads than it used to. If the process is using gevent, it uses an actual native thread to do the disk scan and IO instead of a greenlet, which would have blocked the event loop.

Finally, if running the pruning process is still too expensive and the thread interferes with the work of the process, there's a new option that spawns a separate process to do the cleanup. This can also be used manually to perform a cleanup without opening a storage.

Packing and GC

History-preserving databases now support zc.zodbdgc for multi-database garbage collection.

RelStorage's native packing is now safer for concurrent use in history-free databases thanks to correcting several race conditions.

For both types of databases, packing and pre-packing require substantially less memory. Pre-packing a large database was measured to use 9 times less memory on CPython 3 and 15 times less on CPython 2 (from 3GB to 200 MB).

Performance Grab Bag

Here's a miscellaneous selection of interesting changes, mostly performance related.

Reduce the number of network communications with the database.

RelStorage tries harder to avoid talking to the database more times than necessary. Each round-trip introduces extra latency that was measurable, even on fast connections. Also, native database drivers usually release the GIL during a database operation, so there could be extra overhead introduced acquiring it again. And under gevent, making a query yields to the event loop, which is good, but it could be an arbitrary amount of time before the greenlet regains control to process the response. If locks are being held, too many queries could spell disaster.

This was accomplished in several ways. One way was to move larger sequences of commands into stored procedures (MySQL and PostgreSQL only). For example, previously to finish committing a transaction, RelStorage required 7 database interactions in a history-preserving database: 1 to acquire the lock, 1 to get the previous transaction id, 1 to store transaction metadata, 1 to store objects, 1 to store blobs, 1 to update the current object pointers, and finally one to commit. Now, that's all handled by a single stored procedure using one database operation. The Python process doesn't need to acquire the GIL (or cycle through the event loop) to commit and release locks, that happens immediately on the database server regardless of how responsive the Python process is.

More careful control of transactions eliminated several superflous COMMIT or ROLLBACK queries in all databases. Similarly, more careful tracking of allocated object identifiers (_p_oid) eliminated some unnecessary database updates.

The use of upserts, which eliminate at least one query, was previously limited to a select few places for PostgreSQL and MySQL. That has been extended to more places for all supported databases.
Allocate transaction IDs on the database server.

This was primarily about reducing database communications. However, because transaction IDs are based on the current time, it also has the important side-effect of ensuring that they're more consistently meaningful with only one clock to consider.

Previously, all transaction IDs could be at least as inaccurate as the least-accurate clock writing to the database (if that clock was in the future).
PostgreSQL uses the COPY command to upload data.

Specifically, it uses the binary format of the bulk-loading COPY command to stream data to the server. This can improve storage times by 20% or so.

Conclusion

RelStorage 3 represents a substantial change from RelStorage 2. The pickle cache—both in-memory and on-disk—has been completely rewritten, the locking process has been re-imagined in support of parallel commit, time-sensitive logic moved into stored procedures, and more. Despite that, it should be a drop-in replacement in most situations.

Although RelStorage 3 has been in production usage under heavy load at NextThought for its entire development cycle, and we haven't encountered any problems with it that could lead to data loss, it's still software, and all software has bugs. Please exercise appropriate care when upgrading. Bug reports and pull requests are encouraged and appreciated.

We're very happy with the enhancements, especially around performance, and hope those improvements are applicable to most users. We welcome feedback on whether they are or are not, and also want to hear about where else RelStorage could improve.

Finally, I'd like to say thank you to everyone who has contributed to the development of RelStorage 3, whether through testing pre-releases, filing bug reports, or sharing enhancement ideas and use cases. It's greatly appreciated.

Footnotes

[1]

Why not simply "quote" the reserved word? Much of the SQL queries that RelStorage uses are shared among all supported databases. By default, MySQL uses a different, non-standard quoting syntax that wouldn't work with the other databases. That can be changed by altering the SQL Mode, but I was trying to avoid having to do that. In the end it turned out that another change [3] forced the alteration of the mode, so I should have just done that in the first place. But since there are good reasons to prevent RelStorage 2 and 3 from ever trying to use the same database, the incompatibility didn't seem like a big deal.

[2]

In CPython, the generational (cyclic) garbage collector uses time proportional to the number of objects in a generation (all objects in a generation are stored in a linked list). The more objects that exist, the longer it takes to perform a collection. In RelStorage 2, storing a value in the cache required creating several objects, and the Python garbage collector would have to examine these. The RelStorage 3 cache does not create objects that the garbage collector needs to traverse. Similar remarks hold for PyPy.

[3]	That change was support for SQLite, which requires the entire "transaction" table to be quoted. Amusingly, "transaction" is a word reserved by the SQL standard, while "empty" is not.

[4]	With the exception of PostgreSQL using one thread for 20 objects. The large error bar indicates an outlier event. We're working with relatively small sample sizes, so that throws things off.

[5]

RelStorage 2 implemented some of its cache functions using native code called by CFFI. When CFFI calls native code, Python's GIL is dropped, allowing other threads to run (as long as they weren't trying to use the cache, which used Python locks for thread safety). In contrast, RelStorage 3 uses a thin layer of cython to call into C++ and does not drop the GIL—it depends on the GIL for thread safety. The substantial speed improvement should outweigh the loss of the tiny window where the GIL was dropped.

[6]

We don't have that much granular control over the PostgreSQL driver (psycopg2). With MySQL (mysqlclient), on a connection-by-connection basis we can control if a particular query is going to yield to gevent or block. But with psycopg2, whether it yields or not is global to the entire process.

The reverse is that psycopg2 actually gives us exactly the same control as a gevent socket (yield each time any read/write would block), whereas in mysqlclient we can only wait for the first packet to arrive/go, after that it blocks for the duration (server-side cursors give us a bit more control, allowing yields between fetching groups of rows).

Introduction to ZODB Data Storage

Jason Madden — Mon, 28 Oct 2019 18:07:52 GMT

ZODB is a powerful native object database for Python, widely known for its use in the Zope web framework and the Plone content management system. By enabling transparent object graph persistence with no need to predefine schemas, ZODB enables extremely flexible application development. With pluggable storage engines such as FileStorage, ZEO, and RelStorage, it also provides flexible ways to store data.

This post provides an introduction to ZODB, focusing on some of the lower-level mechanics of storing data. This post doesn't discuss persistent objects.

Disclaimer

This was written in support of the RelStorage 3.0 release so it may be biased in that direction. It is not an exhaustive list of all storage options. For example, it doesn't discuss NEO, a distributed, redundant storage. Partial lists of included and non-included storages may be found in the ZODB storage documentation.

Contents

What is ZODB?

ZODB [1] is a native object database for Python, enabling transparent object persistence. It provides the illusion of an infinite memory space holding application-defined objects. That memory space is shared between processes running at different times on the same or different machine. Only those objects actually used are brought into physical memory. Think of it something like operating system paging, but for objects, and distributed across time and space. (Apple's CoreData framework has a similar technique it calls "faulting".)

In addition, ZODB provides a transactional view of these objects with snapshot isolation. Any given connection to the database sees a consistent view of all the objects in the database (whether it reads or writes to any particular object or not) as-of the moment it began. When adding or updating objects, no changes are published and made visible to other connections until the writing connection commits its transaction, at which point either all the changes are made visible or none of them are. Existing connections that continue reading (or even writing!) will still not see those changes; they're "stuck" at the snapshot view of the objects they started with. (The ability for readers to continue to be able to retrieve old data that's been replaced in newer transactions is known as multi-version concurrency control, or MVCC.)

Many connections may be reading and writing to the database at once. ZODB uses optimistic concurrency control. Readers don't block other readers or writers, and writers are allowed to proceed as if they were the only one making changes right up until they commit. Writes are defined to occur in a strict order. If a writer discovers that an earlier transaction had modified objects that it too wants to modify, a conflict occurs. Instead of just rolling back the writing transaction and forcing it to start over, taking the modified object into account, ZODB gives the application the chance to resolve the conflict using a three-way merge between the object as it existed when the transaction began, the object that the connection wants to commit, and the object that was committed by the other writer. Only if it cannot do so is the transaction rolled back.

What is a ZODB storage?

ZODB uses a pluggable storage architecture, allowing different ways to store the objects it manages. Storage engines are responsible for allocating persistent object identifiers (OIDs) for each object ZODB manages, storing object state data [2] when an object is added or changed [3], and later retrieving the data for that particular object given its OID. The storage is also responsible for implementing snapshot isolation, ordering (serializing) writes and assigning incrementing transaction identifiers (TIDs), and detecting and handling conflicting writes.

FileStorage

Out of the box, in addition to a few different transient (in-memory) storage engines, ZODB comes with one persistent (on-disk) storage engine. FileStorage uses a single file to store an append-only transaction log for all the objects in the database. An additional in-memory and on-disk structure is used to record the relationship between objects (OIDs) and the transactions (TIDs) they appear in.

As an append-only file, writing to FileStorage can be quite fast. It requires memory (and extra storage space) proportional to the size of the database to record object positions for fast access. If that extra index data isn't saved to disk, it requires time proportional to the size of the database to scan the file on startup to re-create that index.

Because of its append-only nature, previous versions of objects are still found in the file and can be accessed by providing a proper TID. A FileStorage is thus said to be "history preserving." That's how snapshot isolation is implemented: each connection is explicitly associated with a TID and when it needs to read an object it asks the FileStorage to provide the revision of the object most recently written before that TID [4]. This can also be used like a version control system to view and even recover or undo changes to objects. Periodically, a FileStorage must be "packed" to remove obsolete historical data and prevent the file from growing forever.

FileStorage is widely deployed and has a long history of stability. It can only be used by a single process at a time, however. Within that process, only a single thread can be in the process of committing a transaction at a time (FileStorage uses a database-wide lock to provide serialization).

ZEO

A common method to extend access to a FileStorage to more than one process and/or to more than one machine is to deploy a ZEO [5] server. ZEO uses a client/server architecture. The server process opens one or more storages (in practice, always a FileStorage [6]) and exposes a network API to provide access to this storage. Client processes connect to this server and send it read and write requests. The server mediates access to the underlying storage for the clients.

ZEO inherits many of the strengths and weaknesses of its underlying storage and adds some of its own. For example, clients can be configured with a persistent local cache for cheap access to common objects or even read-only access when the server isn't available. But the central ZEO process has to contend with Python's GIL, which may limit scalability, and it defaults to resolving conflicts by loading application code into the server process, which can complicate deployments due to the need to keep client and server processes all running compatible code.

ZRS [7] is a storage wrapper implemented in Python and commonly wrapped around a ZEO storage that provides replication of data.

What is RelStorage?

RelStorage is a ZODB storage engine that's meant to solve many of the same problems as ZEO and ZRS, but taking a different approach with a different set of tradeoffs. RelStorage uses a relational database—MySQL, PostgreSQL, Oracle, or SQLite—to provide the final storage for object state data. It pushes the responsibility for OID allocation, locks, transaction management and snapshot isolation, and replication down to these systems.

The next section is mostly a copy of RelStorage's own description of its features. It makes references to ZEO and FileStorage described above.

Features

It is a drop-in replacement for FileStorage and ZEO, with several enhancements:
- Supports undo, packing, and object history preservation just like FileStorage.
- RelStorage can be configured not to keep object histories for reduced disk space usage and improved performance.
- Multiple processes on a single machine can read and write a local ZODB database using SQLite without needing to start and manage another process (i.e., ZEO).
- Blobs can be stored on a shared filesystem, or (recommended) in the relational database and only cached locally.
- Multiple threads in the same process share a high-performance in-memory pickle cache to reduce the number of queries to the RDBMS. This is similar to ZEO, and the ZEO cache trace tools are supported.
- The in-memory pickle cache can be saved to disk and read when a process starts up. This can dramatically speed up site warmup time by eliminating a flood of RDBMS queries. Unlike ZEO, this cache is automatically shared by all processes on the machine (no need to configure separate client identifiers.)
Ideal for large, high volume sites.
- Multiple Python processes on multiple machines can read and write the same ZODB database concurrently. This is similar to ZEO, but RelStorage does not require ZEO.
- Supports ZODB 5's parallel commit feature: Database writers only block each other when they would conflict (except for a small window at the end of the twophase commit protocol when the transaction ID is allocated; that still requires a global database lock).
- According to some tests, RelStorage handles concurrency better than the standard combination of ZEO and FileStorage.
- Whereas FileStorage takes longer to start as the database grows due to an in-memory index of all objects, RelStorage starts quickly regardless of database size.
- Capable of failover to replicated SQL databases.
Tested integration with gevent for PostgreSQL and MySQL.
There is a simple way (zodbconvert) to (incrementally) convert FileStorage to RelStorage and back again. You can also convert a RelStorage instance to a different relational database. This is a general tool that can be used to convert between any two ZODB storage implementations.
There is a simple way (zodbpack) to pack databases.
Supports zodburi .
Free, open source (ZPL 2.1)

MVCC and History Free Storage

One thing in particular I'd like to highlight is that RelStorage can implement snapshot isolation and conflict resolution without preserving history. To do this, it relies on the RDBMS's native implementation of MVCC, the repeatable read isolation level, and the read committed isolation level.

When a transaction begins, a RDBMS transaction is opened on a connection at the repeatable read (or higher) level. This connection is used for loading data from the database. This isolation level causes the RDBMS to establish its own snapshot view of the database as-of that moment of time.

A second connection is used to write data to the database. This connection is in the lower isolation level of simply read committed. This level ensures that each query it makes to the database returns the latest committed data. Objects being written are first placed in a temporary table; they are moved to their final table (overwriting an old revision for history free storages) only after any possible conflicts have been found and resolved.

The difference in the two connections' isolation levels matters specifically because of conflict resolution, as does the use of a temporary table. Recall that resolving conflicts needs three versions of the object: the object that existed when the transaction began (the original object), the object that is currently committed and was changed by someone else (the committed object), and the object that the writer would like to store (the new object). The task of the conflict resolution is to find the delta between the original object and the new object and apply those same changes to the committed object. This produces a new object to store which will become the committed object.

Ignoring caches, the only place that original object can come from is that load connection at repeatable read isolation level. By definition, any fresh connection or transaction that looked at the database now would see the currently committed object (or something even later)—the original object has been overwritten and that change committed, so it's gone. RelStorage relies on the underlying database to keep it visible to the load connection.

Likewise, getting the currently committed object requires a connection that can read the current state of the database. That's where the second connection comes in. It can see the current data in the database.

Q & A

Why two connections? Why not put the data in the temporary table, commit, and begin a new transaction to update the current view of the database?

Because that would lose access to the original object.

Why not preemptively store off all the original objects somewhere (e.g., download them or copy them to a temp table) before committing?

Because ZODB uses an optimistic concurrency model. We assume that conflicts are few and far between. If that's true, that would be doing a bunch of extra work that we don't usually need to do. Remember, there's no way to know if there's going to be a conflict or not without a current view of the database.

Well then, why not just have a single shared connection for the current view of the database and use it to check for conflicts and only then save the original objects that have conflicts?

Because that connection wouldn't know what objects to check for conflicts on. Those objects are already in the database in temporary tables that are connection specific and unreadable to a different connection. We'd have to pass a list of object IDs back to the database, and not all databases support array operations to do that efficiently. Or we'd have to write to a persistent table, which doesn't sound appealing (we'd have to arrange to delete from it too.)

Also, because RDBMS connections aren't thread-safe, that would introduce a per-process lock into the commit process.

Still, perhaps that's worth looking into more.

Couldn't a history-preserving database implement snapshot isolation just like FileStorage and use only one read committed connection?

Quite possibly, yes. That could make for some moderately ugly or inefficient SQL queries though.

SELECT *
FROM object_state
WHERE zoid = :zoid
AND tid <= :tid
ORDER BY tid DESC
LIMIT 1

Why temp tables? Why not store directly to the final table?

For history free databases, the final table is where we get the data to resolve conflicts, so we can't overwrite it.

For history preserving databases, we don't yet have the necessary transaction ID we need to store to the final table. (The primary key is (OID, TID), and the TID is a foreign key reference to another tables as well).

We could allocate the TID earlier, before storing temporary data, but that defeats much of the benefit of ZODB 5 parallel commit.

We could use a fake TID and update it in-place, but altering primary keys tends to be expensive.

Conflict Resolution

RelStorage supports conflict resolution. Conflict resolution is performed in each individual process in a distributed fashion. There's no central server that has to be updated with application code in order to resolve conflicts. ZEO 5 supports a similar feature.

Summary

ZODB is a flexible and powerful object database for Python, supporting transactions, optimistic concurrency, and conflict resolution. It uses a layered architecture with the definition and serialization of individual objects handled by the persistent library, the generic transactional API provided by transaction, and data storage and MVCC semantics provided by the pluggable storage layer.

ZODB comes with a storage implementation using an append-only file, as well as an in-memory dict-based storage plus a change-tracking demo storage. These are all restricted to a single process, but ZEO allows utilizing them from multiple processes.

RelStorage is a storage layer based on a SQL database, intended to be highly scalable.

Updates

Add additional links to more resources about included and third-party storages.

Footnotes

[1]	ZODB may stand for "Zope Object Database," or it may stand for "Z Object Database."

[2]	To the storage engine, the object state data is just an opaque sequence of bytes. In reality, the ZODB Connection uses Python's standard pickle protocol to serialize objects into bytes.

[3]	Storages also handle non-object data in the form of BLOBs, each of which is associated with an object and assigned an OID.

[4]	Actually, the layer that implements snapshot isolation on top of an arbitrary history preserving storage is found in the core of ZODB. This was one of the major changes in ZODB 5.

[5]	Previously "Zope Enterprise Objects".

[6]	Though one possibly wrapped in something like zlibstorage to provide compression or `cipher.encryptingstorage` to provide encryption.

[7]	"ZODB Replicated Storage"

Python Argument Surprise

Jason Madden — Fri, 23 Nov 2018 15:11:07 GMT

Python function signatures are flexible, complex beasts, allowing for positional, keyword, variable, and variable keyword arguments (and parameters). This can be extremely useful, but sometimes the intersection between these features can be confusing or even surprising, especially on Python 2. What do you expect this to return?

>>> def test(arg1, **kwargs):
...     return arg1
>>> test(**{'arg1': 42})
...

Contents

Terminology

Before we move on, let's take a minute to define some terms.

parameter: The name of a variable listed by a function definition. These are sometimes called "formal parameters" or "formal arguments." In def foo(a, b), a and b are parameters.
argument: The expression given to a function application (function call). In foo(1, "str"), 1 and "str" are arguments.
function signature: The set of parameters in a function definition. This is also known as a "parameter list."
binding: The process of associating function call arguments to the parameter names given in the function's signature. In foo(1, "str"), the parameter a will be assigned the value 1, and the parameter b will be assigned the value "str". This is also called "argument filling."
default parameter: In a function signature, a parameter that is assigned a value. An argument for this parameter does not have to be given in a function application; when it is not, the default value is bound to the parameter. In def foo(a, b=42), b=42 creates a default parameter. It can also be said that b has a default parameter value. The function can be called as foo(1).
positional argument: An argument in a function call that's given in order of the parameters in the function signature, from left to right. In foo(1, 2), 1 and 2 are positional arguments that will be bound to the parameters a and b.
keyword argument: An argument in a function call that's given by name, matching the name of a parameter. In foo(a=1), a=1 is a keyword argument, and the parameter a will have the value 1.
variable parameters: A function signature that contains *args (where args is an arbitrary identifier) accepts an arbitrary number of unnamed arguments in addition to any explicit parameters. Extra parameters are bound to args as a list. def foo(a, b, *args) creates a function that has variable parameters, and foo(1, 2), foo(1, 2, 3), foo(1, 2, 3, 4) are all valid ways to call it. This is commonly called "varargs," for "variable arguments" (even though it is a parameter definition).
variable positional arguments: Passing an arbitrary (usually unknown from the function call itself) number of arguments to a function by unpacking a sequence. Variable arguments can be given to a function whether or not it accepts variable parameters (if it doesn't, the number of variable arguments must match the number of parameters). This is done using the * syntax: foo(*(1, 2)) is the same as writing foo(1, 2), but more often the arguments are created dynamically. For example, args = (1, 2) if full_moon else (3, 4); foo(*args).
variable keyword parameters: A function signature that contains **kwargs (where kwargs is an arbitrary identifier) accepts an arbitrary number of keyword arguments in addition to any explicit parameters (with or without default values). The definition def foo(a, b, **kwargs) creates a function with a variable keyword parameter. It can be called like foo(1, 2) or foo(1, 2, c=42, d=420).
variable positional arguments: Similar to variable positional arguments, but using keyword arguments. The syntax is **, and the object to be unpacked must be a mapping; extra arguments are placed in a mapping bound to the parameter identifier. A simple example is foo(**{'b':"str", 'a':1}).

Some language communities are fairly strict about the usage of these terms, but the Python community is often fairly informal. This is especially true when it comes to the distinction between parameters and arguments (despite it being a FAQ) which helps lead to some of the surprises we discuss below.

Surprises

On to the surprises. These will all come from the intersection of the various terms defined above. Not all of these will surprise everyone, but I would be surprised if most people didn't discover at least one mildly surprising thing.

Non-Default Parameters Accept Keyword Arguments

Any parameter can be called using a keyword argument, whether or not it has a default parameter value:

>>> def test(a, b, c=42):
...    return (a, b, c)
>>> test(1, 2)
(1, 2, 42)
>>> test(1, b='b')
(1, 'b', 42)
>>> test(c=1, b=2, a=3)
(3, 2, 1)

This is surprising because sometimes parameters with a default value are referred to as "keyword parameters" or "keyword arguments," suggesting that only they can be called using a keyword argument. In reality, the parameter just has a default value. It's the function call site that determines whether to use a keyword argument or not.

One consequence of this: the parameter names of public functions, even if they don't have a default, are part of the public signature of the function. If you distribute a library, people can and will call functions using keyword arguments for parameters you didn't expect them to. Changing parameter names can thus break backwards compatibility. (Below we'll see how Python 3 can help with this.)

Corollary: Variable Keyword Arguments Can Bind Non-Default Parameters

If we introduce variable keyword arguments, we see that this behaviour is consistent:

>>> kwargs = {'a': 'akw', 'b': 'bkw'}
>>> test(**kwargs)
('akw', 'bkw', 42)

Corollary: Positional Parameters Consume Keyword Arguments

Knowing what we know now, we can answer the teaser from the beginning of the article:

>>> def test(arg1, **kwargs):
...     return arg1, kwargs
>>> test(**{'arg1': 42})
(42, {})

The named parameter arg1, even when passed in a variable keyword argument, is still bound by name. There are no extra arguments to place in kwargs.

Default Parameters Accept Positional Arguments

Any parameter can be called using a positional argument, whether or not it has a default parameter value:

>>> def test(a=1, b=2, c=3):
...   return (a, b, c)
>>> test('a', 'b', 'c')
('a', 'b', 'c')

This is the inverse of the previous surprise. It may be surprising for the same reason, the conflation of keyword arguments and default parameter values.

Of course, convention often dictates that default parameters are passed using keyword arguments, but as you can see, that's not a requirement of the language.

Corollary: Variable Positional Arguments Can Bind Default Parameters

Introducing variable positional arguments shows consistent behaviour:

>>> pos_args = ('apos', 'bpos')
>>> test(*pos_args)
('apos', 'bpos', 3)

Mixing Variable Parameters and Keyword Arguments Will Break

Suppose we'd like to define some parameters with default values (expecting them to be passed as keyword arguments by convention), and then also allow for some extra arguments to be passed:

>>> def test(a, b=1, *args):
...    return (a, b, args)

The definition works. Now lets call it in some common patterns:

>>> test('a', 'b')
('a', 'b', ())
>>> test('a', 'b', 'c')
('a', 'b', ('c',))
>>> test('a', b=1)
('a', 1, ())

>>> test('a', b='b', *(1, 2))
Traceback (most recent call last):
...
TypeError: test() got multiple values for argument 'b'

As long as we don't mix keyword and variable (extra) arguments, everything works out. But as soon as we mix the two, the variable positional arguments are bound first, and then we have a duplicate keyword argument left over for b.

This is a common enough source of errors that, as we'll see below, Python 3 added some extra help for it, and linters warn about it.

Functions Implemented In C Can Break The Rules

We generally expect to be able to call functions with keyword arguments, especially when the corresponding parameters have default values, and we expect that the order of keyword arguments doesn't matter. But if the function is not implemented in Python, and instead is a built-in function implemented in C, that may not be the case. Let's look at the built-in function math.tan. On Python 3, if we ask for the function signature, we get something like this:

>>> import math
>>> help(math.tan)
Help on built-in function tan in module math:
<BLANKLINE>
...
    tan(x)
<BLANKLINE>
    Return the tangent of x (measured in radians).
<BLANKLINE>

That sure looks like a parameter with a name, so we expect to be able to call it with a keyword argument:

>>> math.tan(x=1)
Traceback (most recent call last):
...
TypeError: tan() takes no keyword arguments

This is due to how C functions bind Python arguments into C variables, using functions like PyArg_ParseTuple.

In newer versions of Python, this is indicated with a trailing / in the function signature, showing that the preceding arguments are positional only paramaters. (Note that Python has no syntax for this.)

>>> help(abs)
Help on built-in function abs in module builtins:
<BLANKLINE>
abs(x, /)
    Return the absolute value of the argument.

Python 3 Improvements

Python 3 offers ways to reduce some of these surprising characteristics. (For backwards compatibility, it doesn't actually eliminate any of them.)

We've already seen that functions implemented in C can use new syntax in their function signatures to signify positional-only arguments. Plus, more C functions can accept keyword arguments for any arbitrary parameter thanks to new functions and the use of tools like Argument Clinic.

The most important improvements, though, are available to Python functions and are outlined in the confusingly named PEP 3102: Keyword-Only Arguments.

With this PEP, functions are allowed to define parameters that can only be filled by keyword arguments. In addition, this allows functions to accept both variable arguments and keyword arguments without raising TypeError.

This in done by simply moving the variable positional parameter before any parameters that should only be allowed by keyword:

>>> def test(a, *args, b=42):
...    return (a, b, args)
>>> test(1, 2, 3)
(1, 42, (2, 3))
>>> test(1, 2, 3, b='b')
(1, 'b', (2, 3))
>>> test(1, 2, 3, b='b', c='c')
Traceback (most recent call last):
...
TypeError: test() got an unexpected keyword argument 'c'
>>> test()
Traceback (most recent call last):
...
TypeError: test() missing 1 required positional argument: 'a'

What if you don't want to allow arbitrary unnamed arguments? In that case, simply omit the variable argument parameter name:

>>> def test(a, *, b=42):
...    return (a, b)
>>> test(1, b='b')
(1, 'b')

Trying to pass extra arguments will fail:

>>> test(1, 2, b='b')
Traceback (most recent call last):
...
TypeError: test() takes 1 positional argument but 2 positional arguments (and 1 keyword-only argument) were given

Finally, what if you want to require certain parameters to be passed by name? In that case, you can simply leave off the default value for the keyword-only parameters:

>>> def test(a, *, b):
...    return (a, b)
>>> test(1)
Traceback (most recent call last):
...
TypeError: test() missing 1 required keyword-only argument: 'b'
>>> test(1, b='b')
(1, 'b')

The above examples all produce SyntaxError on Python 2. Much of the functionality can be achieved on Python 2 using variable arguments and variable keyword arguments and manual argument binding, but that's slower and uglier than what's available on Python 3. Lets look at an example of implementing the first function from this section in Python 2:

>>> def test(*args, **kwargs):
...    "test(a, *args, b=42) -> tuple" # docstring signature for Sphinx
...    # This raises an IndexError instead of a TypeError if 'a'
...    # is missing; that's easy to fix, but it's a programmer error
...    a = args[0]
...    args = args[1:]
...    b = kwargs.pop('b', 42)
...    if kwargs:
...        raise TypeError("Got extra keyword args %s" % (list(kwargs)))
...    return (a, b, args)
>>> test(1, 2, 3)
(1, 42, (2, 3))
>>> test(1, 2, 3, b='b')
(1, 'b', (2, 3))
>>> test(1, 2, 3, b='b', c='c')
Traceback (most recent call last):
...
TypeError: Got extra keyword args ['c']
>>> test()
Traceback (most recent call last):
...
IndexError: tuple index out of range

Getting Started With Python inside PostgreSQL

Jason Madden — Fri, 21 Sep 2018 15:11:07 GMT

PostgreSQL uses the tagline "the world's most advanced open source relational database." For PostgreSQL, part of being "advanced" means supporting multiple server-side procedural languages, both built-in and provided by third parties. Luckily for us here at the blog, one of the built-in languages is Python. Unluckily, it's not completely obvious how to get started using Python inside PostgreSQL. This post will provide a short walkthrough demonstrating how to do that. It's written for macOS, but many of the steps are generic.

Contents

Server Programming Basics

Relational database servers often support user-defined functions and/or procedures. In the best implementations, they can be used whenever a built-in function or procedure can be used. For example, they might be used in a SQL SELECT statement to compute values for a column in a result set, or they might be executed automatically as a trigger when data in a table changes, or they might even be called directly by a client to perform some complex action.

The database servers that support user-defined functions and procedures tend to offer a custom language to write them in. These languages tend to be somewhat similar to SQL. While that can ease writing queries and updates, it can also lead to baroque code when the logic gets more complicated.

In PostgreSQL, that SQL-derived language is called PL/pgSQL, in Oracle database it's called PL/SQL, and in Microsoft SQL server, it's called T-SQL. MySQL doesn't seem to give a name to its dialect, simply referring to SQL Compound-Statements. All of these languages are very different from one another (although PL/pgSQL and PL/SQL share some similarities).

PostgreSQL goes beyond that, using its extension mechanism to support multiple server-side procedural languages. These include:

Installing PostgreSQL

The first step to working with any of those procedural languages is to install PostgreSQL. Because all of these languages are extensions, and thus optional, we need to also compile the extensions we wish to use. Here we'll be using MacPorts to install PostgreSQL 10 with support for Python:

$ sudo port install postgresql10 +python
--->  Computing dependencies for postgresql10
--->  Fetching archive for postgresql10
--->  Some of the ports you installed have notes:
  postgresql10 has the following notes:
    To use the postgresql server, install the postgresql10-server port

We want to use the server, not just the client, so we follow the instructions to install the server:

$ sudo port install postgresql10-server
--->  Computing dependencies for postgresql10-server
--->  Fetching archive for postgresql10-server
...

--->  Some of the ports you installed have notes:
  postgresql10-server has the following notes:
    To create a database instance, after install do
      sudo mkdir -p /opt/local/var/db/postgresql10/defaultdb
      sudo chown postgres:postgres /opt/local/var/db/postgresql10/defaultdb
      sudo su postgres -c 'cd /opt/local/var/db/postgresql10 && /opt/local/lib/postgresql10/bin/initdb -D /opt/local/var/db/postgresql10/defaultdb'

We are prompted to create the initial database, so we follow the instructions there too:

$ sudo mkdir -p /opt/local/var/db/postgresql10/defaultdb
$ sudo chown postgres:postgres /opt/local/var/db/postgresql10/defaultdb
$ sudo su postgres -c 'cd /opt/local/var/db/postgresql10 && /opt/local/lib/postgresql10/bin/initdb -D /opt/local/var/db/postgresql10/defaultdb'
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
...

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

     /opt/local/lib/postgresql10/bin/pg_ctl -D /opt/local/var/db/postgresql10/defaultdb -l logfile start

The instructions at the end for starting the database server will spin it up in the background and write logging information to the file logfile. For the purposes of debugging and understanding, I prefer to run the server in the foreground, like so:

$ sudo -u postgres /opt/local/lib/postgresql10/bin/postgres -D /opt/local/var/db/postgresql10/defaultdb/
2018-09-21 07:48:14.363 CDT [18023] LOG:  listening on IPv6 address "::1", port 5432
2018-09-21 07:48:14.364 CDT [18023] LOG:  listening on IPv6 address "fe80::1%lo0", port 5432
2018-09-21 07:48:14.364 CDT [18023] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2018-09-21 07:48:14.364 CDT [18023] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-09-21 07:48:14.381 CDT [18024] LOG:  database system was shut down at 2018-09-21 07:46:28 CDT
2018-09-21 07:48:14.384 CDT [18023] LOG:  database system is ready to accept connections
...

We now leave that program running in its own terminal window where we can watch its output as we proceed. Everything else from now on will be done in a new terminal window.

Creating Users

Now that the server is running, we're ready to connect a client and start experimenting. We'll be using the stock psql interactive terminal that comes with PostgreSQL. MacPorts installs that as /opt/local/bin/psql10. (pgcli is an excellent alternative shell.)

$ psql10
psql10: FATAL:  role "jmadden" does not exist

When we ran initdb, we were warned that trust authentication would be used for local connections. What this means is that our local macOS user name would be used to connect to the server. My local user name is 'jmadden', and there is no matching user on the server (users and roles in PostgreSQL are essentially interchangeable), so we need to create one. PostgreSQL ships a createuser command for this purpose, but it's more instructive to look under the hood to see what this does by actually issuing the commands ourself.

We do this by connecting as the only user that's pre-existing and can login, the postgres super-user, and creating the role:

$ sudo -u postgres psql10
psql10 (10.5)
Type "help" for help.

postgres=# create role jmadden;
CREATE ROLE

We're not done yet, we need to grant login rights to that role:

postgres=# \du
                                List of roles
Role name |                         Attributes                         | Member of
-----------+------------------------------------------------------------+-----------
jmadden   | Cannot login                                               | {}
postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

postgres=# alter role jmadden with createdb login createrole;
ALTER ROLE

postgres=# \du
                                List of roles
Role name |                         Attributes                         | Member of
-----------+------------------------------------------------------------+-----------
jmadden   | Create role, Create DB                                     | {}
postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

In psql, \du is a shortcut command for "display users". In PostgreSQL, almost all information about database objects is kept in system tables, where it can be read and manipulated with SQL. In this case, the table we're updating is the pg_roles table:

postgres=# select * from pg_roles;
      rolname        | rolsuper | rolinherit | rolcreaterole | rolcreatedb | rolcanlogin | rolreplication | rolconnlimit | rolpassword | rolvaliduntil | rolbypassrls | rolconfig |  oid
----------------------+----------+------------+---------------+-------------+-------------+----------------+--------------+-------------+---------------+--------------+-----------+-------
pg_signal_backend    | f        | t          | f             | f           | f           | f              |           -1 | ********    |               | f            |           |  4200
postgres             | t        | t          | t             | t           | t           | t              |           -1 | ********    |               | t            |           |    10
pg_read_all_stats    | f        | t          | f             | f           | f           | f              |           -1 | ********    |               | f            |           |  3375
pg_monitor           | f        | t          | f             | f           | f           | f              |           -1 | ********    |               | f            |           |  3373
jmadden              | f        | t          | t             | t           | t           | f              |           -1 | ********    |               | f            |           | 16384
pg_read_all_settings | f        | t          | f             | f           | f           | f              |           -1 | ********    |               | f            |           |  3374
pg_stat_scan_tables  | f        | t          | f             | f           | f           | f              |           -1 | ********    |               | f            |           |  3377
(7 rows)

Now that we can login, surely we can connect:

$ psql10
psql10: FATAL:  database "jmadden" does not exist

Nope! By default, PostgreSQL will connect our session to a database matching our user name. That database doesn't exist yet. It's a good idea to create one to hold our experiments in anyway, as opposed to using one of the existing databases (to which we probably don't have access rights anyway). We'll use the createdb command to create this database and then try to login again:

$ /opt/local/lib/postgresql10/bin/createdb jmadden
$ psql10
psql10 (10.5)
Type "help" for help.

jmadden=> \l
                               List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 jmadden   | jmadden  | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)

Tip

Notice that the psql prompt is different for regular users and the superuser; regular users have => in the prompt, while the super user has =#.

Creating Languages

Fantastic! Now we're logged in to our own database. Let's create the first example Python function:

jmadden=> CREATE FUNCTION pymax (a integer, b integer)
          RETURNS integer
          AS $$
             if a > b:
                 return a
             return b
          $$ LANGUAGE plpythonu;
ERROR:  language "plpythonu" does not exist
HINT:  Use CREATE LANGUAGE to load the language into the database.

Well that didn't work. If we back up to the introduction to PL/Python, it says we need to create the extension first (the "PL" means "procedural language"; we'll see what the "u" means in a little bit):

jmadden=> create extension plpythonu;
ERROR:  permission denied to create extension "plpythonu"
HINT:  Must be superuser to create this extension.

Hmm, ok, I suppose that makes sense. Lets use our superuser login to create the extension:

$ sudo -u postgres psql10
psql10 (10.5)
Type "help" for help.

postgres=# create extension plpythonu;
CREATE EXTENSION

Ok, extension created. Now lets go back to our own database and create that example function:

jmadden=> CREATE FUNCTION pymax (a integer, b integer)
          RETURNS integer
          AS $$
             if a > b:
                 return a
             return b
          $$ LANGUAGE plpythonu;
ERROR:  language "plpythonu" does not exist
HINT:  Use CREATE LANGUAGE to load the language into the database.

Same error! Argh!

After examining the documentation a bit, it turns out that extensions are local to particular databases. So we need to use the 'jmadden' database as the superuser 'postgres' to create the extension in that database:

$ sudo -u postgres psql10 jmadden
psql10 (10.5)
Type "help" for help.

jmadden=# create extension plpythonu;
CREATE EXTENSION

Untrusted Languages

Surely now we can create that simple example function:

jmadden=> CREATE FUNCTION pymax (a integer, b integer)
          RETURNS integer
          AS $$
             if a > b:
                 return a
             return b
          $$ LANGUAGE plpythonu;
ERROR:  permission denied for language plpythonu

At least we got a different error this time.

It turns out that the "u" in "plpythonu" means that the language is "untrusted." I'll quote the docs for what that means:

PL/Python is only available as an “untrusted” language, meaning it does not offer any way of restricting what users can do in it and is therefore named plpythonu. The writer of a function in untrusted PL/Python must take care that the function cannot be used to do anything unwanted, since it will be able to do anything that could be done by a user logged in as the database administrator. Only superusers can create functions in untrusted languages such as plpythonu.

The safest way to work with untrusted languages is to only temporarily escalate your privileges by using a superuser role to create "safe" functions, as outlined in this StackExchange post. But that's no fun. Since we're only experimenting on our own machine, and we don't want to have superuser priviliges if we can avoid it (to limit the risks of accidentally damaging our database), we have another option: we can mark the language as trusted.

To do this, we'll need our superuser shell again:

postgres=# select * from pg_language;
  lanname  | lanowner | lanispl | lanpltrusted | lanplcallfoid | laninline | lanvalidator | lanacl
-----------+----------+---------+--------------+---------------+-----------+--------------+--------
 internal  |       10 | f       | f            |             0 |         0 |         2246 |
 c         |       10 | f       | f            |             0 |         0 |         2247 |
 sql       |       10 | f       | t            |             0 |         0 |         2248 |
 plpgsql   |       10 | t       | t            |         12545 |     12546 |        12547 |
 plpythonu |       10 | t       | f            |         16416 |     16417 |        16418 |
(5 rows)

postgres=# update pg_language set lanpltrusted = true where lanname = 'plpythonu';
UPDATE 1

Now our regular user can create and execute this function:

jmadden=> CREATE FUNCTION pymax (a integer, b integer)
          RETURNS integer
          AS $$
             if a > b:
                 return a
             return b
          $$ LANGUAGE plpythonu;
CREATE FUNCTION

jmadden=> select pymax(1, 2);
 pymax
-------
   2
(1 row)

PL/Python can do much more than just compare integers, of course. It can query the database, processing rows in batches or iteratively. It can access and produce PostgreSQL "compound types." It can be used for row or statement level triggers. That's all beyond the scope of this post, but you can read more about it in the PostgreSQL documentation.

CPython vs PyPy Memory Usage

Jason Madden — Wed, 08 Aug 2018 13:22:07 GMT

If you have lots of "small" objects in a Python program (objects which have few instance attributes), you may find that the object overhead starts to become considerable. The common wisdom says that to reduce this in CPython you need to re-define the classes to use __slots__, eliminating the attribute dictionary. But this comes with the downsides of limiting flexibility and eliminating the use of class defaults. Would it surprise you to learn that PyPy can significantly, and without any effort by the programmer, reduce that overhead automatically?

Let's take a look.

Contents

Contrary to advice, instead of starting at the very beginning, we'll jump right to the end. The following graph shows the peak memory usage of the example program we'll be talking about in this post across seven different Python implementations: PyPy2 v6.0, PyPy3 v6.0, CPython 2.7.15, 3.4.9, 3.5.6, 3.6.6, and 3.7.0 [1].

For regular objects ("Point3D"), PyPy needs less than 700MB to create 10,000,000, where CPython 2.7 needs almost 3.5 GB, and CPython 3.x needs between 1.5 and 2.1 GB [6]. Moving to __slots__ ("Point3DSlot") brings the CPython overhead closer to—but still higher than—that of PyPy. In particular, note that the PyPy memory usage is essentially the same whether or not slots are used.

The third group of data is the same as the second group, except instead of using small integers that should be in the CPython internal integer object cache [7], I used larger numbers that shouldn't be cached. This is just an interesting data point showing the allocation of three times as many objects, and won't be discussed further.

What Is Being Measured

In the script I used to produce these numbers [2], I'm using the excellent psutil library's Process.memory_info to record the "unique set size" ("the memory which is unique to a process and which would be freed if the process was terminated right now") before and then after allocating a large number of objects.

def check(klass, x, y, z):
    before = get_memory().uss
    inst = klass(0, x, y, z)
    print("Size of", type(inst).__name__, sizeof_object(inst))
    del inst

    print("Count      AbsoluteUsage     Delta")
    print("=======================================")

    for count in 100, 1000, 10000, 100000, 1000000, 10000000:
        l = [None] * count

        for i in range(count):
            l[i] = klass(i, x, y, z)

        after = get_memory().uss
        print("%9d" % count, format_memory(after - global_starting_memory.uss),
              format_memory(after - before))
        l = None
    print()

This gives us a fairly accurate idea of how much memory the processes needed to allocate from the operating system to be able to create all the objects we asked for. (get_memory is a helper function that runs the garbage collector to be sure we have the most stable numbers.)

What Is Not Being Measured

In this example output from a run of PyPy, the AbsoluteUsage is the total growth from when the program started, while the Delta is the growth just within this function.

Memory for Point3D(1, 2, 3)
Count      AbsoluteUsage     Delta
=======================================
      100           0.02           0.02
     1000           0.03           0.03
    10000           0.51           0.51
   100000           7.20           7.20
  1000000          69.11          69.11
 10000000         691.90         691.90

This was the first of the test runs within this particular process. The second test run within this process reports higher absolute deltas since the beginning of the program, although the overall deltas are smaller. This indicates how much memory the program has allocated from the operating system but not returned to it, even though it may technically free from the standpoint of the Python runtime; this accounts for things like internal caches, or in PyPy's case, jitted code.

Memory for Point3DSlot(1, 2, 3)
Size of Point3DSlot -1
Count      AbsoluteUsage     Delta
=======================================
      100          86.09           0.00
     1000          86.12           0.03
    10000          86.56           0.46
   100000          87.33           1.23
  1000000         138.70          52.60
 10000000         692.05         605.95

Although I captured the data, this post is not about the startup or initial memory allocation of the various interpreters, nor about how much can easily be shared between forked processes, nor about how much memory is returned to the operating system while the process is still running. We're only talking about the memory size needed to allocate a given number of objects, e.g., the Delta column.

Object Internals

To understand what's happening, let's look at the two types of objects we're comparing:

class Point3D(object):

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z


class Point3DSlot(object):
    __slots__ = ('x', 'y', 'z')

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

These are both small classes with three instance attributes. One is a standard, default, object, and one specifies its instance attributes in __slots__.

Objects with Dictionaries

Standard objects, like Point3D, have a special attribute __dict__, that is a normal Python dictionary object that is used to hold all the instance attributes for the object. We previously looked at how __getattribute__ can be used to customize all attribute reads for an object; likewise, __setattr__ can customize all attribute writes. The default __getattribute__ and __setattr__ that a class inherits from object function something like they were written to access the __dict__:

class Object:

    def __getattribute__(self, name):
        if name in self.__dict__:
             return self.__dict__[name]
        return getattr(type(self), name)

    def __setattr__(self, name, value):
        self.__dict__[name] = value

One advantage of having a __dict__ underlying an object is the flexibility it provides: you don't have to pre-declare your attributes for every object, and any object can have any attribute, so it facilitates subclasses adding new attributes, or even other libraries adding new, specialized, attributes to implement caching of expensive computed properties.

One disadvantage is that a __dict__ is a generic Python dictionary, not specialized at all [3], and as such it has overhead.

On CPython, we can ask the interpreter how much memory any given object uses with sys.getsizeof. On my machine under a 64-bit CPython 2.7.15, a bare object takes 16 bytes, while a trivial subclass takes a full 64 bytes (due to the overhead of being tracked by the garbage collector):

>>> import sys
>>> sys.getsizeof(object())
16
>>> class TrivialSubclass(object): pass
>>> sys.getsizeof(TriviasSubclass())
64

An empty dict occupies 280 bytes:

>>> sys.getsizeof({})
280

And so when you combine the size of the trivial subclass, with the size of its __dict__ you arrive at a minimum object size of 344 bytes:

>>> sys.getsizeof(TrivialSubclass().__dict__)
280
>>> sys.getsizeof(TrivialSubclass()) + sys.getsizeof(TrivialSubclass().__dict__)
344

A fully occupied Point3D object is also 344 bytes:

>>> pd = Point3D(1, 2, 3)
>>> sys.getsizeof(pd) + sys.getsizeof(pd.__dict__)
344

Because of the way dictionaries are implemented [8], there's always a little spare room for extra attributes. We don't find a jump in size until we've added three more attributes:

>>> pd.a = 1
>>> sys.getsizeof(pd) + sys.getsizeof(pd.__dict__)
344
>>> pd.b = 1
>>> sys.getsizeof(pd) + sys.getsizeof(pd.__dict__)
344
>>> pd.c = 1
>>> sys.getsizeof(pd) + sys.getsizeof(pd.__dict__)
1112

Note

These values can change quite a bit across Python versions, typically improving over time. In CPython 3.4 and 3.5, getsizeof({}) returns 288, while it returns 240 in both 3.6 and 3.7. In addition, getsizeof(pd.__dict__) returns 96 and 112 [4]. The answer to getsizeof(pd) is 56 in all four versions.

Objects With Slots

Objects with a __slots__ declaration, like Point3DSlot do not have a __dict__ by default. The documentation notes that this can be a space savings. Indeed, on CPython 2.7, a Point3DSlot has a size of only 72 bytes, only one full pointer larger than a trivial subclass (when we do not factor in the __dict__):

>>> pds = Point3DSlot(1, 2, 3)
>>> sys.getsizeof(pds)
72

If they don't have an instance dictionary, where do they store their attributes? And why, if Point3DSlot has three defined attributes, is it only one pointer larger than Point3D?

Slots, like @property, @classmethod and @staticmethod, are implemented using descriptors. For our purpose, descriptors are a way to extend the workings of __getattribute__ and friends. A descriptor is an object whose type implements a __get__ method, and when that object is found in a type's dictionary, it is called instead of checking the __dict__. Something like this [5]:

class Object:

    def __getattribute__(self, name):
        if (name in dir(type(self))
           and hasattr(getattr(type(self), name), '__get__')):
           return getattr(type(self), name).__get__(self, type(self))

        if name in self.__dict__:
             return self.__dict__[name]
        return getattr(type(self), name)

    def __setattr__(self, name, value):
        if (name in dir(type(self))
           and hasattr(getattr(type(self), name), '__set__')):
           getattr(type(self), name).__set__(self, type(self))
           return
        self.__dict__[name] = value

When the class statement (indeed, when the type metaclass) finds __slots__ in the class body (the class dictionary), it takes special steps. Most importantly, it creates a descriptor for each mentioned slot and places it in the class's __dict__. So our Point3DSlot class gets three such descriptors:

>>> dict(Point3DSlot.__dict__)
{'__doc__': None,
'__init__': <function __main__.__init__>,
'__module__': '__main__',
'__slots__': ('x', 'y', 'z'),
'x': <member 'x' of 'Point3DSlot' objects>,
'y': <member 'y' of 'Point3DSlot' objects>,
'z': <member 'z' of 'Point3DSlot' objects>}
>>> pds.x
1
>>> Point3DSlot.x.__get__
>>> <method-wrapper '__get__' of member_descriptor object at 0x10b6fc2d8>
>>> Point3DSlot.x.__get__(pds, Point3DSlot)
1

Variable Storage

We've established how we can access these magic, hidden slotted attributes (through the descriptor protocol). (We've also established why we can't have defaults for slotted attributes in the class.) But we still haven't found out where they are stored. If they're not in a dictionary, where are they?

The answer is that they're stored directly in the object itself. Every type has a member called tp_basicsize, exposed to Python as __basicsize__. When the interpreter allocates an object, it allocates __basicsize__ bytes for it (every object has a minimum basic size, the size of object). The type metaclass arranges for __basicsize__ to be big enough to hold (a pointer to) each of the slotted attributes, which are kept in memory immediately after the data for the basic object . The descriptor for each attribute, then, just does some pointer arithmetic off of self to read and write the value. In a way, it's very similar to how collections.namedtuple works, except using pointers instead of indices.

That may be hard to follow, so here's an example.

The basic size of object exactly matches the reported size of its instances:

>>> object.__basicsize__
16
>>> sys.getsizeof(object())
16

We get the same when we create an object that cannot have any instance variables, and hence does not need to be tracked by the garbage collector:

>>> class NoSlots(object):
...     __slots__ = ()
...
>>> NoSlots.__basicsize__
16
>>> sys.getsizeof(NoSlots())
16

When we add one slot to an object, its basic size increases by one pointer (8 bytes), and because that object could be tracked by the garbage collector, this object needs to be tracked by the collector, so getsizeof reports some extra overhead:

>>> class OneSlot(object):
...     __slots__ = ('a',)
...
>>> OneSlot.__basicsize__
24
>>> sys.getsizeof(OneSlot())
56

The basic size for an object with 3 slots is 16 (the size of object) + 3 pointers, or 40. What's the basic size for an object that has a __dict__?

>>> Point3DSlot.__basicsize__
40
>>> Point.__basicsize__
32

Hmm, it's 16 + 2 pointers. What could those two pointers be? Documentation to the rescue:

__slots__ allow us to explicitly declare data members (like properties) and deny the creation of __dict__ and __weakref__ (unless explicitly declared in __slots__...)

So those two pointers are for __dict__ and __weakref__, things that standard objects get automatically, but which we have to opt-in to if we want them with __slots__. Thus, an object with three slots is one pointer size bigger than a standard object.

How PyPy Does Better

By now we should understand why the memory usage dropped significantly when we added __slots__ to our objects on CPython (although that comes with a cost). That leaves the question: how does PyPy get such good memory performance with a __dict__ that __slots__ doesn't even matter?

Earlier I wrote that the __dict__ of an instance is just a standard dictionary, not specialized at all. That's basically true on CPython, but it's not at all true on PyPy. PyPy basically fakes __dict__ by using __slots__ for all objects.

A given set of attributes (such as our "x", "y", "z" attributes for Point3DSlot) is called a "map". Each instance refers to its map, which tells PyPy how to efficiently access a given attribute. When an attribute is added or deleted, a new map is created (or re-used from an existing object; objects of completely unrelated types, but having common attributes can share the same maps) and assigned to the object, re-arranging things as needed. It's as if __slots__ was assigned to each instance, with descriptors added and removed for the instance on the fly.

If the program ever directly accesses an instance's __dict__, PyPy creates a thin wrapper object that operates on the object's map.

So for a program that has many simalar looking objects, even if unrelated, PyPy's approach can save a lot of memory. On the other hand, if the program creates objects that have a very diverse set of attributes, and that program frequently directly accessess __dict__, it's theoretically possible that PyPy could use more memory than CPython.

You can read more about this approach in this PyPy blog post.

Footnotes

[1]	All 64-bit builds, all tested on macOS. The results on Linux were very similar.

[2]	Available at this gist.

[3]	In CPython. But I'm getting ahead of myself.

[4]	The CPython dict implementation was completely overhauled in CPython 3.6. And based on the sizes of `{}` versus `pd.__dict__` we can see some sort of specialization for instance dictionaries, at least in terms of their fill factor.

[5]	This is very rough, and actually inaccurate in some small but important details. Refer to the documentation for the full protocol.

[6]	No, I'm not totally sure why Python 3.7 is such an outlier and uses more memory than the other Python 3.x versions.

[7]	See PyLong_FromLong

[8]	With a particular desired load factor.

Class Warfare in Python 2

Jason Madden — Sat, 28 Jul 2018 14:17:07 GMT

In Python 2, there are two distinct types of classes, the so-called "old" (or "classic") and (increasingly inaccurately named) "new" style classes. This post will discuss the differences between the two as visible to the Python programmer, and explore a little bit of the implementation decisions behind those differences.

Contents

New Style Classes

New style classes were introduced in Python 2.2 back in December of 2001. [1] New style classes are most familiar to modern Python programmers and are the only kind of classes available in Python 3.

Let's look at some of the observable properties of new style classes.

>>> class NewStyle(object):
...    pass

In Python 2, we create new style classes by (explicitly or implicitly) inheriting from object [2]. This means that the class NewStyle is an instance of type, that is, type is the metaclass (alternately, we can say that any instance of type is a new style class):

>>> NewStyle
<class 'NewStyle'>
>>> isinstance(NewStyle, type)
True
>>> type(NewStyle)
<type 'type'>

Instances of this class are, well, instances of this class, and their type is this class:

>>> new_style_instance = NewStyle()
>>> type(new_style_instance) is NewStyle
True
>>> isinstance(new_style_instance, NewStyle)
True

Unsurprisingly, instances of NewStyle classes are also instances of object; it's right there in the class definition and the class __bases__ and the method resolution order (MRO):

>>> isinstance(new_style_instance, object)
True
>>> NewStyle.__bases__
(<type 'object'>,)
>>> NewStyle.mro()
[<class 'NewStyle'>, <type 'object'>]

The default repr of a new style instance includes its type's name:

>>> print(repr(new_style_instance))
<NewStyle object at 0x...>

Old Style Classes

If that's a new style class, what's an old style class? Old style, or "classic" classes, were the only type of class available in Python 2.1 and earlier. For backwards compatibility when moving older code to Python 2.2 or later, they remained the default (we'll see shortly some of the practical behaviour differences between old and new style classes).

This means that we create an old style class by not specifying any ancestors. We also create old style classes when only specifying ancestors that themselves are old style classes.

>>> class OldStyle:
...    pass

Let's look at the same observable properties as we did with new style classes and compare them. We've already seen that we didn't specify object in the inheritance list. Is this class a type?

>>> OldStyle
<class __builtin__.OldStyle at 0x...>
>>> isinstance(OldStyle, type)
False
>>> type(OldStyle)
<type 'classobj'>

If you've been paying attention, you're not surprised to see that old style classes are not instances of type (after all, that's the definition we used for new style classes). They have a different repr, which is not too surprising, but they also have a different type.

>>> type(type(OldStyle))
<type 'type'>

Hmm, and the type of that different type (i.e., its metaclass) is type. That makes me wonder, do all old-style classes share a type? Let's continue exploring.

Instances of the old style class are of course still instances of the old style class, but their type is not that class, unlike with new style classes:

>>> old_style_instance = OldStyle()
>>> isinstance(old_style_instance, OldStyle)
True
>>> type(old_style_instance) is OldStyle
False
>>> type(old_style_instance)
<type 'instance'>

Instead, their type is something called instance. Now I'm wondering again, do all instances of old style classes share a type? If so, how does class-specific behaviour get implemented?

>>> class OldStyle2:
...    pass
>>> old_style2_instance = OldStyle2()
>>> type(old_style2_instance)
<type 'instance'>
>>> type(old_style2_instance) is type(old_style_instance)
True

They do! Curiouser and curiouser. How is type-specific behaviour implemented then? Let's keep going.

We saw that new style classes are instances of object, and it exists in their __bases__ and MRO. What about old classes?

>>> isinstance(old_style_instance, object)
True
>>> OldStyle.__bases__
()
>>> OldStyle.mro()
Traceback (most recent call last):
...
AttributeError: class OldStyle has no attribute 'mro'

Strangely, the old style instance is still an instance of object, even though it doesn't inherit from it, as seen by it not being in the __bases__ (and it doesn't even have the concept of an exposed MRO).

Finally, the repr of old style classes is also different:

>>> print(repr(old_style_instance))
<__builtin__.OldStyle instance at 0x...>

Practically Speaking

OK, that's enough time spent down in the weeds dealing with subtle differences in inheritance bases and reprs. Before we talk about implementation, are there any practical differences a Python programmer should be concerned about, or at least know about?

Yes. I'll hit some of the highlights here, there are probably others. (TL;DR: Always write new-style classes.)

Operators

For new style classes, all the Python special "dunder" methods (like __repr__ and the arithmetic and comparison operators such as __add__ and __eq__) must be defined on the class itself. If we want to change the repr, we can't do it in the instance, we have to do so on the class:

>>> new_style_instance.__repr__ = lambda: "Hi, I'm a new instance"
>>> print(repr(new_style_instance))
<NewStyle object at 0x...>
>>> NewStyle.__repr__ = lambda self: "Hi, I'm the new class"
>>> print(repr(new_style_instance))
Hi, I'm the new class

(We monkey-patched a class after its definition here, which does work thanks to some magic well discuss soon, but that's not usually the way its done.)

In contrast, old style classes allowed dunders to be assigned to an instance, even overriding one set on the class, in exactly the same way that regular methods work on both new and old style classes.

>>> OldStyle.__repr__ = lambda self: "Hi, I'm the old class"
>>> print(repr(old_style_instance))
Hi, I'm the old class
>>> old_style_instance.__repr__ = lambda: "Hi, I'm the old instance"
>>> print(repr(old_style_instance))
Hi, I'm the old instance

Old Style Instances are Bigger

On a 64-bit platform, old style instances are one pointer size larger:

>>> import sys
>>> sys.getsizeof(new_style_instance)
64
>>> sys.getsizeof(old_style_instance)
72

Old Style Classes Ignore `slots`

The slots declaration can be used not only to optimize memory usage for small frequently used objects but offer tighter control on what attributes an instance can have, which may be useful in some scenarios. Unfortunately, old style classes accept but ignore this declaration.

>>> class NewWithSlots(object):
...      __slots__ = ('attr',)
>>> new_with_slots = NewWithSlots()
>>> sys.getsizeof(new_with_slots)
56
>>> new_with_slots.not_a_slot = 42
Traceback (most recent call last):
...
AttributeError: 'NewWithSlots' object has no attribute 'not_a_slot'
>>> new_with_slots.attr = 42
>>> sys.getsizeof(new_with_slots)
56

>>> class OldWithSlots:
...      __slots__ = ('attr',)
>>> old_with_slots = OldWithSlots()
>>> sys.getsizeof(old_with_slots)
72
>>> old_with_slots.not_a_slot = 42
>>> sys.getsizeof(old_with_slots)
72

Old Style Classes Ignore `getattribute`

The getattribute dunder method allows an instance to intercept all attribute access. It's what makes it possible to implement transparently persistent objects in pure-Python. Indeed, it's the same basic feature that allowed for descriptors to be implemented in Python 2.2. Old style classes only support __getattr__, which is only used when the attribute cannot be found.

>>> class NewWithCustomAttrs(object):
...   def __getattribute__(self, name):
...       print("Looking up %s" % name)
...       if name == 'computed': return 'computed'
...       return object.__getattribute__(self, name)
...   def __getattr__(self, name):
...       print("Returning default for %s" % name)
...       return 1
>>> custom_new = NewWithCustomAttrs()
>>> custom_new.foo
Looking up foo
Returning default for foo
1
>>> custom_new.foo = 42
>>> custom_new.foo
Looking up foo
42
>>> custom_new.computed = 42
>>> custom_new.computed
Looking up computed
'computed'

>>> class OldWithCustomAttrs:
...   def __getattribute__(self, name):
...       print("Looking up %s" % name)
...       if name == 'computed': return 'computed'
...       return object.__getattribute__(self, name)
...   def __getattr__(self, name):
...       print("Returning default for %s" % name)
...       return 1
>>> custom_old = OldWithCustomAttrs()
>>> custom_old.foo
Returning default for foo
1
>>> custom_old.foo = 42
>>> custom_old.foo
42
>>> custom_old.computed = 42
>>> custom_old.computed
42

Contrary to some information that can be found on the Internet, old style classes can use most descriptors, including @property.

Old Style Classes Have a Simplistic MRO

When multiple inheritance is involved, the MRO of an old style class can be quite surprising. I'll let Guido provide the explanation.

Old Style Classes are Slow

This is especially true on PyPy, but even on CPython 2.7 an old style class is three times slower at basic arithmetic.

>>> import timeit
>>> class NewAdd(object):
...    def __add__(self, other):
...        return 1
>>> timeit.new_add = NewAdd() # Cheating, sneak this into the globals
>>> timeit.timeit('new_add + 1')
0.2...
>>> class OldAdd:
...    def __add__(self, other):
...        return 1
>>> timeit.old_add = OldAdd()
>>> timeit.timeit('old_add + 1')
0.6...

Implementing Old Style Classes

The shift to new style classes represents something of a shift away from Python's earlier free-wheeling days when it could almost be thought of as a prototype-based language into something more traditionally class based. Yet Python 2.2 up through Python 2.7 still support the old way of doing things. It would be silly to imagine there are two completely separate object and type hierarchy implementations inside the CPython interpreter, along with two separate ways of doing arithmetic and getting reprs, etc [3], yet the old style classes continue to work as always. How is this done?

We've had some tantalizing clues to help point us in the right direction.

Regarding instances of old style classes, even separate classes are the same type. The same is not true for new style classes.
Old style instances are still instances of object.
Regarding old style classes themselves, they are not instances of type, rather of classobj...which itself is an instance of type.
Regarding perfomance, the change to method lookup "provides significant scope for speed optimisations within the interpreter".
If you followed the link to see what was new in Python 2.2, you'd see that one of the other major features introduced by PEP 253 was the ability to subclass built-in types like list and override their operators. In Python 2.1, just as with old style classes today, all classes implemented in Python shared the same type, but classes implemented in C (like list) could be distinct types; importantly, types were not classes. Here's Python 2.1:
```
>>> class OldStyle:
...     pass
>>> class OldStyle2:
...     pass
>>> type(OldStyle())
<type 'instance'>
>>> type(OldStyle2())
<type 'instance'>
>>> type([])
<type 'list'>
>>> class Foo(list):
...     pass
Traceback (most recent call last):
...
TypeError: base is not a class object
```

To further understand how CPython implements both old and new-style classes, you need to understand just a bit about how it represents types internally and how it invokes those special dunder methods.

CPython Types

In CPython, all types are defined by a C structure called PyTypeObject. It has a series of members (confusingly also called slots) that are C function pointers that implement the behaviour defined by the type. For example, tp_repr is the slot for the __repr__ function, and tp_getattro is the slot for the __getattribute__ function. When the interpreter needs to get the repr of an object, or get an attribute from it, it always goes through those function pointers. An operation like repr(obj) becomes the C equivalent of type(obj).__repr__(obj).

When we create a new style class, the standard type metaclass makes a C function pointer out of each special dunder method defined in the class's __dict__ and uses that to fill in the slots in the C structure. Special methods that aren't present are inherited. Similarly, when we later set such an attribute, the __setattr__ function of the type metaclass checks to see if it needs to update the C slots.

The C structure that defines the object type uses PyObject_GenericGetAttr for its tp_getattro (__getattribute__) slot. That function implements the "standard" way of accessing attributes of objects. It's the same thing we showed above in old_getattribute when we called object.__getattribute__(self, name). The object type also leaves most other slots (like for __add__) empty, meaning that subclasses must define them in the class __dict__ (or assign to them) for them to be found.

Importantly, though, other types implemented in C don't have to do any of that. They could set their tp_getattro and tp_repr slots, along with any of the other slots, to do whatever they want.

You might now understand how PEP 253, which gave us new style classes, also gave us the ability to subclass C types.

instance types

That's exactly what <type 'instance'> does. That type is defined by PyInstance_Type. The metaclass classobj (recall that's the type of old style classes) sets its tp_call slot (aka __call__) to PyInstance_New, a function that returns a new object of <type 'instance'>, aka PyInstance_Type (explaining while all old instances have the same type). (Incidentally, the metaclass also returns instances of itself, not new sub-types as the type metaclass does, which explains why they all have the same type.)

PyInstance_Type, in turn, overrides almost all of the possible slot definitions to implement the old behaviour of first looking in the instance's __dict__ and only if it's not present then looking up the class hierarchy or invoking __getattr__. For example, tp_repr is instance_repr, which uses instance_getattr to dynamically find an appropriate __repr__ function from the instance or class. This, by the way, is the source of the performance difference between old and new style classes.

Implementing Old Style Classes in Python

Can we re-implement old style classes in pure-Python (for example, on Python 3)? I haven't tried, but we can probably get pretty close. It would be similar to implementing transparent object proxies in pure-Python, which is verbose and tedious.

Footnotes

[1]	Thus demonstrating the perils of naming something "new." Here we are 17 years later and still calling them that.

[2]	Or by explicitly specifying a `__metaclass__` of `type`. >>> class NewWithMeta: ... __metaclass__ = type >>> isinstance(NewWithMeta, type) True

[3]	Good programmers are lazy.

The Xerox Alto: gevent for hardware

Jason Madden — Thu, 05 Jul 2018 16:17:07 GMT

Previously I discussed the GNU Pth library which implements cooperative multi-tasking for the C programming language. That's pretty low level. But I recently read about cooperative multi-tasking at an even lower level: inside the microcode of the Xerox Alto central processor! It's hard to get much lower level than that.

The engineers at Seattle's Living Computers museum maintain a blog where they write about their experiences repairing and maintaining the historic computers that make up the museum's exhibits and fill its labs.

In a two-part series (1, 2), one of the engineers discusses the Xerox Alto, the first computer designed specifically to run a graphical user interface. Graphical user interfaces were very new at the time, so there was a lot of experimentation to do. In order to make that experimentation quick and easy, the very CPU of the Alto was itself programmable in microcode.

The second article discusses the microcode in detail. Microcoding CPUs may have been uncommon in 1973 when the Alto was created, but it's common enough today that most computer science programs will include an introduction to it, so there was nothing too surprising. Until I got to the bottom in the section called "Tasks":

To allow the devices in the Alto to share the Alto's processor with minimal overhead, the Alto's designers developed very simple cooperative task-switching hardware.

Conceptually, the processor is shared between sixteen microcode Tasks, with a priority assigned to each one: Task 0 is the lowest priority task, and Task 15 the highest. Each device in the Alto has one or more tasks dedicated to it. To make task switching fast and cheap to implement in hardware, the only state saved by the task-switching hardware for each Task is that task's program counter — the MPC ("Micro Program Counter"").

Only one task is running on the processor at a time, and at any time the microcode for a task may invoke a "task switch" function (named, strangely enough, TASK). When a TASK instruction is executed, the processor selects the highest priority task that needs to run (is "Requesting Wakeup") and switches to it by jumping to the instruction pointed to by that Task’s MPC.

Wow—greenlets implemented in hardware! I've read about a lot of CPUs, and this is the first time I remember reading about one with cooperative task-switching at the microcode level.

It's a stretch to call this gevent for hardware, of course. (For one thing, there doesn't seem to be any concept of an event loop.) But programmers still had to worry about yielding to other "greenlets" in a timely fashion:

Since the Alto’s Task system is cooperative (meaning that task switches only happen when microcode explicitly requests them) the microcode for each task must be careful to yield to other tasks every so often so that they don’t get starved of time. If the Disk Word Task is late in execution, data from the disk is corrupted. If the Display Word Task doesn’t get enough time, the display will flicker or display glitches.

Blocking gevent's Hub Part 2: Finding Blocking

Jason Madden — Wed, 27 Jun 2018 14:36:00 GMT

Last time we looked at the causes and consequences of greenlets that block and avoid switching to the hub. Many times such greenlets (really, the code they run) are obvious, but some times they are not.

This post will look at some new features that gevent 1.3 added to make finding greenlets that block easier. These can be used both during development and in production.

Contents

What It does

gevent 1.3 includes a couple features to help find code that's blocking. One of those we can turn on using just environment variables, no code changes needed (although you can configure it in code as well). This makes it especially handy for debugging on the fly.

Let's take a look at one of our blocking examples from last time:

import time
import gevent

def f():
    time.sleep(3)

now = time.time()
greenlet = gevent.spawn(f)
while not greenlet.dead:
    print('.', end='')
    gevent.sleep(1)
after = time.time()
print("\nElapsed time: %1.1f" % (after - now))

Now we'll tell gevent to let us know if it detects any greenlets blocking the hub and run the example (the duration of time that a greenlet is allowed to block before it gets reported is configurable; we'll go with the defaults here):

$ GEVENT_MONITOR_THREAD_ENABLE=true python /tmp/test.py
================================================================================

2018-06-22T19:20:10Z : Greenlet <Greenlet "Greenlet-0" at 0x1.: f> appears to be blocked
    Reported by <gevent.__tracer.GreenletTracer object at 0x...>
Blocked Stack (for thread id 0x...):
  File "/tmp/test.py", line 5, in f
    time.sleep(3)

Info:
********************************************************************************
* Threads
********************************************************************************
Thread 0x2 (MainThread)

  File "/tmp/test.py", line 5, in f
    time.sleep(3)

********************************************************************************
* Greenlets
********************************************************************************
<greenlet.greenlet object at 0x3>
 :    Parent: None
 :    Monitoring Thread:<PeriodicMonitoringThread >
<greenlet.greenlet object at 0x4>
 :    Parent: None
 +--- <Greenlet "Greenlet-0" at 0x1: f>
 :          Parent: <Hub '' at 0x5 select default pending=0 ref=1 thread_ident=0x2>
 :          Spawned at:
 :            File "/tmp/test.py", line 8, in <module>
 :              greenlet = gevent.spawn(f)
 +--- <Hub '' at 0x5 select default pending=0 ref=1 thread_ident=0x2>
            Parent: <greenlet.greenlet object at 0x4>

 .
 Elapsed time: 3.0

This time, gevent detected the hold up and printed a report about what the problem was. [1]

Testing And Asserting

Either in unittests, from particularly latency sensitive parts of your application, or when trying to further narrow down the cause of blocking, gevent provides the assert_switches() context manager. Wrap this manager around some code to be sure that it switches to another greenlet or back to the hub, optionally within a set amount of time. If nothing switches, then an exception will be raised with the same sort of information as above.

>>> from gevent.util import assert_switches
>>> # This will always raise an exception: nothing switches
>>> with assert_switches():
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "//gevent/util.py", line 592, in __exit__
    raise _FailedToSwitch('\n'.join(report_lines))
gevent.util._FailedToSwitch: ...
>>> # This will never raise an exception; nothing switched,
... # but it happened very fast
... with assert_switches(max_blocking_time=1.0):
...     pass
...
>>>

How It Works

Credit

This technique did not originate with gevent. I've seen it in this blog post from 2012 by Ryan Kelly and I've seen it in this code from Mozilla. gevent 1.3 just generalized it, made it safe for multiple threads and hubs, and incorporated it into the core of gevent.

The main question we want to be able to answer is: did the hub get to run during a certain time period? Running the hub means switching into its greenlet, so what we really want to know is: was the greenlet hub switched into?

The greenlet library provides a hook well suited for this purpose: greenlet.settrace(). This functions similarly to sys.settrace() that's part of the core interpreter. The later calls a function for every "interesting" interpreter event such as a function call, and the former calls a function for every interesting event involving greenlets.

Since greenlets can really only do two interesting things, 'switch' and 'throw' are the only two defined events—for our purposes they are equivalent as they both involve a transfer of control from one greenlet to another, and both events tell us which greenlet is going to be taking control (the "target").

So to answer our question, we only need to keep track of whether or not the greenlet hub was the target of an event and thus assumes control. For the decorator assert_switches, which has a well-defined scope and only needs to answer that question at the exact time it is exited, that's enough [2], but for the general monitoring function that lets you know if (when) it ever found the hub blocked for longer than a certain period, we need to do a little more. To be able to answer that question meaningfully, we also need to know the timestamp of the last event entering the hub so we can compare it to the current timestamp: if it's been "too long," then we can raise an alert.

Performance

gevent is a performance critical library, so monitoring features that are too perfomance sucking to enable are a non-starter. For that reason, the implementation of these features was carefully optimized and a few different tradeoffs are made. Here are some examples.

Getting the current time (time.time()) is relatively expensive, so we try to avoid doing that too often. For example, instead of checking if we've exceeded the blocking threshold on every greenlet switch, we only do so periodically in a background thread.
We use some heuristics to detect if we switched into the hub instead of absolute timestamps.
The greenlet trace functions are compiled with Cython and strongly typed. This makes the relatively complex work that the blocking monitor trace function needs to do faster than the most trivial trace function implemented in Python.

As always, if you find bugs or can think of enhancements or additions to gevent, please feel free to open an issue or create a pull request!

Footnotes

[1]

I admit I cheated here in the output a little bit. I trimmed it: it detected, and reported, the same blockage several times. It should probably recognize that and just print something about "same blockage". I should fix that. Also, I fixed up the spawning line. Things spawned at the module level do not always get the right traceback.

[2]	As long as you don't use the `max_blocking_time` parameter.

Blocking gevent's Hub Part 1: Understanding Blocking

Jason Madden — Tue, 26 Jun 2018 14:36:00 GMT

In the beginning we talked about gevent's hub and how greenlets switch in and out of it to implement IO. Following that we showed how locks in gevent are implemented much the same way, by "parking" a waiting greenlet, switching to the hub to let other greenlets run or do IO, and eventually switching back to the parked greenlet.

That's a lot of switching. What does it mean if that switching doesn't happen? What should a programmer know about switching and its opposite, blocking? (There's also part 2.)

Contents

Cooperation

gevent (and greenlets in general) is a cooperative multitasking system, meaning that any given greenlet runs until it chooses to give up control. Under gevent, choosing to give up control is usually automatic, and usually happens when the greenlet wants to handle IO, such as reading or writing a socket to handle an incoming HTTP request or HTTP response. A greenlet can also give up control automatically by waiting for a lock that another greenlet owns, or it can explicitly give up control by using the gevent.sleep() API. In all those cases, control passes to the hub (recall that the hub is the greenlet running the event loop); this is referred to as yielding to the hub. When a greenlet yields to the hub, another greenlet gets a chance to run or IO gets serviced by the event loop. A greenlet that yields to the hub is called cooperative (or sometimes green). When a greenlet doesn't yield, it's non-cooperative.

Examples

Of course, it's not actually the greenlet itself that's yielding, it's the code that the greenlet is running that yields. We can thus say that any given piece of code is cooperative or green or not. Here are some examples:

gevent.sleep() is cooperative

import time
import gevent

def f():
    gevent.sleep(3)

now = time.time()
greenlet = gevent.spawn(f)
while not greenlet.dead:
    print('.', end='')
    gevent.sleep(1)
after = time.time()
print("\nElapsed time: %1.0f" % (after - now))

On my system, that prints three dots, indicating that the greenlets are yielding to the hub and switching back and forth:

$ python test.py
...
Elapsed time: 3.0

I will omit the timing part of the main greenlet in the rest of the examples unless it changes.

time.sleep() is not cooperative

Here's that same example, with the inner greenlet using time.sleep() from the standard library.
```
import time
import gevent

def f():
    time.sleep(3)
```
```
$ python test.py
.
Elapsed time: 3.0
```
We get only one dot, indicating that as soon as the main greenlet yielded to the hub, which in turn handed control over to the greenlet running f, that greenlet failed to yield again until it was completely finished.

Code that is CPU-bound is not cooperative

If a function is in a tight computation loop, there is no opportunity for it to yield control.
```
import time
import gevent

def f():
    i = 0
    while i < 2 ** 28:
        i += 1
```
```
$ python test.py
.
Elapsed time: 28.2
```
If that had been cooperative, we would have expected to see about 28 dots, since that's how long the inner greenlet ran. We only got one.

Socket IO using gevent.socket is cooperative

import time
import gevent
import gevent.socket

def f():
    gevent.socket.gethostbyname('www.python.org')

now = time.time()
greenlet = gevent.spawn(f)
while not greenlet.dead:
    print('.', end='')
    gevent.sleep(0.001) # NOTE: We switch to a smaller sleep duration
after = time.time()
print("\nElapsed time: %1.1f" % (after - now))

Lots of dots here.

$ python test.py
.....................
Elapsed time: 0.0

Socket IO using the standard socket is not cooperative [1]

import time
import gevent
import socket

def f():
    socket.gethostbyname('www.python.org')

Again, only one dot, meaning no yielding.

$ python test.py
.
Elapsed time: 0.0

C extension libraries may or may not be cooperative

This is a tricky area. C extensions are so common in Python (much of the standard library is implemented in C) that you may not even realize you're using one. Commonly, C extensions are used for two major purposes.

The first is to accelerate CPU intensive code (for example, numpy to speed up linear algebra). These are probably no more or less cooperative than a counterpart written in Python. (I'll explain what I mean by that in a minute.)

The second is to integrate with other existing native (C) libraries, such as imaging libraries or database clients. Some of these are essentially CPU accelerators, so they basically fall into the first category. But libraries that want to do IO, especially talking to remote systems, particularly databases, are likely to want to handle IO their own way and, without special hooks, are unlikely to be cooperative. (The PostgreSQL driver and its Python C extension psycopg2 is an example of a database driver that provides the necessary hooks to be made cooperative using psycogreen.)

Back to that "no more or less cooperative than a counterpart written in Python" statement. If the C extension calls from C back into Python code, and that Python code itself is cooperative, than thanks to the magic of greenlet being able to switch C call stacks, the whole thing winds up cooperative. The call from C into Python could either be explicit, or implicit (for example, sorting or hashing values can call arbitrary Python code in an object's __eq__ and __hash__ methods).

A few good examples of CPU accelerator extensions that can operate cooperatively are the BTrees (sorted dictionaries) and zope.interface (fast adapter and component registration and lookup) libraries. These libraries need to sort or hash objects. They are commonly used with objects that are persistent and stored in a ZODB database, so sorting or hashing may need to transparently fetch the object from the database. ZODB is implemented in pure Python, so the original C operation winds up being cooperative. [2]

Synergism Matters

The point of having one greenlet periodically print dots in those examples while the other greenlet either cooperated or not was to show why cooperative code matters. If one greenlet is not cooperating, no other greenlet can make any forward progress (print dots). This might stop other greenlets for a noticeably long time (like the CPU-bound example of counting to 268,435,456, or it might be so quick it's difficult for a human to catch (like resolving a hostname). In either case, it can be a problem, especially if you're trying to scale a program such as a server to handle more connections. Every yield lost potentially decreases overall velocity.

Avoiding Blocking

Above, we saw some examples of non-cooperative code. There were some clear transformations to go through to make code cooperative:

Switch from standard library API to gevent API.

Change imports to import from gevent. E.g., import time⟹from gevent import time or import socket⟹from gevent import socket.
Monkey patch

In code that's designed to be portable to a non-gevent system, you can use the standard library API and have gevent automatically make it cooperative by monkey patching. Many third-party libraries such as requests that use only the Python standard API can become cooperative this way.
Break up CPU intensive loops by explicitly yielding

If you know you'll have a CPU intensive loop, like in the example, you can periodically explicitly yield by calling gevent.sleep() in the loop. Beginning in gevent 1.3, this attempts to automatically balance the amount of time CPU intensive tasks get while still allowing other greenlets to perform IO.

gevent also provides a threadpool you can use to offload CPU intensive tasks.

Diagnosing Blocking

You may have done all of that and are still not sure if you're blocking the hub or not: maybe a colleague added a new C extension to your project and you don't know its internals, or maybe there's an inner loop somewhere that might be CPU intensive and you're just not sure.

That's where my next post comes in. We'll discuss how to use gevent's monitoring facilities to find greenlets that aren't cooperating.

Footnotes

[1]	Unless of course you monkey patch, but that's another post. Or at least another section of this post.

[2]	At least when the system is monkey patched. But that's another post. Or at least another section of this post.