Transaction IDs in ZODB and RelStorage

Object IDs

Everything starts from the root. But where does the root object come from? Let's look at the root() method of the Connection:

def root(self):
    """Return the database root object."""
    return RootConvenience(self.get(z64))

That points us to the get() method:

def get(self, oid):
    """Return the persistent object with oid 'oid'."""
    # Simplified,  details removed
    obj = self._cache.get(oid, None)
    if obj is not None:
        return obj
    pickle_data, _ = self._storage.load(oid)
    obj = self._reader.getGhost(pickle_data)
    self._cache.new_ghost(oid, obj)
    return obj

There's a lot going on in these two little snippets, so lets unpack them, starting with get(). What can we deduce from this?

• A Connection caches objects, and when objects can't be found in the cache, it asks the "storage" object to load it. (More on the storage in a minute.)
• Persistent objects are identified by "OIDs" (short for "object identifier", or just "object ID"). Just as any Python object in memory has a unique identifier—returned by id(obj) and commonly seen in repr output as a hexadecimal number, as in <persistent.Persistent object at 0x110a973d0...>—, any persistent object stored in the ZODB has a unique identifier, its OID. An important difference is that id(obj) is only meaningful within the scope of a single process, but the ZODB OID is usable from any process connected to the same ZODB.
• You can think of the OID as (part of) the "address" of a particular persistent object in the same way that id(obj) returns the memory address of an object in CPython. The OID is used to find or store objects in the cache, and also to load the stored data for the object.
• The root object has a special, pre-defined OID known as "z64". That's our starting point for traversing the object graph.

You can't see this just from the code, but ZODB defines OIDs to be 8 byte, or 64-bit, quantities. The special value z64 is just 8 zero bytes in a row. ZODB has some utilities to convert between the 8 byte form or a 64-bit number.

>>> from ZODB.Connection import z64
>>> z64
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> conn.root()._p_oid
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> from ZODB.utils import p64, u64
>>> u64(z64)
0
>>> p64(0)
b'\x00\x00\x00\x00\x00\x00\x00\x00'

Each persistent object knows its own OID and keeps track of it in a special attribute known as _p_oid.

>>> conn.get(z64)._p_oid
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> conn.root()['obj']._p_oid
b'>.\xe6\x8c\x1a\xb9\xf8\xd3'
>>> u64(conn.root()['obj']._p_oid)
4480772168698427603

Storage Details and Referencing Objects

The get() method of a ZODB Connection returns persistent objects, but it uses the lower level "storage" interface to get the actual data stored for the object. Each ZODB Connection has a storage object, and all the Connections of a particular ZODB DB have storage objects that are reading and writing to the same shared data (e.g., a single file for FileStorage, or a single SQL database for RelStorage.) Among other things, the storage is responsible for holding the data stored for the persistent objects. This data, or state, is kept as a byte-string produced by pickle. We can use pickletools to peek at this.

>>> pickletools.dis(conn._storage.load(z64)[0])
    0: \x80 PROTO      3
    2: c    GLOBAL     'persistent.mapping PersistentMapping'
   40: q    BINPUT     0
   42: .    STOP
highest protocol among opcodes = 2
None
>>> pickletools.dis(conn._storage.load(z64)[0][43:])
    0: \x80 PROTO      3
    2: }    EMPTY_DICT
    3: q    BINPUT     1
    5: X    BINUNICODE 'data'
   14: q    BINPUT     2
   16: }    EMPTY_DICT
   17: q    BINPUT     3
   19: X    BINUNICODE 'obj'
   27: q    BINPUT     4
   29: C    SHORT_BINBYTES b'>.\xe6\x8c\x1a\xb9\xf8\xd3'
   39: q    BINPUT     5
   41: c    GLOBAL     'persistent Persistent'
   64: q    BINPUT     6
   66: \x86 TUPLE2
   67: q    BINPUT     7
   69: Q    BINPERSID
   70: s    SETITEM
   71: s    SETITEM
   72: .    STOP
   highest protocol among opcodes = 3

The important part here is the line labeled 29. Notice how that byte string matches the OID of our second persistent object. We can see that our root persistent object is referring to our other persistent object via its OID. The opcode labeled 69, BINPERSID is where ZODB hooks in to the unpickling process. By setting an attribute on the unpickler called "persistent_load", ZODB arranges to be called any time one of these OIDs is located in a pickle. ZODB can then go back to the cache or the storage to find the matching object.

Conflict Detection Through Serial Numbers

We've seen how ZODB can address objects, finding them and loading them by OID, automatically re-creating in memory the object graph we built. But that's not all ZODB does. ZODB provides the ability to modify and save objects, and to detect when more than one Connection attempts to do so at the same time, resulting in a conflict. How does this work?

To explore this, lets have both connections make a change to their version of the root object. We'll be using separate transaction managers, one for each connection, in order to do this.

>>> from transaction import TransactionManager
>>> conn1_txm = TransactionManager()
>>> conn2_txm = TransactionManager()
>>> conn.close()
None
>>> conn2.close()
None
>>> conn = db.open(conn1_txm)
>>> conn2 = db.open(conn2_txm)
>>> conn.root()['obj2'] = Persistent()
>>> conn2.root()['obj3'] = Persistent()

If we commit the changes from one transaction, everything goes as planned:

>>> conn1_txm.commit()
None

But trying to commit the other transaction fails because the object has already been modified in an incompatible way:

>>> conn2_txm.commit()
Traceback (most recent call last):
...
   raise ConflictError(oid=oid, serials=(committedSerial, oldSerial),
ZODB.POSException.ConflictError: database conflict error
    (oid 0x00,
    class persistent.mapping.PersistentMapping,
    serial this txn started with 0x03dfd117d4dbf099 2021-05-03 16:23:49.888861,
    serial currently committed 0x03dfd123995e91dd 2021-05-03 16:35:35.945956)

How did ZODB know the object had changed? In addition to keeping track of its OID, each persistent object also keeps track of its serial number in the _p_serial special attribute. When one connection tries to make a change to an object, the serial number of the object being changed is compared with the serial number ZODB has stored for the object, and if they're not equal, there's a conflict. (The Connection.readCurrent() API does the same thing for objects that haven't been modified.)

We can confirm this by matching the _p_serial of each connection's object with the error message reported above.

>>> conn2.root()._p_serial
b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x99'
>>> hex(u64(conn2.root()._p_serial))
'0x3dfd117d4dbf099'
>>> conn.root()._p_serial
b'\x03\xdf\xd1#\x99^\x91\xdd'
>>> hex(u64(conn.root()._p_serial))
'0x3dfd123995e91dd'

That serial number is the transaction ID. This is sometimes abbreviated to TID.

Thus, to uniquely identify a particular revision of a particular object, we need both its OID and TID. (Together, they form the full "address" of the object.) The storage object has methods to load object states as they existed at particular serial numbers:

>>> db.storage.loadSerial(z64, conn2.root()._p_serial)
(b'\x80\x03cpersistent.mapping\nPersistentMapping\nq\x00.\x80\x03}q\x01'
 b'X\x04\x00\x00\x00dataq\x02}q\x03X\x03\x00\x00\x00objq\x04C\x08>.'
 b'\xe6\x8c\x1a\xb9\xf8\xd3q\x05cpersistent\nPersistent\nq\x06\x86q\x07Qss.')
>>> db.storage.loadSerial(z64, conn.root()._p_serial)
(b'\x80\x03cpersistent.mapping\nPersistentMapping\nq\x00.\x80\x03}q\x01'
 b'X\x04\x00\x00\x00dataq\x02}q\x03(X\x03\x00\x00\x00objq\x04C\x08>'
 b'.\xe6\x8c\x1a\xb9\xf8\xd3q\x05cpersistent\nPersistent\nq\x06\x86q\x07QX\x04'
 b'\x00\x00\x00obj2q\x08C\x08>.\xe6\x8c\x1a\xb9\xf8\xd4q\th\x06\x86q\nQus.')

Setting the Serial Number

You might be wondering when the _p_serial of a persistent object is set. There are two times. First, when an object has been added or modified during a transaction, one of the last things ZODB does is update the _p_serial. Here's part of tpc_finish (the last part of committing a transaction) from Connection:

def tpc_finish(self, transaction):
    """Indicate confirmation that the transaction is done.
    """
    serial = self._storage.tpc_finish(transaction)
    assert type(serial) is bytes, repr(serial)
    for oid_iterator in self._modified, self._creating:
        for oid in oid_iterator:
            obj = self._cache.get(oid)
            # Ignore missing objects and don't update ghosts.
            if obj is not None and obj._p_changed is not None:
                obj._p_changed = 0
                obj._p_serial = serial

Notice that it's the underlying storage that's responsible for allocating the transaction ID. Likewise, the underlying storage is responsible for allocating new OIDs when objects are first stored.

Transaction IDs Are Based On the Current Time

When a transaction commits, ZODB assigns it a transaction ID based on the current time. This is part of tpc_begin (the first part of committing a transaction) from the BaseStorage class, a common base class for many storage implementations:

def tpc_begin(self, transaction, tid=None, status=' '):
    ...
    if tid is None:
        now = time.time()
        t = TimeStamp(*(time.gmtime(now)[:5] + (now % 60,)))
        ...

Transaction IDs Must Increase

Each transaction that commits gets a new transaction ID, based on the current time. Since time moves forward (later is always later than now), that suggests that subsequent transaction IDs will just get bigger and bigger. And normally that would be true. But suppose the computer's clock gets set back, or suppose the database file is moved from one machine whose clock was running way ahead to another machine whose clock is set to the right time. How do we guarantee that a value derived from time.time() is actually going to be larger than the last value?

BaseStorage and the TimeStamp class have us covered. BaseStorage keeps track of the last TID that's been committed, and TimeStamp has a laterThan method that guarantees that the returned value is larger than the previous value:

def tpc_begin(self, transaction, tid=None, status=' '):
    ...
    if tid is None:
        now = time.time()
        t = TimeStamp(*(time.gmtime(now)[:5] + (now % 60,)))
        self._ts = t.laterThan(self._ts)

>>> later = ts.laterThan(ts)
>>> later > ts
True
>>> ts
b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x99'
>>> later
b'\x03\xdf\xd1\x17\xd4\xdb\xf0\x9a'
>>> print(later)
2021-05-03 16:23:49.888861
>>> print(ts)
2021-05-03 16:23:49.888861
>>> later.timeTime()
1620059029.888861
>>> ts.timeTime()
1620059029.888861