Jason Madden (Posts about hub)

Blocking gevent's Hub Part 2: Finding Blocking

Jason Madden — Wed, 27 Jun 2018 14:36:00 GMT

Last time we looked at the causes and consequences of greenlets that block and avoid switching to the hub. Many times such greenlets (really, the code they run) are obvious, but some times they are not.

This post will look at some new features that gevent 1.3 added to make finding greenlets that block easier. These can be used both during development and in production.

Contents

What It does

gevent 1.3 includes a couple features to help find code that's blocking. One of those we can turn on using just environment variables, no code changes needed (although you can configure it in code as well). This makes it especially handy for debugging on the fly.

Let's take a look at one of our blocking examples from last time:

import time
import gevent

def f():
    time.sleep(3)

now = time.time()
greenlet = gevent.spawn(f)
while not greenlet.dead:
    print('.', end='')
    gevent.sleep(1)
after = time.time()
print("\nElapsed time: %1.1f" % (after - now))

Now we'll tell gevent to let us know if it detects any greenlets blocking the hub and run the example (the duration of time that a greenlet is allowed to block before it gets reported is configurable; we'll go with the defaults here):

$ GEVENT_MONITOR_THREAD_ENABLE=true python /tmp/test.py
================================================================================

2018-06-22T19:20:10Z : Greenlet <Greenlet "Greenlet-0" at 0x1.: f> appears to be blocked
    Reported by <gevent.__tracer.GreenletTracer object at 0x...>
Blocked Stack (for thread id 0x...):
  File "/tmp/test.py", line 5, in f
    time.sleep(3)

Info:
********************************************************************************
* Threads
********************************************************************************
Thread 0x2 (MainThread)

  File "/tmp/test.py", line 5, in f
    time.sleep(3)

********************************************************************************
* Greenlets
********************************************************************************
<greenlet.greenlet object at 0x3>
 :    Parent: None
 :    Monitoring Thread:<PeriodicMonitoringThread >
<greenlet.greenlet object at 0x4>
 :    Parent: None
 +--- <Greenlet "Greenlet-0" at 0x1: f>
 :          Parent: <Hub '' at 0x5 select default pending=0 ref=1 thread_ident=0x2>
 :          Spawned at:
 :            File "/tmp/test.py", line 8, in <module>
 :              greenlet = gevent.spawn(f)
 +--- <Hub '' at 0x5 select default pending=0 ref=1 thread_ident=0x2>
            Parent: <greenlet.greenlet object at 0x4>

 .
 Elapsed time: 3.0

This time, gevent detected the hold up and printed a report about what the problem was. [1]

Testing And Asserting

Either in unittests, from particularly latency sensitive parts of your application, or when trying to further narrow down the cause of blocking, gevent provides the assert_switches() context manager. Wrap this manager around some code to be sure that it switches to another greenlet or back to the hub, optionally within a set amount of time. If nothing switches, then an exception will be raised with the same sort of information as above.

>>> from gevent.util import assert_switches
>>> # This will always raise an exception: nothing switches
>>> with assert_switches():
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "//gevent/util.py", line 592, in __exit__
    raise _FailedToSwitch('\n'.join(report_lines))
gevent.util._FailedToSwitch: ...
>>> # This will never raise an exception; nothing switched,
... # but it happened very fast
... with assert_switches(max_blocking_time=1.0):
...     pass
...
>>>

How It Works

Credit

This technique did not originate with gevent. I've seen it in this blog post from 2012 by Ryan Kelly and I've seen it in this code from Mozilla. gevent 1.3 just generalized it, made it safe for multiple threads and hubs, and incorporated it into the core of gevent.

The main question we want to be able to answer is: did the hub get to run during a certain time period? Running the hub means switching into its greenlet, so what we really want to know is: was the greenlet hub switched into?

The greenlet library provides a hook well suited for this purpose: greenlet.settrace(). This functions similarly to sys.settrace() that's part of the core interpreter. The later calls a function for every "interesting" interpreter event such as a function call, and the former calls a function for every interesting event involving greenlets.

Since greenlets can really only do two interesting things, 'switch' and 'throw' are the only two defined events—for our purposes they are equivalent as they both involve a transfer of control from one greenlet to another, and both events tell us which greenlet is going to be taking control (the "target").

So to answer our question, we only need to keep track of whether or not the greenlet hub was the target of an event and thus assumes control. For the decorator assert_switches, which has a well-defined scope and only needs to answer that question at the exact time it is exited, that's enough [2], but for the general monitoring function that lets you know if (when) it ever found the hub blocked for longer than a certain period, we need to do a little more. To be able to answer that question meaningfully, we also need to know the timestamp of the last event entering the hub so we can compare it to the current timestamp: if it's been "too long," then we can raise an alert.

Performance

gevent is a performance critical library, so monitoring features that are too perfomance sucking to enable are a non-starter. For that reason, the implementation of these features was carefully optimized and a few different tradeoffs are made. Here are some examples.

Getting the current time (time.time()) is relatively expensive, so we try to avoid doing that too often. For example, instead of checking if we've exceeded the blocking threshold on every greenlet switch, we only do so periodically in a background thread.
We use some heuristics to detect if we switched into the hub instead of absolute timestamps.
The greenlet trace functions are compiled with Cython and strongly typed. This makes the relatively complex work that the blocking monitor trace function needs to do faster than the most trivial trace function implemented in Python.

As always, if you find bugs or can think of enhancements or additions to gevent, please feel free to open an issue or create a pull request!

Footnotes

[1]

I admit I cheated here in the output a little bit. I trimmed it: it detected, and reported, the same blockage several times. It should probably recognize that and just print something about "same blockage". I should fix that. Also, I fixed up the spawning line. Things spawned at the module level do not always get the right traceback.

[2]	As long as you don't use the `max_blocking_time` parameter.

Blocking gevent's Hub Part 1: Understanding Blocking

Jason Madden — Tue, 26 Jun 2018 14:36:00 GMT

In the beginning we talked about gevent's hub and how greenlets switch in and out of it to implement IO. Following that we showed how locks in gevent are implemented much the same way, by "parking" a waiting greenlet, switching to the hub to let other greenlets run or do IO, and eventually switching back to the parked greenlet.

That's a lot of switching. What does it mean if that switching doesn't happen? What should a programmer know about switching and its opposite, blocking? (There's also part 2.)

Contents

Cooperation

gevent (and greenlets in general) is a cooperative multitasking system, meaning that any given greenlet runs until it chooses to give up control. Under gevent, choosing to give up control is usually automatic, and usually happens when the greenlet wants to handle IO, such as reading or writing a socket to handle an incoming HTTP request or HTTP response. A greenlet can also give up control automatically by waiting for a lock that another greenlet owns, or it can explicitly give up control by using the gevent.sleep() API. In all those cases, control passes to the hub (recall that the hub is the greenlet running the event loop); this is referred to as yielding to the hub. When a greenlet yields to the hub, another greenlet gets a chance to run or IO gets serviced by the event loop. A greenlet that yields to the hub is called cooperative (or sometimes green). When a greenlet doesn't yield, it's non-cooperative.

Examples

Of course, it's not actually the greenlet itself that's yielding, it's the code that the greenlet is running that yields. We can thus say that any given piece of code is cooperative or green or not. Here are some examples:

gevent.sleep() is cooperative

import time
import gevent

def f():
    gevent.sleep(3)

now = time.time()
greenlet = gevent.spawn(f)
while not greenlet.dead:
    print('.', end='')
    gevent.sleep(1)
after = time.time()
print("\nElapsed time: %1.0f" % (after - now))

On my system, that prints three dots, indicating that the greenlets are yielding to the hub and switching back and forth:

$ python test.py
...
Elapsed time: 3.0

I will omit the timing part of the main greenlet in the rest of the examples unless it changes.

time.sleep() is not cooperative

Here's that same example, with the inner greenlet using time.sleep() from the standard library.
```
import time
import gevent

def f():
    time.sleep(3)
```
```
$ python test.py
.
Elapsed time: 3.0
```
We get only one dot, indicating that as soon as the main greenlet yielded to the hub, which in turn handed control over to the greenlet running f, that greenlet failed to yield again until it was completely finished.

Code that is CPU-bound is not cooperative

If a function is in a tight computation loop, there is no opportunity for it to yield control.
```
import time
import gevent

def f():
    i = 0
    while i < 2 ** 28:
        i += 1
```
```
$ python test.py
.
Elapsed time: 28.2
```
If that had been cooperative, we would have expected to see about 28 dots, since that's how long the inner greenlet ran. We only got one.

Socket IO using gevent.socket is cooperative

import time
import gevent
import gevent.socket

def f():
    gevent.socket.gethostbyname('www.python.org')

now = time.time()
greenlet = gevent.spawn(f)
while not greenlet.dead:
    print('.', end='')
    gevent.sleep(0.001) # NOTE: We switch to a smaller sleep duration
after = time.time()
print("\nElapsed time: %1.1f" % (after - now))

Lots of dots here.

$ python test.py
.....................
Elapsed time: 0.0

Socket IO using the standard socket is not cooperative [1]

import time
import gevent
import socket

def f():
    socket.gethostbyname('www.python.org')

Again, only one dot, meaning no yielding.

$ python test.py
.
Elapsed time: 0.0

C extension libraries may or may not be cooperative

This is a tricky area. C extensions are so common in Python (much of the standard library is implemented in C) that you may not even realize you're using one. Commonly, C extensions are used for two major purposes.

The first is to accelerate CPU intensive code (for example, numpy to speed up linear algebra). These are probably no more or less cooperative than a counterpart written in Python. (I'll explain what I mean by that in a minute.)

The second is to integrate with other existing native (C) libraries, such as imaging libraries or database clients. Some of these are essentially CPU accelerators, so they basically fall into the first category. But libraries that want to do IO, especially talking to remote systems, particularly databases, are likely to want to handle IO their own way and, without special hooks, are unlikely to be cooperative. (The PostgreSQL driver and its Python C extension psycopg2 is an example of a database driver that provides the necessary hooks to be made cooperative using psycogreen.)

Back to that "no more or less cooperative than a counterpart written in Python" statement. If the C extension calls from C back into Python code, and that Python code itself is cooperative, than thanks to the magic of greenlet being able to switch C call stacks, the whole thing winds up cooperative. The call from C into Python could either be explicit, or implicit (for example, sorting or hashing values can call arbitrary Python code in an object's __eq__ and __hash__ methods).

A few good examples of CPU accelerator extensions that can operate cooperatively are the BTrees (sorted dictionaries) and zope.interface (fast adapter and component registration and lookup) libraries. These libraries need to sort or hash objects. They are commonly used with objects that are persistent and stored in a ZODB database, so sorting or hashing may need to transparently fetch the object from the database. ZODB is implemented in pure Python, so the original C operation winds up being cooperative. [2]

Synergism Matters

The point of having one greenlet periodically print dots in those examples while the other greenlet either cooperated or not was to show why cooperative code matters. If one greenlet is not cooperating, no other greenlet can make any forward progress (print dots). This might stop other greenlets for a noticeably long time (like the CPU-bound example of counting to 268,435,456, or it might be so quick it's difficult for a human to catch (like resolving a hostname). In either case, it can be a problem, especially if you're trying to scale a program such as a server to handle more connections. Every yield lost potentially decreases overall velocity.

Avoiding Blocking

Above, we saw some examples of non-cooperative code. There were some clear transformations to go through to make code cooperative:

Switch from standard library API to gevent API.

Change imports to import from gevent. E.g., import time⟹from gevent import time or import socket⟹from gevent import socket.
Monkey patch

In code that's designed to be portable to a non-gevent system, you can use the standard library API and have gevent automatically make it cooperative by monkey patching. Many third-party libraries such as requests that use only the Python standard API can become cooperative this way.
Break up CPU intensive loops by explicitly yielding

If you know you'll have a CPU intensive loop, like in the example, you can periodically explicitly yield by calling gevent.sleep() in the loop. Beginning in gevent 1.3, this attempts to automatically balance the amount of time CPU intensive tasks get while still allowing other greenlets to perform IO.

gevent also provides a threadpool you can use to offload CPU intensive tasks.

Diagnosing Blocking

You may have done all of that and are still not sure if you're blocking the hub or not: maybe a colleague added a new C extension to your project and you don't know its internals, or maybe there's an inner loop somewhere that might be CPU intensive and you're just not sure.

That's where my next post comes in. We'll discuss how to use gevent's monitoring facilities to find greenlets that aren't cooperating.

Footnotes

[1]	Unless of course you monkey patch, but that's another post. Or at least another section of this post.

[2]	At least when the system is monkey patched. But that's another post. Or at least another section of this post.

gevent's hub

Jason Madden — Fri, 18 May 2018 18:40:00 GMT

The hub is an important part of gevent, but aside from a few sentences in the high-level description of the event loop, and the API documentation, not a lot has been written about the hub, its purpose, or how it does what it does. This post aims to change that and answer questions like:

What is the hub?
How does the hub run the event loop?
How do greenlets do IO? That is, how do they let the event loop run, while letting other greenlets run, and then pick right back up where they blocked?

Contents

What is a hub?

Let's begin with what gevent's official documentation has to say about the hub:

When a function from gevent's API wants to block, it obtains the gevent.hub.Hub instance—a special greenlet that runs the event loop—and switches to it (it is said that the greenlet yielded control to the Hub). If there's no gevent.hub.Hub instance yet, one is automatically created.

Not exactly self explanatory. The API documentation doesn't add much to that.

To try to understand what that means, let's break it into chunks.

The hub is a greenlet

The hub is a greenlet. greenlets are one implementation of green threads. They're like a "normal" operating system thread in that each greenlet has a call stack (the C call stack of the Python interpreter, plus the Python call stack) and represents one flow of control through a program. They are different in that many greenlets can be associated with a single operating system thread. [1] This has the important effect that only one greenlet attached to a thread is ever actually executing at any given time.

Another way in which greenlets differ from typical operating system threads is that they are cooperatively scheduled, instead of preemptively scheduled. That is, in order for another greenlet to be able to run, the greenlet that is currently running must choose to give up control to it; this is called "switching". Moreover, it must actually choose the greenlet it wants to run next! [2] To do so, the current greenlet invokes the switch method of the destination greenlet When a greenlet switch occurs, the call stack of the current greenlet is saved away, the call stack of the destination greenlet is put in place, and execution resumes where the destination greenlet left off. [3]

The hub runs the event loop

Ok, so the hub is a greenlet—a cooperatively scheduled call stack or thread of execution. That thread of execution is running the event loop.

Here's how gevent's documentation describes an event loop:

Instead of blocking and waiting for socket operations to complete (a technique known as polling), gevent arranges for the operating system to deliver an event letting it know when, for example, data has arrived to be read from the socket. Having done that, gevent can move on to running another greenlet, perhaps one that itself now has an event ready for it. This repeated process of registering for events and reacting to them as they arrive is the event loop.

In a visual sense, an event loop looks something like this (this is a very high-level overview of libuv's event loop; don't worry too much about the specific details or terminology [4]):

"Running the event loop" means that the hub enters the top of that loop and cycles through it forever. The event loops that gevent supports are implemented in C, so entering the top of the loop means making a C function call [5]. That C function call is not expected to return (because it's an infinite loop).

"Registering for events" is done with objects called "watchers". Each watcher object is looking for one particular event to happen.

How, then, do other greenlets—the ones that actually read and write socket data and otherwise make up your application—get to run? This is where it gets interesting.

Getting out of the hub

Look at that diagram again. See those blocks that start with "Call", like "Call pending callbacks", or "Run", like "Run due timers"? Those blocks are our loophole. That's when the event loop—while still inside its single C function—gives gevent a chance to do something, such as respond to an IO event or handle a scheduled event (timer).

For each event that gevent might want to handle, it has created a watcher and given it to the event loop. Each watcher is associated with a callback: a function that the event loop will call when the desired event occurs. (Giving a watcher to the event loop and associating it with a callback is referred to as "starting" the watcher.) That function can do just about anything. If that function happens to invoke destination_greenlet.switch(), then whoosh, just like that, the hub and the event loop it's somewhere in the middle of running is paused (its C call stack is saved away) and some other greenlet takes off running.

Wait, how did we get in the hub in the first place?

We just saw how we can get out of the hub when its busy running the event loop: invoke a callback that calls greenlet.switch. But how did we start running the event loop in the hub in the first place?

Let's look back to the quoted description of the hub:

When a function from gevent's API wants to block, it obtains the gevent.hub.Hub instance—a special greenlet that runs the event loop—and switches to it. If there's no gevent.hub.Hub instance yet, one is automatically created.

So any gevent blocking function (such as gevent.socket.socket.read()) is going to go switch into the hub. If it's the first time the hub has been entered, the hub will start up the event loop. Otherwise, if the hub was already running the event loop, it will pick up where it last left off and either respond to the next event (IO, timer) by calling another callback, or let the event loop wait for the next event.

You can imagine it something like a game of ping pong, with the hub on one side of the table, and all the other greenlets on the other side. The hub hits the ball over the net to a greenlet who (eventually) hits it back, and then the hub hits the ball to another greenlet, and so on, ad infinitum.

Some code

Now we know enough to look at some code and put the pieces together to understand how it works under the covers.

Most blocking functions end up implementing their half of the game by calling Hub.wait(). For example, here's basically what gevent.socket.socket.recv() looks like:

def recv(self, *args):
    while True:
        try:
            return _socket.socket.recv(self._sock, *args)
        except error as ex:
            if ex.args[0] != EWOULDBLOCK:
                raise
        self.hub.wait(self._read_event)

That code is looping until we can actually read some data. If we try to read and we fail because there's nothing to read, hand things over to the event loop using hub.wait(), which will switch into the hub, make sure the event loop is watching this socket, and then carry on.

Hub.wait, in turn, is implemented something like this (this is a dramatically simplified example; the real thing is safer and uses a Waiter):

def wait(self, watcher): # Hub.wait()
    # `watcher` is an event-loop object with a callback. When the
    # event it's waiting for happens, its callback gets called.
    current_greenlet = getcurrent()
    # The callback for this event will switch back to
    # this current greenlet
    watcher.start(current_greenlet.switch) # Ask the event loop to watch this
    try:
        # Start running the hub.
        self.switch()
        # Once we get here, it's because the watcher's callback
        # fired and this greenlet got switched into.
        return # Let the blocking code continue, its event is ready
    finally:
        watcher.stop()

A few loose threads (get it?)

What happens when a greenlet finishes execution?: Control is returned to its parent greenlet. gevent arranges for the hub to be the parent of every greenlet it runs, so when a greenlet dies, whether through successful completion or an uncaught exception, the hub gets a chance to run the event loop.
What happens when we want to start a new greenlet?: gevent.spawn() creates a new greenlet and schedules it to start running in the next iteration of the event loop. It does this by, you guessed it, setting up an event watcher with a callback that's new_greenlet.switch(). That event watcher is a "prepare" watcher, a type of event that becomes available at the start of each iteration of the loop.
How are gevent locks and timeouts implemented?: You'll have to wait for another post for that! But if you want to look into the implementation, this post should provide most of the background you need.

Footnotes

[1]	For simplicity's sake, we'll assume there's only one operating system thread in the process.

[2]	In most cases. When a greenlet finishes, control is automatically handed back to its parent greenlet.

[3]	This is quite a fascinating technical accomplishment and involves assembly code for each supported platform.

[4]	In particular, do not worry about the difference between "run" and "call." Neither libuv nor libev really run their event loop in exactly this fashion. The important point is that it is a loop that does the same things over and over, in the same order each time.

[5]	In the case of libuv, that's uv_run().