Python Argument Surprise

Python function signatures are flexible, complex beasts, allowing for positional, keyword, variable, and variable keyword arguments (and parameters). This can be extremely useful, but sometimes the intersection between these features can be confusing or even surprising, especially on Python 2. What do you expect this to return?

>>> def test(arg1, **kwargs):
...     return arg1
>>> test(**{'arg1': 42})
...

Contents

Terminology

Before we move on, let's take a minute to define some terms.

parameter: The name of a variable listed by a function definition. These are sometimes called "formal parameters" or "formal arguments." In def foo(a, b), a and b are parameters.
argument: The expression given to a function application (function call). In foo(1, "str"), 1 and "str" are arguments.
function signature: The set of parameters in a function definition. This is also known as a "parameter list."
binding: The process of associating function call arguments to the parameter names given in the function's signature. In foo(1, "str"), the parameter a will be assigned the value 1, and the parameter b will be assigned the value "str". This is also called "argument filling."
default parameter: In a function signature, a parameter that is assigned a value. An argument for this parameter does not have to be given in a function application; when it is not, the default value is bound to the parameter. In def foo(a, b=42), b=42 creates a default parameter. It can also be said that b has a default parameter value. The function can be called as foo(1).
positional argument: An argument in a function call that's given in order of the parameters in the function signature, from left to right. In foo(1, 2), 1 and 2 are positional arguments that will be bound to the parameters a and b.
keyword argument: An argument in a function call that's given by name, matching the name of a parameter. In foo(a=1), a=1 is a keyword argument, and the parameter a will have the value 1.
variable parameters: A function signature that contains *args (where args is an arbitrary identifier) accepts an arbitrary number of unnamed arguments in addition to any explicit parameters. Extra parameters are bound to args as a list. def foo(a, b, *args) creates a function that has variable parameters, and foo(1, 2), foo(1, 2, 3), foo(1, 2, 3, 4) are all valid ways to call it. This is commonly called "varargs," for "variable arguments" (even though it is a parameter definition).
variable positional arguments: Passing an arbitrary (usually unknown from the function call itself) number of arguments to a function by unpacking a sequence. Variable arguments can be given to a function whether or not it accepts variable parameters (if it doesn't, the number of variable arguments must match the number of parameters). This is done using the * syntax: foo(*(1, 2)) is the same as writing foo(1, 2), but more often the arguments are created dynamically. For example, args = (1, 2) if full_moon else (3, 4); foo(*args).
variable keyword parameters: A function signature that contains **kwargs (where kwargs is an arbitrary identifier) accepts an arbitrary number of keyword arguments in addition to any explicit parameters (with or without default values). The definition def foo(a, b, **kwargs) creates a function with a variable keyword parameter. It can be called like foo(1, 2) or foo(1, 2, c=42, d=420).
variable positional arguments: Similar to variable positional arguments, but using keyword arguments. The syntax is **, and the object to be unpacked must be a mapping; extra arguments are placed in a mapping bound to the parameter identifier. A simple example is foo(**{'b':"str", 'a':1}).

Some language communities are fairly strict about the usage of these terms, but the Python community is often fairly informal. This is especially true when it comes to the distinction between parameters and arguments (despite it being a FAQ) which helps lead to some of the surprises we discuss below.

Surprises

On to the surprises. These will all come from the intersection of the various terms defined above. Not all of these will surprise everyone, but I would be surprised if most people didn't discover at least one mildly surprising thing.

Non-Default Parameters Accept Keyword Arguments

Any parameter can be called using a keyword argument, whether or not it has a default parameter value:

>>> def test(a, b, c=42):
...    return (a, b, c)
>>> test(1, 2)
(1, 2, 42)
>>> test(1, b='b')
(1, 'b', 42)
>>> test(c=1, b=2, a=3)
(3, 2, 1)

This is surprising because sometimes parameters with a default value are referred to as "keyword parameters" or "keyword arguments," suggesting that only they can be called using a keyword argument. In reality, the parameter just has a default value. It's the function call site that determines whether to use a keyword argument or not.

One consequence of this: the parameter names of public functions, even if they don't have a default, are part of the public signature of the function. If you distribute a library, people can and will call functions using keyword arguments for parameters you didn't expect them to. Changing parameter names can thus break backwards compatibility. (Below we'll see how Python 3 can help with this.)

Corollary: Variable Keyword Arguments Can Bind Non-Default Parameters

If we introduce variable keyword arguments, we see that this behaviour is consistent:

>>> kwargs = {'a': 'akw', 'b': 'bkw'}
>>> test(**kwargs)
('akw', 'bkw', 42)

Corollary: Positional Parameters Consume Keyword Arguments

Knowing what we know now, we can answer the teaser from the beginning of the article:

>>> def test(arg1, **kwargs):
...     return arg1, kwargs
>>> test(**{'arg1': 42})
(42, {})

The named parameter arg1, even when passed in a variable keyword argument, is still bound by name. There are no extra arguments to place in kwargs.

Default Parameters Accept Positional Arguments

Any parameter can be called using a positional argument, whether or not it has a default parameter value:

>>> def test(a=1, b=2, c=3):
...   return (a, b, c)
>>> test('a', 'b', 'c')
('a', 'b', 'c')

This is the inverse of the previous surprise. It may be surprising for the same reason, the conflation of keyword arguments and default parameter values.

Of course, convention often dictates that default parameters are passed using keyword arguments, but as you can see, that's not a requirement of the language.

Corollary: Variable Positional Arguments Can Bind Default Parameters

Introducing variable positional arguments shows consistent behaviour:

>>> pos_args = ('apos', 'bpos')
>>> test(*pos_args)
('apos', 'bpos', 3)

Mixing Variable Parameters and Keyword Arguments Will Break

Suppose we'd like to define some parameters with default values (expecting them to be passed as keyword arguments by convention), and then also allow for some extra arguments to be passed:

>>> def test(a, b=1, *args):
...    return (a, b, args)

The definition works. Now lets call it in some common patterns:

>>> test('a', 'b')
('a', 'b', ())
>>> test('a', 'b', 'c')
('a', 'b', ('c',))
>>> test('a', b=1)
('a', 1, ())

>>> test('a', b='b', *(1, 2))
Traceback (most recent call last):
...
TypeError: test() got multiple values for argument 'b'

As long as we don't mix keyword and variable (extra) arguments, everything works out. But as soon as we mix the two, the variable positional arguments are bound first, and then we have a duplicate keyword argument left over for b.

This is a common enough source of errors that, as we'll see below, Python 3 added some extra help for it, and linters warn about it.

Functions Implemented In C Can Break The Rules

We generally expect to be able to call functions with keyword arguments, especially when the corresponding parameters have default values, and we expect that the order of keyword arguments doesn't matter. But if the function is not implemented in Python, and instead is a built-in function implemented in C, that may not be the case. Let's look at the built-in function math.tan. On Python 3, if we ask for the function signature, we get something like this:

>>> import math
>>> help(math.tan)
Help on built-in function tan in module math:
<BLANKLINE>
...
    tan(x)
<BLANKLINE>
    Return the tangent of x (measured in radians).
<BLANKLINE>

That sure looks like a parameter with a name, so we expect to be able to call it with a keyword argument:

>>> math.tan(x=1)
Traceback (most recent call last):
...
TypeError: tan() takes no keyword arguments

This is due to how C functions bind Python arguments into C variables, using functions like PyArg_ParseTuple.

In newer versions of Python, this is indicated with a trailing / in the function signature, showing that the preceding arguments are positional only paramaters. (Note that Python has no syntax for this.)

>>> help(abs)
Help on built-in function abs in module builtins:
<BLANKLINE>
abs(x, /)
    Return the absolute value of the argument.

Python 3 Improvements

Python 3 offers ways to reduce some of these surprising characteristics. (For backwards compatibility, it doesn't actually eliminate any of them.)

We've already seen that functions implemented in C can use new syntax in their function signatures to signify positional-only arguments. Plus, more C functions can accept keyword arguments for any arbitrary parameter thanks to new functions and the use of tools like Argument Clinic.

The most important improvements, though, are available to Python functions and are outlined in the confusingly named PEP 3102: Keyword-Only Arguments.

With this PEP, functions are allowed to define parameters that can only be filled by keyword arguments. In addition, this allows functions to accept both variable arguments and keyword arguments without raising TypeError.

This in done by simply moving the variable positional parameter before any parameters that should only be allowed by keyword:

>>> def test(a, *args, b=42):
...    return (a, b, args)
>>> test(1, 2, 3)
(1, 42, (2, 3))
>>> test(1, 2, 3, b='b')
(1, 'b', (2, 3))
>>> test(1, 2, 3, b='b', c='c')
Traceback (most recent call last):
...
TypeError: test() got an unexpected keyword argument 'c'
>>> test()
Traceback (most recent call last):
...
TypeError: test() missing 1 required positional argument: 'a'

What if you don't want to allow arbitrary unnamed arguments? In that case, simply omit the variable argument parameter name:

>>> def test(a, *, b=42):
...    return (a, b)
>>> test(1, b='b')
(1, 'b')

Trying to pass extra arguments will fail:

>>> test(1, 2, b='b')
Traceback (most recent call last):
...
TypeError: test() takes 1 positional argument but 2 positional arguments (and 1 keyword-only argument) were given

Finally, what if you want to require certain parameters to be passed by name? In that case, you can simply leave off the default value for the keyword-only parameters:

>>> def test(a, *, b):
...    return (a, b)
>>> test(1)
Traceback (most recent call last):
...
TypeError: test() missing 1 required keyword-only argument: 'b'
>>> test(1, b='b')
(1, 'b')

The above examples all produce SyntaxError on Python 2. Much of the functionality can be achieved on Python 2 using variable arguments and variable keyword arguments and manual argument binding, but that's slower and uglier than what's available on Python 3. Lets look at an example of implementing the first function from this section in Python 2:

>>> def test(*args, **kwargs):
...    "test(a, *args, b=42) -> tuple" # docstring signature for Sphinx
...    # This raises an IndexError instead of a TypeError if 'a'
...    # is missing; that's easy to fix, but it's a programmer error
...    a = args[0]
...    args = args[1:]
...    b = kwargs.pop('b', 42)
...    if kwargs:
...        raise TypeError("Got extra keyword args %s" % (list(kwargs)))
...    return (a, b, args)
>>> test(1, 2, 3)
(1, 42, (2, 3))
>>> test(1, 2, 3, b='b')
(1, 'b', (2, 3))
>>> test(1, 2, 3, b='b', c='c')
Traceback (most recent call last):
...
TypeError: Got extra keyword args ['c']
>>> test()
Traceback (most recent call last):
...
IndexError: tuple index out of range