Fluent Python Study Notes - Function decorators and closures

Posted on   |   In   |  

Decorators

A decorator is a callable that takes another functions as argument (the decorated function). The decorator may perform some processing with the decorated function, and returns it or repalces it with another function or callable object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def deco(func):
    def inner():
        print 'running inner'
        return func()
    return inner

@deco
def target():
    print 'running target'

target()
# output => running inner
# output => running target

print target
# output => <function inner at 0x7f44d20e8c80>
# now target is a reference to function inner

Strictly speaking, decorators are just syntactic sugar. you can always simply call a decorator like any regular callable, passing another function.

How decorators executed

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
registry = []

def register(func):
    def inner():
        print('running register(%s)' % func)
        registry.append(func)
        return func()
    return inner 

@register
def f1():
    print('running f1()')

@register
def f2():
    print('running f2()')

def f3():
    print('running f3()')

def main():
    print('running main()')
    print('register 1 -> %r' % registry)
    f1()
    f2()
    f3()
    print('register 2 -> %r' % registry)

if __name__ == '__main__':
    main()
# output =>
# running main()
# register 1 -> []
# running register(<function f1 at 0x7f2c481038c8>)
# running f1()
# running register(<function f2 at 0x7f2c481039d8>)
# running f2()
# running f3()
# register 2 -> [<function f1 at 0x7f2c481038c8>, <function f2 at 0x7f2c481039d8>]

Test in python 2.9 and python 3.5, it is different with the author said as the decorators is runing right after the decorated function is defined in books(page 185, 186). the decorators is running just before the decorated funtion is running.

Most decorators do not change the decorated function, they usually do it by defining an inner function and returning it to replace the decorated fucniton.

Variable Scope rules

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
b = 6
def f1(a):
    print a
    print b

def f2(a):
    print a
    print b
    b = 9

f1(3)
# output => 3
# output => 6

f2(3)
# output => 6
# Traceback (most recent call last):
#  File "deco.py", line 58, in <module>
#    f2(3)
#  File "deco.py", line 54, in f2
#    print b
# UnboundLocalError: local variable 'b' referenced before assignment

Python compiles the body of the function, it decides that b is a local variable because it is assined within the function.
When call f2(3), the body of f2 fetches and prints the value of the local variable a, but when trying to fetch b, it discoves that b is unbound.

This is a design choice: python does not require you to declare variables, but assumes that a variable assigned in th body of function is local.

If you want the interpreter to treat b as gloabl variable in spite of the assignment within the function, you can use the gloabl declaration.

1
2
3
4
5
6
7
8
9
10
b = 6
def f3(a):
    global b
    print a
    print b
    b = 9

f3(3)
# output => 3
# output => 6

Closures

A closure is function with extended scope that encompasses non-global variables referenced in the body of the function but not defined there. It does not matter whether the function is anonymous or not, what matters is that it can access non-global varialble that are defined outside its body.

Closures are functions that retains the bindings of the free varaibles that exist when the function is defined, so that they can be used later when the functions is invoked and the defining scope is no longer available.

The nonlocal declaration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def make_avg():
    count = 0
    total = 0
    
    def avg(val):
        count += 1
        total += val
        return total / count
    return avg

avg = make_avg()
print avg(10)
# output => 
# Traceback (most recent call last):
#   File "deco.py", line 72, in <module>
#     avg(10)
#   File "deco.py", line 66, in avg
#     count += 1
# UnboundLocalError: local variable 'count' referenced before assignment

The problem is that the statement count += 1 actually means the same as count = count + 1, when count is a number or any immutable type. So we are actually assigning to count in the body of avg, and that makes it local variable. The same problem affacts the total variable.

To save the above problem, the nonlocal declaration was introduced in python 3. It lets you flag a variable as a free variable even when it is assigned a new value within the function. If a new value is assigned to a nonlocal variable, the binding stored in the closure is changed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# python 3.5
def make_avg():
    count = 0
    total = 0

    def avg(val):
        nonlocal count, total
        count += 1
        total += val
        return total / count
    
    return avg

print(avg(10))
# output => 10.0
print(avg(12))
# output => 11.0

Decorators in the standard library

Python has three built-in functions that are designed to decorate methods: property, classmethod and staticmethod.
Another frequently seen decorator is functools.wraps, a helper for building well-behaved decorators.

functools.lru_cache

functools.lru_cache is very practical decorator that implements memoization: an optimization technique which works by saving the results of previous invocations of an expensive function, avoiding repeat computations on previously used arguments.

lru_cache can be tuned by passing two optional arguments. It’s full signature is:

1
functools.lru_cache(maxize=128, typed=False)

The maxsize arguments determines how many call results are stored. After the cache is full, older results are discarded to make room. For optimal performance, maxsize should be a power of 2. Type typed argument, if set to True, stores results of different argument types separately.

Generic functions with single dispatch

The new functools.singledispatch decorator is python 3.4 allows each module to contribute to the overall solution, and lets you easily provide a specialized function even for classes that you can’t edit. If you decorate a plain function with @singledispatch it becomes a generic functions: a group of functions to perform the same operation in different ways, depending on the type of the first argument. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# python 3.5
from functools import singledispatch
from collections import abc
import numbers
import html

@singledispatch 
def htmlize(obj):
    print('in htmlize(obj)')
    content = html.escape(repr(obj))
    return '<pre>{}</pre>'.format(content)

@htmlize.register(str)
def _(text):
    print('in _(text)')
    content = html.escape(text).replace('\n', '<br>\n')
    return '<p>{0}</p>'.format(content)

@htmlize.register(numbers.Integral)
def _(n):
    print('in _(n)')
    return '<pre>{0} (0x{0:x})</pre>'.format(n)

@htmlize.register(tuple)
@htmlize.register(abc.MutableSequence)
def _(seq):
    print('in _(seq)')
    inner = '</li>\n<li>'.join(htmlize(item) for item in seq)
    return '<ul>\n<li>' + inner + '</li>\n</ul>'

print(htmlize({1, 2, 3}))
# output => in htmlize(obj)
# output => <pre>{1, 2, 3}</pre>
print(htmlize('feifeiyu'))
# output => in _(text)
# output => <p>feifeiyu</p>
print(htmlize(abs))
# output => in htmlize(obj)
# output => <pre>&lt;built-in function abs&gt;</pre>
print(htmlize(42))
# output => in _(n)
# output => <pre>42 (0x2a)</pre>
print(htmlize(['alpha', 66, {3,2,1}]))
# output => in _(seq)
# output => in _(text)
# output => in _(n)
# output => in htmlize(obj)
# output => <ul>
# output => <li><p>alpha</p></li>
# output => <li><pre>66 (0x42)</pre></li>
# output => <li><pre>{1, 2, 3}</pre></li>
# output => </ul>

@singledispatch: register the specialized functions to hanlde ABCs (abstract classes) such as numbers.Integral and abc.MutableSequence instead of concrete implementations like int and list. This allows your code to support a greater variety of compatible types. For example, a python extension can provide alternatives to the int type with fixed bit lengths as subclasses of numbers.Integral.

A notable quality of the singledispatch mechanism is that you can register specialized functions anywhere in the system, in any module.

Stacked decorators

When two decorators @d1 and @d2 are applied to a function f in that order, the result is the same as f = d1(d2(f))

Parametrized Decorators

When parsing a decorator in source code, python takes the decorated function and passes it as the first argument to the decorator function. How to make a decorator accept other arguments? The answer is: make a decorator factory that takes those arguments and returns a decorator, which is then applied to the function to be decorated.

A parametrized registration decorator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
registry = set()

def register(active=True):
    def decorator(func):
        print 'runing register(avctive=%s) -> decorate(%s)' % (active, func)
        if active:
            registry.add(func)
        else:
            registry.discard(func)
        return func
    return decorator

@register(active=False)
def f1():
    print 'running f1()'

@register()
def f2():
    print 'running f2()'

def f3():
    print 'running f3()'

f1()
f2()
f3()
print registry
# output =>
# runing register(avctive=False) -> decorate(<function f1 at 0x7f1bfbd61c80>)
# runing register(avctive=True) -> decorate(<function f2 at 0x7f1bfbd61cf8>)
# running f1()
# running f2()
# running f3()
# set([<function f2 at 0x7f1bfbd61cf8>])

In order to make it easy to enable or disable the function registration performed by register, an optional active parameter which, if False skips registering the decorated function.

As shown in above example, the new register function is not a decorator but a decorator factory. When called, it returns the actual decorator that will be applied to the target function.

instead of using the @ syntax, we used register as a regular function, the syntax ineeded to decorate a function f would be register()(f) to add f to the register(active=False)(f) to remove it.

End