Fluent Python Study Notes - First-class Functions

Posted on   |   In   |  

First-class Objects

Functions in python area first-class objects:

  • created at runtime
  • assigned to a variable or element in a data structure
  • passed as an argument to function
  • returned as the result of a function

Treating a function like an object

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def factorial(n):
    """this is doc"""
    return 1 if n < 2 else n * factorial(n-1)

factorial(5) # output => 120
# __doc__ is one of several attributes of function objects
factorial.__doc__ # output => this is doc

type(factorial) # output => <type 'function'>
# assign function to a variable
fact = factorial # output => <function factorial at 0x7feb48d505f0>
fact(5) # ouput => 120
# pass factorial as an argument to map function
map(factorial, range(5)) # oupput => [1, 1, 2, 6, 24]

Higher-order Functions

A function that takes a function as argument or returns a function as result is a higher-order function. e.g. map function.

1
2
3
4
5
6
fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
def reverse(word):
    return word[::-1]
# to sort a list of words, according to the function reverse, 
# sorted tokes reverse as an argument, sorted is a higher-order functions
sorted(fruits, key=reverse)

Modern replacements for map, filter, and reduce

Functional languages commonly offer the map, filter and reduce higher-order functions.

1
2
3
4
5
6
7
8
9
#python 2.7
map(fact, filter(lambda n: n % 2, range(5)))
# output => [1, 6]

# python 3.5
map(fact, filter(lambda n: n % 2, range(5)))
# ouput => <map object at 0x7f6ac35e5e10>
list(map(fact, filter(lambda n: n % 2, range(5))))
# output => [1, 6]

In python 3, map and filter return generators – a form of iterator, so their direct substitute is now a generator expression.
In python 2, map and filter return lists, therefore their closest alternative is a listcomp

The reduce function was demoted from a built-in in python 2 to the functools module in python 3. Its most common use case, summation, is better served by the sum built-in available since python 2.3.

1
2
3
4
5
6
7
8
9
10
# python 2.7
from operator import add
sum(range(100)) # output => 4950
reduce(add, range(100)) # output => 4950

# python 3.5
from operator import add
from functools import reduce
sum(range(100)) # ouput => 4950
reduce(add, range(100)) # output => 4950

Anonymous Functions

The lambda keyword creates an anonymous function within a python expression. The simple syntax of python limits the body of lambda functions to be pure expressions. In other words, the body of a lambda cannot make assignments or use any other python statement such as while, try etc.

The lambda syntax is just syntactic sugar: a lambda expression creates a function object just like the def statement.

The seven flavors of callable objects

The call operator, i.e. (), may be applied to other objects beyond user-defined functions. To determine whether an objects is callable, use the callable() built-in function.

The python Data Model documentation lists seven callable types:

  • User-defined functions

    create with def statements or lambda expressions.

  • Built-in functions

    a function implemented in C (for CPython), like len or time.strftime

  • Built-in methods

    methods implemented in C, like dict.get.

  • Methods

    functions defined in the body of a class.

  • Classes

    when invoked, a class runs its new method to create an instance, then init to initialize it, and finally the instance is returned to the caller.

  • Class instances

    if a class defines a call method, then its instances may be invoked as functions.

  • Generator functions

    functions or methods that use the yield keyword. Then called, generator functions return a generator object.

1
2
3
4
5
6
7
8
def test():
    print '1234'

abs, str, test, 13
# output => (<built-in function abs>, <type 'str'>, <function test at 0x7f686d9115f0>, 13)

[callable(obj) for obj in (abs, str, test, 13)]
# output => [True, True, True, False]

Function introspection

Functions objects have many attributes. Using built-in function dir(), we can revals all the attributes of a object.

1
2
dir(abs)
# output=> ['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']

From positional to keyword-only paramters

One of the best features of python functions is the extremely flexible parameter handling mechanism, enhanced with keywords-only argumetns in python 3. Closely related are the use of * and ** to “explode” iterables and mappings into seperate arguments when we call a function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
def tag(name, cls=None, *content, **attrs):
    # cls can't placed behind *content in python 2.7
    # tag(name, *content, cls=None, **attrs) works in python 3.5
    """Generate on of more HTML tags"""
    print('name: %r, content: %r, cls: %r, attrs: %r' %(name, content, cls, attrs))
    if cls != None:
        attrs['calss'] = cls
    if attrs:
        attr_str = ''.join(' %s="%s"' % (attr, value)
                            for attr, value in sorted(attrs.items())
    else:
        attr_str = ''
    if content:
        return '\n'.join('<%s%s>%s</%s>' % 
                        (name, attr_str, c, name) for c in content)
    else:
        return '<%s%s />' % (name, attr_str)

# A signle positional argument produces an empty tag with that name
print(tag('br'))
# output =>  name: 'br', content: (), cls: None, attrs: {}
# output => <br />

# Any number of arguments after the second are captured by *content as a tuple
print(tag('p', None, 'hello'))
# output => name: 'p', content: ('hello',), cls: None, attrs: {}
# output => <p>hello</p>
print(tag('p', None, 'hello', 'world'))
# output => name: 'p', content: ('hello', 'world'), cls: None, attrs: {}
# output => <p>hello</p>\n<p>world</p>
print(tag('p', 'sidebar', 'hello', 'world'))
# output => name: 'p', content: ('hello', 'world'), cls: 'sidebar', attrs: {}
# output => <p calss="sidebar">hello</p>\n<p calss="sidebar">world</p>

# Keyword arguments not explicitly named in the tag signature are captured by **attrs as a dict
print(tag(content='testing', name='img'))
# output => name: 'img', content: (), cls: None, attrs: {'content': 'testing'}
# output => <img content="testing" />

# Prefixing th my_tag dict with ** passes all its items as seperate arguments which are then bound to the named paramters, with the remaining caught by **attrs.
my_tag = {
    'name': 'img',
    'title': 'Sunset',
    'src': 'sunset.jpg',
    'cls': 'framed'
}
print(tag(**my_tag))
# output => name: 'img', content: (), cls: 'framed', attrs: {'src': 'sunset.jpg', 'title': 'Sunset'}
# output => <img calss="framed" src="sunset.jpg" title="Sunset" />

Function Annotaions

Python 3 provides syntax to attach metadata to the paramters of a function declaration and its return value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# python 3.5
# Annotated clip function
def clip(text:str, max_len:'int > 0'=80) -> str:
    print('text: %s, max_len: %s', (text, max_len))
    return 'hello feifeiyu'

clip.__annotation__
# output => {'max_len': 'int > 0', 'text': <class 'str'>, 'return': <class 'str'>}

# Extracting annotations from the function signature
from inspect import signature
sig = signature(clip)
sig.return_annotation
# output => <class 'str'>

for param in sig.parameters.values():
    note = repr(param.annotation).ljust(13)
    print(note, ':', param.name, '=', param.default)

# output => <class 'str'> : text = <class 'inspect._empty'>
#           'int > 0'     : max_len = 80

Each argument in the function declaration may have an annotation expression preceded by “:”. If there is a default value, the annotation goes between the argument name and the = sign. To annotate the return value, add -> and another expression between th ) and the : at the tail of the function declaration.

The only thing python does with annotations is to store them in the annotation attribute of the function. Nothing else: no checks, enforcement, validation, or any other actions is performed.

Packages for functional programing

The operator module

Often in functional programming it is convenient to use an arithmetic operator as a function.

1
2
3
4
5
6
7
8
from operator import mul
from functools import reduce

def fact(n):
    return reduce(mul, range(1, n+1))

fact(5)
# output => 120

itemgetter: it is function to pick items from sequences.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
metro_data = [
    ('Tokyo', 'JP', 36.933, (35.689722, 139.691667)),
    ('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889)),
    ('Mexico City', 'MX', 20.142, (19.433333, -99.133333)),
]

from operator import itemgetter
cc_name = itemgetter(1, 0)
for city in metro_data:
    print(cc_name(city))

# output => ('JP', 'Tokyo')
# output => ('IN', 'Delhi NCR')
# output => ('MX', 'Mexico City')

for city in sorted(metro_data, key=itemgetter(1))
    print(city)

# output => ('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889))
# output => ('Tokyo', 'JP', 36.933, (35.689722, 139.691667))
# output => ('Mexico City', 'MX', 20.142, (19.433333, -99.133333))

A sibling of itemgetter is attrgetter, which creates functions to extract object attributes by name.

1
2
3
4
5
6
7
dc = {'name': 'feifeiyu'}
# use dir to check attributes of object dict
dir(dc)
# use attrgetter to extract attribute 'pop' of dict
ag = attrgetter('pop')
ag(dc)
# ouput => <function pop>

methodcaller: it creates calls a method by name on the object given as argument

1
2
3
4
5
6
7
from operator import methodcaller
num = range(10)
list_pop = methodcaller('pop')
list_pop(num)
# output => 9
num
# output => [0, 1, 2, 3, 4, 5, 6, 7, 8]

Freezing arguments with functools.partial

functools.partial is a higher-roder function that allows partial application of a function. Given a function, a partial application produces a new callable with some of the arguments of the original function fixed.

1
2
3
4
5
6
7
8
9
10
11
12
from operator import mul
from functools import partial
triple = partial(mul, 3)
triple(7)
# output => 21
list(map(triple, range(10)))
# output => [0, 3, 6, 9, 12, 15, 18, 21, 24, 27]

# tag function 
pic = partial(tag, 'img', cls='pic-frame')
pic(src='feifeiyu.png')
# output => <img class="pic-frame" src="feifeiyu.png" />

END