Python並行処理：注意すべきポイント

An exploration of threads, processes, and coroutines in Python, with interesting examples that illuminate the differences between each.

As a data scientist who is spending more time on software engineering, I was recently forced to confront an ugly gap in my knowledge of Python: concurrency. To be honest, I never completely understood how the terms async, threads, pools and coroutines were different and how these mechanisms could work together. Every time I tried to learn about the subject, the examples were a bit too abstract for me, and I hard time internalizing how everything worked.

This changed when a friend of mine2 recommended a live coding talk by David Beazley, an accomplished Python educator.

Because of restrictions with this YouTube video, I couldn’t embed the video in this article, so you will have to open it in a different window.

This talk is incredibly intimidating at first. Not only is it coded live from scratch, but it also jumps immediately into socket programming, something that I had never encountered as a data scientist. However, if you go through it slowly and understand all the components (as we do in this blog post) it turns out to be the best educational material on Python concurrency I have ever encountered. This blog post documents what I learned along the way so others can benefit, too.

Before getting started, David sets up the following infrastructure that is used to demonstrate concurrency.

To demonstrate concurrency, it is useful to create a task that can saturate your CPU (such as mathematical operations) for a noticeable period of time. David uses a function that computes a Fibonacci number.

1 2 3 4 #fib.py def fib(n): if n <= 2: return 1 else: return fib(n-1) + fib(n-2) This function takes much longer for large inputs versus smaller inputs3, which allows us to profile different workloads.

A web server is one of the best ways to illustrate different types of concurrency. However, to really demonstrate how things work it is useful to use something that is sufficiently low level enough to see how all the pieces work. For this, David sets up a web server using socket programming. If you aren&rsquo;t familiar with socket programming, I&rsquo;ll explain the important bits below, but feel free to dive deeper with this tutorial later if you like.

To begin with, David starts with the below code (I&rsquo;ve highlighted the most interesting bits):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # server-1.py from socket import * from fib import fib def fib_server(address): sock = socket(AF_INET, SOCK_STREAM) sock.setsockopt(SOL_SOCKET, SO_REUSEADDR,1) sock.bind(address) sock.listen(5) while True: client,addr = sock.accept() # waits for a connection to be established print(&#34;Connection&#34;, addr) fib_handler(client) # passes the client to a handler which will listen for input data. def fib_handler(client): while True: req = client.recv(100) # waits for data that sent by the client. if not req: break result = fib(int(req)) resp = str(result).encode('ascii') + b'\n' client.send(resp) # sends data back to the client. print(&#34;Closed&#34;) fib_server(('', 25000)) Here is an explanation of this code:

Lines 6-9 are socket programming boilerplate. It&rsquo;s ok to just take this for granted as a reasonable way to set up a socket server. This also matches the the tutorial I linked to above.

Line 11 waits for an incoming connection from a client. Once a connection is made, the server can begin receiving data from a client. The code will stop execution on this line until a connection is made.

Line 13: Once a connection is established, the client object is passed to a function which can handle data sent by the client.

Line 17: waits for data to be sent by the client. The code will stop execution on this line until data is received from the client.

Line 21: The server sends a response back to the client. The code could stop execution on this line if the send buffers are full, but unlikely in this toy example.

Testing the non-concurrent code

In the above example, the server will only be able to accept a connection from a single client, because the call to fib_handler will never return (because it will run in an infinite loop unless a kill signal is received). This means that sock.accept() can only be called once.

You can test this out by first running the server:

1 python server-1.py Then establish a client:

1 telnet localhost 25000 You can type numbers in as David does in his video and verifies that fibonacci numbers are returned. However, if you try to connect with another client at the same time from a different terminal session:

1 telnet localhost 25000 You will notice that the second client just hangs and doesn&rsquo;t return anything from the server. This is because the server is only able to accept a single connection. Next, we explore how to tackle this issue.

We can solve this issue with threads. You can add threads to the handler so that mo