Python直譯器簡介（5）：深入主迴圈

v7發表於2015-06-30

本文將會帶領大家瞭解 CPython 3.3 中的 Python 直譯器。我們首先一起來看 Python 直譯器的一個簡短的高層概述，然後對直譯器實現過程中的一些有意思的程式碼塊進行深入的探討。我已經把這裡探討的函式名和檔名囊括進來了，你可以在原始碼中找到它們自行閱讀深究。

概述

我們從 Python 虛擬機器（又叫 Python 直譯器）的一個高層概述開始。

Python 虛擬機器有一個棧幀的呼叫棧。一個棧幀包含了給出程式碼塊的資訊和上下文，其中包括最後執行的位元組碼指令、全域性和區域性的名稱空間、異常狀態和呼叫棧幀的引用。每個棧幀有兩個與其相關聯的棧：block 棧和資料棧，其中 block 棧在一些控制流（比如異常處理）中使用。Python 虛擬機器的主要工作就是操作這三個型別的棧。

具體一些，我們假設有下面這樣一段程式碼，直譯器執行到被標記的行。下面便是當前情況下呼叫棧、block 棧以及資料棧的情況。

main.py

def foo():
    x = 1
    def bar(y):
        z = y + 2  # <--- (3) ... and the interpreter is here.
        return z
    return bar(x)  # <--- (2) ... which is returning a call to bar ...
foo()              # <--- (1) We're in the middle of a call to foo ...

main.py

def foo():

x = 1

def bar(y):

z = y + 2 # <--- (3) ... and the interpreter is here.

return z

return bar(x) # <--- (2) ... which is returning a call to bar ...

foo() # <--- (1) We're in the middle of a call to foo ...

main.py

c   ---------------------------
a  | bar Frame                 | -> block stack: []
l  |     (newest)              | -> data stack: [1, 2]
l   ---------------------------
   | foo Frame                 | -> block stack: []
s  |                           | -> data stack: [<Function foo.<locals>.bar at 0x10d389680>, 1]
t   ---------------------------
a  | main (module) Frame       | -> block stack: []
c  |       (oldest)            | -> data stack: [<Function foo at 0x10d3540e0>]
k   ---------------------------

c ---------------------------

a | bar Frame | -> block stack: []

l | (newest) | -> data stack: [1, 2]

l ---------------------------

| foo Frame | -> block stack: []

s | | -> data stack: [<Function foo.<locals>.bar at 0x10d389680>, 1]

t ---------------------------

a | main (module) Frame | -> block stack: []

c | (oldest) | -> data stack: [<Function foo at 0x10d3540e0>]

k ---------------------------

在這一時刻，直譯器在巢狀函式的中間位置呼叫 bar 函式。此時在呼叫棧中有三個棧幀：模組層級的棧幀、foo 函式的棧幀以及 bar 函式的棧幀。當 bar 函式完成動作返回，呼叫棧中與 bar 函式關聯的棧幀將會彈棧。通常每一個模組都會有一個與其對應的擁有新作用域的棧幀，函式呼叫和類定義也是如此。注意，每一次函式呼叫都會建立一個棧幀，在遞迴函式中，每一層的遞迴呼叫都會擁有自己的一個棧幀。

每一個棧幀都有自己的資料棧和 block 棧。獨立的資料棧和 block 棧使直譯器可以中斷或恢復棧幀，這與生成器相似。

這裡的情況示意很清楚了，我們深入到程式碼內部看一下。

堆疊結構物件 frameobj.c 建立一個 ceval.c 檔案中定義的 PyEval_EvalCodeEx 棧幀。這個棧幀在 ceval.c 檔案中執行 PyEval_EvalFrameEx 棧幀。

棧幀都從哪兒來？

ceval.c 檔案中的 PyEval_EvalCodeEx 函式建立了新的棧幀。我們在下面摘錄了執行 code 物件的 PyEval_EvalCodeEx 函式。這個函式首先建立了一個新的棧幀，之後解析命令列引數（如果有的話）。倘若 code 物件是生成器，那麼函式返回新的生成器；否則，棧幀將會執行直到返回，而返回值將被傳遞到上層。

ceval.c

PyObject *
PyEval_EvalCodeEx(PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject **args, int argcount, PyObject **kws, int kwcount,
           PyObject **defs, int defcount, PyObject *kwdefs, PyObject *closure)
{
    PyCodeObject* co = (PyCodeObject*)_co;
    PyFrameObject *f;
    PyObject *retval = NULL;
    PyObject **fastlocals, **freevars;
    PyThreadState *tstate = PyThreadState_GET();

    /* [snip error-checking] */

    f = PyFrame_New(tstate, co, globals, locals);   /* <--------- new frame */
    if (f == NULL)
        return NULL;

    fastlocals = f->f_localsplus;
    freevars = f->f_localsplus + co->co_nlocals;

    /* [snip 150 lines of argument parsing] */

    if (co->co_flags & CO_GENERATOR) { /* <---- if the "it's a generator" flag is set */
        /* [snip] */

        /* Create a new generator that owns the ready to run frame
         * and return that as the value. */
        return PyGen_New(f); /* <----- new generator object */
    }

    retval = PyEval_EvalFrameEx(f,0); /* <---- otherwise, run the frame */

fail: /* Jump here from prelude on failure */
    /* [snip some cleanup] */
    return retval;                   /* <---- return the value from running the frame */
}

PyObject *

PyEval_EvalCodeEx(PyObject *_co, PyObject *globals, PyObject *locals,

PyObject **args, int argcount, PyObject **kws, int kwcount,

PyObject **defs, int defcount, PyObject *kwdefs, PyObject *closure)

{

PyCodeObject* co = (PyCodeObject*)_co;

PyFrameObject *f;

PyObject *retval = NULL;

PyObject **fastlocals, **freevars;

PyThreadState *tstate = PyThreadState_GET();

/* [snip error-checking] */

f = PyFrame_New(tstate, co, globals, locals); /* <--------- new frame */

if (f == NULL)

return NULL;

fastlocals = f->f_localsplus;

freevars = f->f_localsplus + co->co_nlocals;

/* [snip 150 lines of argument parsing] */

if (co->co_flags & CO_GENERATOR) { /* <---- if the "it's a generator" flag is set */

/* [snip] */

/* Create a new generator that owns the ready to run frame

* and return that as the value. */

return PyGen_New(f); /* <----- new generator object */

}

retval = PyEval_EvalFrameEx(f,0); /* <---- otherwise, run the frame */

fail: /* Jump here from prelude on failure */

/* [snip some cleanup] */

return retval; /* <---- return the value from running the frame */

}

大部分情況下是 C 函式 function_call 在呼叫 PyEval_EvalCodeEx。所有 Python 物件被呼叫的時候都會呼叫 function_call。每當一個可呼叫的 Python 物件被呼叫，都會寫入一個 code 物件用來建立棧幀。

funcobject.c

static PyObject *
function_call(PyObject *func, PyObject *arg, PyObject *kw)
{
    PyObject *result;
    PyObject *argdefs;
    PyObject *kwtuple = NULL;
    PyObject **d, **k;
    Py_ssize_t nk, nd;

    /* [snip 30 lines of argument parsing] */

    result = PyEval_EvalCodeEx(
        PyFunction_GET_CODE(func),
        PyFunction_GET_GLOBALS(func), (PyObject *)NULL,
        &PyTuple_GET_ITEM(arg, 0), PyTuple_GET_SIZE(arg),
        k, nk, d, nd,
        PyFunction_GET_KW_DEFAULTS(func),
        PyFunction_GET_CLOSURE(func));

    return result;
}

static PyObject *

function_call(PyObject *func, PyObject *arg, PyObject *kw)

{

PyObject *result;

PyObject *argdefs;

PyObject *kwtuple = NULL;

PyObject **d, **k;

Py_ssize_t nk, nd;

/* [snip 30 lines of argument parsing] */

result = PyEval_EvalCodeEx(

PyFunction_GET_CODE(func),

PyFunction_GET_GLOBALS(func), (PyObject *)NULL,

&PyTuple_GET_ITEM(arg, 0), PyTuple_GET_SIZE(arg),

k, nk, d, nd,

PyFunction_GET_KW_DEFAULTS(func),

PyFunction_GET_CLOSURE(func));

return result;

}

上面說大部分情況下是 function_call 在呼叫 PyEval_EvalCodeEx。而 PyEval_EvalCodeEx 也可以被 entry point 呼叫，比如 pythonrun.c 中的 run_pyc_file 以及 import.c 中的 exec_code_in_module。這些函式很相似，區別只是在於它們取得 code 物件的方式（通過編譯還是通過讀檔案）和執行的環境（比如名稱空間不同）。

我們回到 PyEval_EvalFrameEx。這個函式大約有2400行程式碼，佔了 ceval.c 檔案的大部分；其中1500行程式碼是一個龐大的 switch 宣告。我的那篇《 1,500 line switch statement powering your Python》提到的就是它。PyEval_EvalFrameEx 佔用了一個單獨的棧幀，並且會執行直到它返回。

在本系列文章的第 3 篇我們介紹了位元組碼。對直譯器來講，位元組碼是一序列位元組指令。我們回到本篇開頭的例子，下面列出了這段程式碼和函式中 code 物件的詳細拆解。

PY3 >>> def foo():
PY3 ...     x = 1
PY3 ...     def bar(y):
PY3 ...         z = y + 2
PY3 ...         return z
PY3 ...     return bar(x)
PY3 ...
PY3 >>> dis.dis(foo)
  2           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)

  3           6 LOAD_CONST               2 (<code object bar at 0x107b548a0, file "<stdin>", line 3>)
              9 LOAD_CONST               3 ('foo.<locals>.bar')
             12 MAKE_FUNCTION            0
             15 STORE_FAST               1 (bar)

  6          18 LOAD_FAST                1 (bar)
             21 LOAD_FAST                0 (x)
             24 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             27 RETURN_VALUE

PY3 >>> bar_code_obj = foo.__code__.co_consts[2]
PY3 >>> dis.dis(bar_code_obj)
  4           0 LOAD_FAST                0 (y)
              3 LOAD_CONST               1 (2)
              6 BINARY_ADD
              7 STORE_FAST               1 (z)

  5          10 LOAD_FAST                1 (z)
             13 RETURN_VALUE

PY3 >>> def foo():

PY3 ... x = 1

PY3 ... def bar(y):

PY3 ... z = y + 2

PY3 ... return z

PY3 ... return bar(x)

PY3 ...

PY3 >>> dis.dis(foo)

2 0 LOAD_CONST 1 (1)

3 STORE_FAST 0 (x)

3 6 LOAD_CONST 2 (<code object bar at 0x107b548a0, file "<stdin>", line 3>)

9 LOAD_CONST 3 ('foo.<locals>.bar')

12 MAKE_FUNCTION 0

15 STORE_FAST 1 (bar)

6 18 LOAD_FAST 1 (bar)

21 LOAD_FAST 0 (x)

24 CALL_FUNCTION 1 (1 positional, 0 keyword pair)

27 RETURN_VALUE

PY3 >>> bar_code_obj = foo.__code__.co_consts[2]

PY3 >>> dis.dis(bar_code_obj)

4 0 LOAD_FAST 0 (y)

3 LOAD_CONST 1 (2)

6 BINARY_ADD

7 STORE_FAST 1 (z)

5 10 LOAD_FAST 1 (z)

13 RETURN_VALUE

PyEval_EvalFrameEx 從位元組碼中的第一個位元組開始執行，在本例中，從 foo 函式位元組碼中的 LOAD_CONST 位元組開始，並且去找那個龐大 switch 中對應的 case。當執行完 switch 中找到的操作指令，棧幀將移動到下一個相關操作碼繼續整個程式。在一些地方，操作碼會中止迴圈通過 goto 跳出switch，通過下面 RETURN_VALUE 示例。

為使你更好地理解它如何工作，我在下面列出了 PyEval_EvalFrameEx 函式中的一段摘錄。不過像往常一樣，我更希望你去閱讀 CPython 的完整程式碼。

ceval.c

/* Interpreter main loop */

PyObject *
PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
{
    PyObject **stack_pointer;  /* Next free slot in value stack */
    unsigned char *next_instr;
    int opcode;        /* Current opcode */
    int oparg;         /* Current opcode argument, if any */
    enum why_code why; /* Reason for block stack unwind */
    PyObject **fastlocals, **freevars;
    PyObject *retval = NULL;            /* Return value */
    PyThreadState *tstate = PyThreadState_GET();
    PyCodeObject *co;

    unsigned char *first_instr;
    PyObject *names;
    PyObject *consts;

#define TARGET(op) 
    case op:
#define DISPATCH() continue
#define FAST_DISPATCH() goto fast_next_opcode

#define NEXTOP()        (*next_instr++)

/* Stack manipulation macros */
#define STACK_LEVEL()     ((int)(stack_pointer - f->f_valuestack))
#define TOP()             (stack_pointer[-1])
#define PUSH(v)     (*stack_pointer++ = (v))
#define POP()       (*--stack_pointer)

/* Start of code */

    /* push frame */
    tstate->frame = f;

    co = f->f_code;
    names = co->co_names;
    consts = co->co_consts;
    fastlocals = f->f_localsplus;
    freevars = f->f_localsplus + co->co_nlocals;
    first_instr = (unsigned char*) PyBytes_AS_STRING(co->co_code);
    next_instr = first_instr + f->f_lasti + 1;
    stack_pointer = f->f_stacktop;
    f->f_stacktop = NULL;       /* remains NULL unless yield suspends frame */
    f->f_executing = 1;

    why = WHY_NOT;

    for (;;) {

    fast_next_opcode:
        f->f_lasti = INSTR_OFFSET();

        /* Extract opcode and argument */

        opcode = NEXTOP();
        oparg = 0;
        if (HAS_ARG(opcode))
            oparg = NEXTARG();

        /* Main switch on opcode */

        switch (opcode) {
            /* [snip 1,500 lines of switch, about 100 cases] */
            /* [three cases shown below] */

        TARGET(BINARY_ADD) {
            PyObject *right = POP();
            PyObject *left = TOP();
            PyObject *sum;
            sum = PyNumber_Add(left, right);
            Py_DECREF(left);
            Py_DECREF(right);
            SET_TOP(sum);
            if (sum == NULL)
                goto error;
            DISPATCH();
        }

        TARGET(LOAD_CONST) {
            PyObject *value = GETITEM(consts, oparg);
            Py_INCREF(value);
            PUSH(value);
            FAST_DISPATCH();
        }

        TARGET(RETURN_VALUE) {
            retval = POP();
            why = WHY_RETURN;
            goto fast_block_end;
        }

        } /* end switch */

    /* complex block cleanup code here */

    } /* main loop */

/* pop frame */
exit_eval_frame:
    f->f_executing = 0;
    tstate->frame = f->f_back;

    return retval;
}

100

101

102

103

104

105

106

107

/* Interpreter main loop */

PyObject *

PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)

{

PyObject **stack_pointer; /* Next free slot in value stack */

unsigned char *next_instr;

int opcode; /* Current opcode */

int oparg; /* Current opcode argument, if any */

enum why_code why; /* Reason for block stack unwind */

PyObject **fastlocals, **freevars;

PyObject *retval = NULL; /* Return value */

PyThreadState *tstate = PyThreadState_GET();

PyCodeObject *co;

unsigned char *first_instr;

PyObject *names;

PyObject *consts;

#define TARGET(op)

case op:

#define DISPATCH() continue

#define FAST_DISPATCH() goto fast_next_opcode

#define NEXTOP() (*next_instr++)

/* Stack manipulation macros */

#define STACK_LEVEL() ((int)(stack_pointer - f->f_valuestack))

#define TOP() (stack_pointer[-1])

#define PUSH(v) (*stack_pointer++ = (v))

#define POP() (*--stack_pointer)

/* Start of code */

/* push frame */

tstate->frame = f;

co = f->f_code;

names = co->co_names;

consts = co->co_consts;

fastlocals = f->f_localsplus;

freevars = f->f_localsplus + co->co_nlocals;

first_instr = (unsigned char*) PyBytes_AS_STRING(co->co_code);

next_instr = first_instr + f->f_lasti + 1;

stack_pointer = f->f_stacktop;

f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */

f->f_executing = 1;

why = WHY_NOT;

for (;;) {

fast_next_opcode:

f->f_lasti = INSTR_OFFSET();

/* Extract opcode and argument */

opcode = NEXTOP();

oparg = 0;

if (HAS_ARG(opcode))

oparg = NEXTARG();

/* Main switch on opcode */

switch (opcode) {

/* [snip 1,500 lines of switch, about 100 cases] */

/* [three cases shown below] */

TARGET(BINARY_ADD) {

PyObject *right = POP();

PyObject *left = TOP();

PyObject *sum;

sum = PyNumber_Add(left, right);

Py_DECREF(left);

Py_DECREF(right);

SET_TOP(sum);

if (sum == NULL)

goto error;

DISPATCH();

}

TARGET(LOAD_CONST) {

PyObject *value = GETITEM(consts, oparg);

Py_INCREF(value);

PUSH(value);

FAST_DISPATCH();

}

TARGET(RETURN_VALUE) {

retval = POP();

why = WHY_RETURN;

goto fast_block_end;

}

} /* end switch */

/* complex block cleanup code here */

} /* main loop */

/* pop frame */

exit_eval_frame:

f->f_executing = 0;

tstate->frame = f->f_back;

return retval;

}

在第三篇中，我們看到 BINARY_ADD 沒有引數，從上面的 dis 的第四行輸出來看並沒有命令列引數。這有點奇怪，我們原本希望看到一個帶有兩個引數的二進位制函式。現在通過看直譯器的情況，我們便知道發生了什麼：這兩個引數在棧幀的資料棧棧頂。下面我給出了 bar 函式執行時候資料棧的情況。

    data: [] <-- the data stack starts out empty
4 0 LOAD_FAST 0 (y)
    data: [1] <-- y has the value 1 when bar is called
3 LOAD_CONST               1 (2)
    data: [1, 2]
6 BINARY_ADD
    data: [3] <--- BINARY_ADD adds the top two things on the stack
                                 and pushes the answer onto the stack