关于gunicorn与异步兼容性问题：AttributeError: module 'select' has no attribute 'epoll'
背景：
- 介绍：
-
异常：
[2024-01-15 10:22:16 +0800] [31655] [ERROR] Exception in worker process Traceback (most recent call last): File "/home/venv/lib/python3.10/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process super().init_process() File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 133, in init_process self.load_wsgi() File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi self.wsgi = self.app.wsgi() File "/home/venv/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/home/venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 49, in load return self.load_wsgiapp() File "/home/venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp return util.import_app(self.app_uri) File "/home/venv/lib/python3.10/site-packages/gunicorn/util.py", line 331, in import_app mod = importlib.import_module(module) File "/home/nlu_common/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed 。。。。。。包含很多具体代码，没有意义。。。。。。 File "/home/venv/lib/python3.10/site-packages/dns/resolver.py", line 38, in <module> import dns.query File "/home/venv/lib/python3.10/site-packages/dns/query.py", line 52, in <module> import httpx File "/home/venv/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module> from ._api import delete, get, head, options, patch, post, put, request, stream File "/home/venv/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module> from ._client import Client File "/home/venv/lib/python3.10/site-packages/httpx/_client.py", line 30, in <module> from ._transports.default import AsyncHTTPTransport, HTTPTransport File "/home/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 30, in <module> import httpcore File "/home/venv/lib/python3.10/site-packages/httpcore/__init__.py", line 1, in <module> from ._api import request, stream File "/home/venv/lib/python3.10/site-packages/httpcore/_api.py", line 5, in <module> from ._sync.connection_pool import ConnectionPool File "/home/venv/lib/python3.10/site-packages/httpcore/_sync/__init__.py", line 1, in <module> from .connection import HTTPConnection File "/home/venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 12, in <module> from .._synchronization import Lock File "/home/venv/lib/python3.10/site-packages/httpcore/_synchronization.py", line 13, in <module> import trio File "/home/venv/lib/python3.10/site-packages/trio/__init__.py", line 22, in <module> from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED # isort: split File "/home/venv/lib/python3.10/site-packages/trio/_core/__init__.py", line 21, in <module> from ._local import RunVar, RunVarToken File "/home/venv/lib/python3.10/site-packages/trio/_core/_local.py", line 9, in <module> from . import _run File "/home/venv/lib/python3.10/site-packages/trio/_core/_run.py", line 2787, in <module> from ._io_epoll import ( File "/home/venv/lib/python3.10/site-packages/trio/_core/_io_epoll.py", line 202, in <module> class EpollIOManager: File "/home/venv/lib/python3.10/site-packages/trio/_core/_io_epoll.py", line 203, in EpollIOManager _epoll: select.epoll = attr.ib(factory=select.epoll) AttributeError: module 'select' has no attribute 'epoll'. Did you mean: 'poll'? [2024-01-15 10:22:16 +0800] [31655] [INFO] Worker exiting (pid: 31655) [2024-01-15 10:22:17 +0800] [31652] [INFO] Shutting down: Master [2024-01-15 10:22:17 +0800] [31652] [INFO] Reason: Worker failed to boot.- 表现：AttributeError: module 'select' has no attribute 'epoll'. Did you mean: 'poll'?
项目介绍
- 框架使用Flask + flask-sockets 搭建
- 内部业务使用异步转同步，例如：asnyc、httpx等
- 交互协议http、grpc、websocket
- 部署：gunicorn、gevent
问题分析
-
部署方式分析：
-
非Gunicorn部署（window & linux）
from http_server.handle import app from gevent import pywsgi from geventwebsocket.handler import WebSocketHandler server = pywsgi.WSGIServer(('0.0.0.0', 50003), app, handler_class=WebSocketHandler) server.serve_forever() 可正常提供服务 epoll支持：Python官方文档 select.epoll([suzegubt=-1]) (only supported on Linux 2.5.44 and newer.) Returns an edge polling object, which can be used as Edge or Level Triggered interface for I/O events; see section Edge and Level Trigger Polling (epoll) Objects below for the methods supported by epolling objects. - Gunicorn部署：如背景介绍
- 结论：上述问题与gunicorn兼容性问题：gunicorn（gevent） <=> trio(epoll) ......> httpcore ......> httpx ......>dnspython
-
非Gunicorn部署（window & linux）
-
资源搜索分析：
-
Gunicorn（Gevent） 与 trio 存在兼容性问题：https://github.com/gevent/gevent/issues/2008
- 无直接解决方案
-
通过配置
from gevent import monkey monkey.patch_all(select=False)
export EVENTLET_NO_GREENDNS=yes
-
- 对结果无效
-
Gunicorn（Gevent） 与 trio 存在兼容性问题：https://github.com/gevent/gevent/issues/2008
-
版本兼容性处理-降级
- 当前版本
-
gevent 23.9.1 gevent-websocket 0.10.1 gunicorn 21.2.0 dnspython 2.4.2 httpcore 0.17.3 httpx 0.24.1 trio 0.24.0 trio-websocket 0.11.1
-
- 经过相关包降级：
- 当前版本
-
-
- dsnpython => 2.3.0 无用
- trio => 0.22.2 升降级无用
- httpcore => 0.17.1 解决
-
结论：
AttributeError: module 'select' has no attribute 'epoll' 异常原因：- gunicorn部署表现：trio 与 gunicorn（gevent）的不兼容
- 解决办法：使用httpcore低版本 <= 0.17.12