pip install mysqlclient
环境: ubuntu, mysql5.7.37, python3.9.12, pip21.2.4
1. 卸载原有airflow,并卸载;
pip uninstall apache-airflow
查找airflow相关文件
find / -name airflow
删除原airflow数据文件夹
rm -rf airflow
删除或修改环境变量 AIRFLOW_HOME, AIRFLOWPATH
vi /etc/profile
source /etc/profile
2. 设置环境变量
vi /etc/profile
安装包的路径
#airflow export AIRFLOW_HOME=/home/dev/airflow2.2.5
wq!保存退出,并使之生效
source /etc/profile
3. pip下载对应版本airflow
pip --default-timeout=100 install apache-airflow==2.2.5
4.默认sqlite存储并验证
4.1 查看安装信息
airflow version 命令查看版本信息
(base) root@ecs-df00:~# airflow version 2.2.5
airflow info 命令查看工具信息
(base) root@ecs-df00:~# airflow info
/home/dev/anaconda3/lib/python3.9/site-packages/airflow/configuration.py:276: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version):
Apache Airflow
version | 2.2.5
executor | SequentialExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | sqlite:////home/dev/airflow2.2.5/airflow.db
dags_folder | /home/dev/airflow2.2.5/dags
plugins_folder | /home/dev/airflow2.2.5/plugins
base_log_folder | /home/dev/airflow2.2.5/logs
remote_base_log_folder |
System info
OS | Linux
architecture | x86_64
uname | uname_result(system='Linux', node='ecs-df00', release='5.4.0-100-generic', version='#113-Ubuntu SMP Thu Feb 3
| 18:43:29 UTC 2022', machine='x86_64')
locale | ('en_US', 'UTF-8')
python_version | 3.9.12 (main, Apr 5 2022, 06:56:58) [GCC 7.5.0]
python_location | /home/dev/anaconda3/bin/python
Tools info
git | git version 2.25.1
ssh | OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
kubectl | NOT AVAILABLE
gcloud | NOT AVAILABLE
cloud_sql_proxy | NOT AVAILABLE
mysql | mysql Ver 14.14 Distrib 5.7.37, for Linux (x86_64) using EditLine wrapper
sqlite3 | 3.38.2 2022-03-26 13:51:10 d33c709cc0af66bc5b6dc6216eba9f1f0b40960b9ae83694c986fbf4c1d6f08f
psql | NOT AVAILABLE
Paths info
airflow_home | /home/dev/airflow2.2.5
system_path | /home/dev/anaconda3/bin:/home/dev/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:
| /usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/jdk-11/bin:/etc/mysql/bin:/home/dev/anaconda3/bin
python_path | /home/dev/anaconda3/bin:/home/dev/anaconda3/lib/python39.zip:/home/dev/anaconda3/lib/python3.9:/home/dev/anaconda3
| /lib/python3.9/lib-dynload:/home/dev/anaconda3/lib/python3.9/site-packages:/home/dev/airflow2.2.5/dags:/home/dev/a
| irflow2.2.5/config:/home/dev/airflow2.2.5/plugins
airflow_on_path | True
Providers info
apache-airflow-providers-common-sql | 1.0.0
apache-airflow-providers-ftp | 3.1.0
apache-airflow-providers-http | 4.0.0
apache-airflow-providers-imap | 3.0.0
初始化成功后会在airflow目录生成 airflow.cfg , logs 和 webserver_config.py 文件(文件夹);
(base) root@ecs-df00:/home/dev/airflow2.2.5# ll total 64 drwxr-xr-x 3 root root 4096 Aug 19 01:18 ./ drwxr-xr-x 5 root root 4096 Aug 19 01:10 ../ -rw-r--r-- 1 root root 44548 Aug 19 01:18 airflow.cfg drwxr-xr-x 3 root root 4096 Aug 19 01:18 logs/ -rw-r--r-- 1 root root 4695 Aug 19 01:18 webserver_config.py
4.2 初始化数据库
airflow db init
创建用户
airflow users create --username 账号 --firstname 姓 --lastname 名字 --role Admin --email [email protected]
(base) root@ecs-df00:/home/dev/airflow2.2.5# airflow users create --username airflow --firstname liu --lastname yifei --role Admin --email [email protected] /home/dev/anaconda3/lib/python3.9/site-packages/airflow/configuration.py:276: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version): [2022-08-22 01:26:40,019] {manager.py:779} WARNING - No user yet created, use flask fab command to do it. [2022-08-22 01:26:40,274] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create Admin Password: Repeat for confirmation: [2022-08-22 01:26:49,790] {manager.py:214} INFO - Added user airflow User "airflow" created with role "Admin"
4.3 启动服务
cd /home/dev/anaconda3/bin
nohup airflow webserver -D -p 8090 nohup airflow scheduler -D
4.4 webUI页面
http://ip:8090
5 mysql存储数据,并验证
默认环境使用的是sqlite存储数据,生产环境使用mysql/postgresql等数据库存储,这里以mysql为例
5.1 修改配置文件 $AIRFLOW_HOME/airflow.cfg
executor=LocalExecutor sql_alchemy_conn=mysql://root:123456@localhost:33066/airflow default_timezone=Asia/Shanghai default_ui_timezone=Asia/Shanghai web_server_port=8090
5.2 mysql数据库上创建数据库
mysql -h localhost -u root -p create database airflow; create user 'airflow'@'%' identified by 'admin'; grant all privileges on *.* to 'root'@'%' identified by '123456'; flush privileges;
exit;
5.3 安装mysql客户端
pip install mysqlclient
5.4 初始化数据库
airflow db init
创建用户
airflow users create --username 账号 --firstname 姓 --lastname 名字 --role Admin --email [email protected]
Initialization done (base) root@ecs-df00:/home/dev/anaconda3/bin# airflow users create --username airflow --firstname liu --lastname yifei --role Admin --email [email protected] [2022-08-25 00:57:59,636] {manager.py:779} WARNING - No user yet created, use flask fab command to do it. [2022-08-25 00:57:59,724] {manager.py:496} INFO - Created Permission View: menu access on List Users [2022-08-25 00:57:59,731] {manager.py:558} INFO - Added Permission menu access on List Users to role Admin [2022-08-25 00:57:59,746] {manager.py:496} INFO - Created Permission View: menu access on Security [2022-08-25 00:57:59,753] {manager.py:558} INFO - Added Permission menu access on Security to role Admin [2022-08-25 00:57:59,790] {manager.py:496} INFO - Created Permission View: menu access on List Roles [2022-08-25 00:57:59,796] {manager.py:558} INFO - Added Permission menu access on List Roles to role Admin [2022-08-25 00:57:59,817] {manager.py:496} INFO - Created Permission View: can read on User Stats Chart [2022-08-25 00:57:59,823] {manager.py:558} INFO - Added Permission can read on User Stats Chart to role Admin [2022-08-25 00:57:59,836] {manager.py:496} INFO - Created Permission View: menu access on User's Statistics [2022-08-25 00:57:59,842] {manager.py:558} INFO - Added Permission menu access on User's Statistics to role Admin [2022-08-25 00:57:59,877] {manager.py:496} INFO - Created Permission View: menu access on Base Permissions [2022-08-25 00:57:59,884] {manager.py:558} INFO - Added Permission menu access on Base Permissions to role Admin [2022-08-25 00:57:59,915] {manager.py:496} INFO - Created Permission View: can read on View Menus [2022-08-25 00:57:59,922] {manager.py:558} INFO - Added Permission can read on View Menus to role Admin [2022-08-25 00:57:59,937] {manager.py:496} INFO - Created Permission View: menu access on Views/Menus [2022-08-25 00:57:59,943] {manager.py:558} INFO - Added Permission menu access on Views/Menus to role Admin [2022-08-25 00:57:59,975] {manager.py:496} INFO - Created Permission View: can read on Permission Views [2022-08-25 00:57:59,980] {manager.py:558} INFO - Added Permission can read on Permission Views to role Admin [2022-08-25 00:57:59,993] {manager.py:496} INFO - Created Permission View: menu access on Permission on Views/Menus [2022-08-25 00:57:59,999] {manager.py:558} INFO - Added Permission menu access on Permission on Views/Menus to role Admin [2022-08-25 00:58:00,026] {manager.py:496} INFO - Created Permission View: can get on MenuApi [2022-08-25 00:58:00,032] {manager.py:558} INFO - Added Permission can get on MenuApi to role Admin [2022-08-25 00:58:00,130] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create Admin [2022-08-25 00:58:00,507] {manager.py:496} INFO - Created Permission View: menu access on Providers [2022-08-25 00:58:00,512] {manager.py:558} INFO - Added Permission menu access on Providers to role Admin [2022-08-25 00:58:00,576] {manager.py:496} INFO - Created Permission View: can create on XComs [2022-08-25 00:58:00,582] {manager.py:558} INFO - Added Permission can create on XComs to role Admin Password: Repeat for confirmation: [2022-08-25 00:58:25,117] {manager.py:214} INFO - Added user airflow User "airflow" created with role "Admin"
记录下账号密码;
5.5 后端启动服务
cd /home/dev/anaconda3/bin
nohup airflow webserver -D -p 8090 nohup airflow scheduler -D
安装过程可能遇到的问题:
1. 错误提示: pip's dependency resolver does not currently take into account all the packages that are installed.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. spyder 5.1.5 requires pyqt5<5.13, which is not installed. spyder 5.1.5 requires pyqtwebengine<5.13, which is not installed. conda-repo-cli 1.0.4 requires pathlib, which is not installed.
解决方法: 安装对应版本的包
pip --default-timeout=100 install pyqt==5.10 pip --default-timeout=100 install pyqtwebengine==5.12 pip install pathlib
2. 安装mysql客户端时提示"OSError: mysql_config not found"
Downloading mysqlclient-1.3.13.tar.gz (90 kB)
|████████████████████████████████| 90 kB 19 kB/s
ERROR: Command errored out with exit status 1:
command: /home/dev/anaconda3/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gz83emtr/mysqlclient_326fa202c0a04195bee596413d30353a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gz83emtr/mysqlclient_326fa202c0a04195bee596413d30353a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-715u_941
cwd: /tmp/pip-install-gz83emtr/mysqlclient_326fa202c0a04195bee596413d30353a/
Complete output (10 lines):
/bin/sh: 1: mysql_config: not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-gz83emtr/mysqlclient_326fa202c0a04195bee596413d30353a/setup.py", line 18, in <module>
metadata, options = get_config()
File "/tmp/pip-install-gz83emtr/mysqlclient_326fa202c0a04195bee596413d30353a/setup_posix.py", line 53, in get_config
libs = mysql_config("libs_r")
File "/tmp/pip-install-gz83emtr/mysqlclient_326fa202c0a04195bee596413d30353a/setup_posix.py", line 28, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
OSError: mysql_config not found
----------------------------------------
解决方法:
apt-get install libmysqlclient-dev
问题: 在airflow db init 命令时出现错误: Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
解决方法:
进入数据库执行命令
set global explicit_defaults_for_timestamp =1;
(base) root@ecs-df00:/home/dev/anaconda3/bin# mysql -h localhost -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 22 Server version: 5.7.37 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> set global explicit_defaults_for_timestamp =1; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like '%timestamp%'; +---------------------------------+-------+ | Variable_name | Value | +---------------------------------+-------+ | explicit_defaults_for_timestamp | ON | | log_timestamps | UTC | +---------------------------------+-------+ 2 rows in set (0.00 sec) mysql>
问题: 重装airflow后日志出现提示 airflow-webserver-monitor.pid is already locked
解决方法: 删除airflow-webserver-monitor.pid 并重启;
ps -ef | grep -e "airflow webserver -D" -e "airflow-webserver" | grep -v grep | awk '{print $2}' | xargs kill -9
rm -rf airflow-scheduler.pid airflow-webserver-monitor.pid
nohup airflow webserver -D -p 8090
nohup airflow scheduler -D
标签：00,airflow,部署,08,py,manager,2022,Linux From： https://www.cnblogs.com/chencoolandclear/p/16600659.html