技术背景
假设我们在一个局域网内有多台工作站（不是服务器），那么有没有一个简单的方案可以实现一个小集群，提交分布式的任务呢？Ray为我们提供了一个很好的解决方案，允许你通过conda和Python灵活的构建集群环境，并提交分布式的任务。其基本架构为：
那么本文简单的介绍一下Ray的安装与基本使用。
安装
由于是一个Python的框架，Ray可以直接使用pip进行安装和管理：
$ python3 -m pip install ray[default]
但是需要注意的是，在所有需要构建集群的设备上，需要统一Python和Ray的版本，因此建议先使用conda创建同样的虚拟环境之后，再安装统一版本的ray。否则在添加集群节点的时候就有可能出现如下问题：
RuntimeError: Version mismatch: The cluster was started with:
Ray: 2.7.2
Python: 3.7.13
This process on node 172.17.0.2 was started with:
Ray: 2.7.2
Python: 3.7.5
启动和连接服务
一般在配置集群的时候可以先配置下密钥登陆：
$ ssh-keygen -t rsa
$ ssh-copy-id user_name@ip_address
就这么两步，就可以配置远程服务器ssh免密登陆（配置的过程中有可能需要输入一次密码）。然后在主节点（配置一个master节点）启动ray服务：
$ ray start --head --dashboard-host='0.0.0.0' --dashboard-port=8265
Usage stats collection is enabled. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.
Local node IP: xxx.xxx.xxx.xxx
--------------------
Ray runtime started.
--------------------
Next steps
To add another node to this Ray cluster, run
ray start --address='xxx.xxx.xxx.xxx:6379'
To connect to this Ray cluster:
import ray
ray.init()
To submit a Ray job using the Ray Jobs CLI:
RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . -- python my_script.py
See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
for more information on submitting Ray jobs to the Ray cluster.
To terminate the Ray runtime, run
ray stop
To view the status of the cluster, use
ray status
To monitor and debug Ray, view the dashboard at
xxx.xxx.xxx.xxx:8265
If connection to the dashboard fails, check your firewall settings and network configuration.
这就启动完成了，并给你指示了下一步的操作，例如在另一个节点上配置添加到集群中，可以使用指令：
$ ray start --address='xxx.xxx.xxx.xxx:6379'
但是前面提到了，这里要求Python和Ray版本要一致，如果版本不一致就会出现这样的报错：
RuntimeError: Version mismatch: The cluster was started with:
Ray: 2.7.2
Python: 3.7.13
This process on node 172.17.0.2 was started with:
Ray: 2.7.2
Python: 3.7.5
到这里其实Ray集群就已经部署完成了，非常的简单方便。
基础使用
我们先用一个最简单的案例来测试一下：
# test_ray.py
import os
import ray
ray.init()
print('''This cluster consists of
{} nodes in total
{} CPU resources in total
'''.format(len(ray.nodes()), ray.cluster_resources()['CPU']))
这个Python脚本打印了远程节点的计算资源，那么我们可以用这样的方式去提交一个本地的job：
$ RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . -- python test_ray.py
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-27 07:35:10,751 INFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_4b79155b5de665ce.zip.
2024-08-27 07:35:10,751 INFO packaging.py:518 -- Creating a file package for local directory '.'.
-------------------------------------------------------
Job 'raysubmit_7Uqy8LjP4dxjZxGa' submitted successfully
-------------------------------------------------------
Next steps
Query the logs of the job:
ray job logs raysubmit_7Uqy8LjP4dxjZxGa
Query the status of the job:
ray job status raysubmit_7Uqy8LjP4dxjZxGa
Request the job to be stopped:
ray job stop raysubmit_7Uqy8LjP4dxjZxGa
Tailing logs until the job exits (disable with --no-wait):
2024-08-27 15:35:14,079 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-27 15:35:14,079 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-27 15:35:14,103 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265
This cluster consists of
1 nodes in total
48.0 CPU resources in total
------------------------------------------
Job 'raysubmit_7Uqy8LjP4dxjZxGa' succeeded
------------------------------------------
这里的信息说明，远程的集群只有一个节点，该节点上有48个可用的CPU核资源。这些输出信息不仅可以在终端窗口上看到，还可以从这里给出的dashboard链接里面看到更加详细的任务管理情况：
这里也顺便提交一个输出软件位置信息的指令，确认下任务是在远程执行而不是在本地执行：
import ray
ray.init()
import numpy as np
print (np.__file__)
返回的日志为：
$ RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . -- python test_ray.py
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-27 07:46:10,645 INFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_5bba1a7144beb522.zip.
2024-08-27 07:46:10,658 INFO packaging.py:518 -- Creating a file package for local directory '.'.
-------------------------------------------------------
Job 'raysubmit_kQ3XgE4Hxp3dkmuU' submitted successfully
-------------------------------------------------------
Next steps
Query the logs of the job:
ray job logs raysubmit_kQ3XgE4Hxp3dkmuU
Query the status of the job:
ray job status raysubmit_kQ3XgE4Hxp3dkmuU
Request the job to be stopped:
ray job stop raysubmit_kQ3XgE4Hxp3dkmuU
Tailing logs until the job exits (disable with --no-wait):
2024-08-27 15:46:12,456 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-27 15:46:12,457 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-27 15:46:12,470 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265
/home/dechin/anaconda3/envs/mindspore-latest/lib/python3.7/site-packages/numpy/__init__.py
------------------------------------------
Job 'raysubmit_kQ3XgE4Hxp3dkmuU' succeeded
------------------------------------------
$ python3 -m pip show numpy
Name: numpy
Version: 1.21.6
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD
Location: /usr/local/python-3.7.5/lib/python3.7/site-packages
Requires:
Required-by: CyFES, h5py, hadder, matplotlib, mindinsight, mindspore, mindspore-serving, pandas, ray, scikit-learn, scipy
这里可以看到，提交的任务中numpy是保存在mindspore-latest虚拟环境中的，而本地的numpy不在虚拟环境中，说明任务确实是在远程执行的。类似的可以在dashboard上面看到提交日志：
接下来测试一下分布式框架ray的并发特性：
import ray
ray.init()
@ray.remote(num_returns=1)
def cpu_task():
import time
time.sleep(2)
import numpy as np
nums = 100000
arr = np.random.random((2, nums))
arr2 = arr[1]**2 + arr[0]**2
pi = np.where(arr2<=1, 1, 0).sum() * 4 / nums
return pi
num_conc = 10
res = ray.get([cpu_task.remote() for _ in range(num_conc)])
print (sum(res) / num_conc)
这个案例的内容是用蒙特卡洛算法计算圆周率的值，一次提交10个任务，每个任务中撒点100000个，并休眠2s。那么如果是顺序执行的话，理论上需要休眠20s。而这里提交任务之后，输出如下：
$ time RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . --entrypoint-num-cpus 10 -- python te
st_ray.py
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-27 08:30:13,315 INFO dashboard_sdk.py:385 -- Package gcs://_ray_pkg_d66b052eb6944465.zip already exists, skipping upload.
-------------------------------------------------------
Job 'raysubmit_Ur6MAvP7DYiCT6Uz' submitted successfully
-------------------------------------------------------
Next steps
Query the logs of the job:
ray job logs raysubmit_Ur6MAvP7DYiCT6Uz
Query the status of the job:
ray job status raysubmit_Ur6MAvP7DYiCT6Uz
Request the job to be stopped:
ray job stop raysubmit_Ur6MAvP7DYiCT6Uz
Tailing logs until the job exits (disable with --no-wait):
2024-08-27 16:30:15,032 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-27 16:30:15,033 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-27 16:30:15,058 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265
3.141656
------------------------------------------
Job 'raysubmit_Ur6MAvP7DYiCT6Uz' succeeded
------------------------------------------
real 0m7.656s
user 0m0.414s
sys 0m0.010s
总的运行时间在7.656秒，其中5s左右的时间是来自网络delay。所以实际上并发之后的总运行时间就在2s左右，跟单任务休眠的时间差不多。也就是说，远程提交的任务确实是并发执行的。最终返回的结果进行加和处理，得到的圆周率估计为：3.141656。而且除了普通的CPU任务之外，还可以上传GPU任务：
import ray
ray.init()
@ray.remote(num_returns=1, num_gpus=1)
def test_ms():
import os
os.environ['GLOG_v']='4'
os.environ['CUDA_VISIBLE_DEVICE']='0'
import mindspore as ms
ms.set_context(device_target="GPU", device_id=0)
a = ms.Tensor([1, 2, 3], ms.float32)
return a.asnumpy().sum()
res = ray.get(test_ms.remote())
ray.shutdown()
print (res)
这个任务是用mindspore简单创建了一个Tensor，并计算了Tensor的总和返回给本地，输出内容为：
$ RAY_ADDRESS='http://xxx.xxx.xxx.xxx:8265' ray job submit --working-dir . --entrypoint-num-gpus 1 -- python test_ray.py
Job submission server address: http://xxx.xxx.xxx.xxx:8265
2024-08-28 01:16:38,712 INFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_10019cd9fa9bdc38.zip.
2024-08-28 01:16:38,712 INFO packaging.py:518 -- Creating a file package for local directory '.'.
-------------------------------------------------------
Job 'raysubmit_RUvkEqnkjNitKmnJ' submitted successfully
-------------------------------------------------------
Next steps
Query the logs of the job:
ray job logs raysubmit_RUvkEqnkjNitKmnJ
Query the status of the job:
ray job status raysubmit_RUvkEqnkjNitKmnJ
Request the job to be stopped:
ray job stop raysubmit_RUvkEqnkjNitKmnJ
Tailing logs until the job exits (disable with --no-wait):
2024-08-28 09:16:41,960 INFO worker.py:1330 -- Using address xxx.xxx.xxx.xxx:6379 set in the environment variable RAY_ADDRESS
2024-08-28 09:16:41,960 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: xxx.xxx.xxx.xxx:6379...
2024-08-28 09:16:41,974 INFO worker.py:1639 -- Connected to Ray cluster. View the dashboard at http://xxx.xxx.xxx.xxx:8265
6.0
------------------------------------------
Job 'raysubmit_RUvkEqnkjNitKmnJ' succeeded
------------------------------------------
返回的计算结果是6.0，那么也是正确的。
查看和管理任务
前面的任务输出信息中，都有相应的job_id，我们可以根据这个job_id在主节点上面查看相关任务的执行情况：
$ ray job status raysubmit_RUvkEqnkjNitKmnJ
可以查看该任务的输出内容：
$ ray job logs raysubmit_RUvkEqnkjNitKmnJ
还可以终止该任务的运行：
$ ray job stop raysubmit_RUvkEqnkjNitKmnJ
总结概要
本文介绍了基于Python的分布式框架Ray的基本安装与使用。Ray框架下不仅可以通过conda和Python十分方便的构建一个集群，还可以自动的对分布式任务进行并发处理，且支持GPU分布式任务的提交，极大的简化了手动分布式开发的工作量。
版权声明
本文首发链接为：https://www.cnblogs.com/dechinphy/p/ray.html
作者ID：DechinPhy
更多原著文章：https://www.cnblogs.com/dechinphy/
请博主喝咖啡：https://www.cnblogs.com/dechinphy/gallery/image/379634.html
标签：py,Ray,Python,xxx,--,job,ray,分布式 From： https://www.cnblogs.com/dechinphy/p/18384118/ray