14.基础云性能优化

CSIG/云产品部/企业云平台产品中心

Exported on 11/09/2020

Table of Contents

1. mariadb 3
2. haproxy 4
3. keystone 5
4. nova 6
   1. 控制节点服务 6
   2. 计算节点服务 6
5. cinder 7
6. neutron 11
7. 注意事项 12

mariadb

1. 修改mariadb配置文件，在mariadb所有副本节点上修改配置文件/etc/my.conf

[mysqld] max_allowed_packet = 16M connect_timeout = 30

net_read_timeout = 90

net_write_timeout = 180

重启mariadb

systemctl restart mariadb

haproxy

1. 修改haproxy的默认超时时间，并增加cinder服务的tcp连接超时时间，在frontent与backend下面分别增加如下timeout配置，修改/etc/haproxy/haproxy.conf

defaults

mode tcp maxconn 4000

timeout connect 10s timeout client 5m timeout server 5m

frontend vip-cinder #...

timeout client 600s backend cinder-vms

#...

timeout server 600s

重启haproxy服务

systemctl restart haproxy

keystone

1. 修改keystone token超时时间为24小时，编辑配置文件/etc/keystone/keystone.conf

[token]

expiration = 86400

nova

控制节点服务

• • 1. 禁用nova quota功能，编辑配置文件/etc/nova/nova.conf

[quota]

driver = nova.quota.NoopQuotaDriver

• • 1. 配置数据库连接数，编辑配置文件/etc/nova/nova.conf

[api_database] max_pool_size = 100

max_overflow = 1000

max_retries = -1

[database] max_pool_size = 100

max_overflow = 1000

max_retries = -1

计算节点服务

• • 1. 虚拟机默认使用config drive，编辑配置文件/etc/nova/compute.conf

[DEFAULT]

force_config_drive = True

• • 1. 计算节点禁用自动disable，默认在连续创建10个虚拟机失败后，会自动禁用nova-compute服务，编辑配置文件/etc/nova/compute.conf

[compute] consecutive_build_service_disable_threshold = 0

• • 1. 禁用nova quota功能，编辑配置文件/etc/nova/compute.conf

[quota]

driver = nova.quota.NoopQuotaDriver

cinder

禁用cinder quota功能，确保具有

<

%E4%B8%AD%E7%9A%84%E4%BB%A3%E7%A0%81%E6%8F%90%E4%BA%A4%EF%BC%88%E5%9F%BA

%E7%A1%80%E4%BA%91stable/ tstack_19.7%E5%8F%8A%E4%B9%8B%E5%90%8E%E7%9A%84%E5%88%86%E6%94%AF%E9%83%BD%E 5%90%88%E5%85%A5%E4%BA%86%E6%AD%A4%E9%83%A8%E5%88%86%E4%BB%A3%E7%A0%81%E6

%8F%90%E4%BA%A4%EF%BC%89%E3%80%82%E7%84%B6%E5%90%8E%E4%BF%AE%E6%94%B9%E9

%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6/etc/cinder/cinder.conf">https://git.code.oa.com/ openstack/cinder_Pike/commit/2f197fecf5ca654e80ee6b1e4a6f71e5fb271ae5>中的代码提交（基础云 stable/tstack_19.7及之后的分支都合入了此部分代码提交）。然后修改配置文件/etc/cinder/

cinder.conf(see page 3)

[DEFAULT]

quota_driver = cinder.quota.NoopQuotaDriver

1. 设置cinder rpc同步调用超时时间，默认为60秒,并增加随机调度。修改配置文件/etc/cinder/ cinder.conf

[DEFAULT]

rpc_response_timeout = 600

scheduler_default_weighers = CapacityWeigher,ChanceWeigher

1. 设置cinder数据库连接数。修改配置文件/etc/cinder/cinder.conf

[database] max_pool_size = 100

max_overflow = 1000

max_retries = -1

1. 调整cinder-api服务进程数量。修改配置文件/etc/cinder/cinder.conf

[DEFAULT]

osapi_volume_workers = 5

解决cinder中大量出现的Lost connection问题，更新cinder代码，确保具有

<

%E4%B8%AD%E7%9A%84%E4%BB%A3%E7%A0%81%E6%8F%90%E4%BA%A4%28%E5%9F%BA%E7%A1

%80%E4%BA%91stable/ tstack_20.3%E5%8F%8A%E4%B9%8B%E5%90%8E%E7%9A%84%E5%88%86%E6%94%AF%E9%83%BD%E 5%90%88%E5%85%A5%E4%BA%86%E6%AD%A4%E9%83%A8%E5%88%86%E4%BB%A3%E7%A0%81%29

%E3%80%82">https://git.code.oa.com/openstack/cinder_Pike/commit/ efdb9ea7d67384996f2b683b32883321eb4734cc>中的代码提交(基础云stable/tstack_20.3及之后的分支都合入了此部分代码)。(see page 3)

provision上报

6.1 原理说明

在cinder-scheduler中通过查找数据库并计算相应ceph后端driver所创建的卷的所有size总和赋予 provisioned_capacity_gb,而cinder-volume就不需要去遍历访问ceph池，此字段主要是在创建卷时用来计算超分比

6.2 注意事项

（1）查看所有的ceph driver名字，通过cinder service-list查看，如下

+------------------+----------------------------------------+------+---------

+-------+----------------------------+ +

| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |

+------------------+----------------------------------------+------+---------

+-------+----------------------------+ +

| cinder-backup | openstack-B-con01

up | 2020-10-06T01:50:26.000000 | - |

| cinder-backup | openstack-B-con02

up | 2020-10-06T01:50:28.000000 | - |

| cinder-backup | openstack-B-con03

up | 2020-10-06T01:50:31.000000 | - |

| cinder-scheduler | openstack-B-con01

up | 2020-10-06T01:50:30.000000 | - |

| cinder-scheduler | openstack-B-con02

up | 2020-10-06T01:50:30.000000 | - |

| cinder-scheduler | openstack-B-con03

up | 2020-10-06T01:50:29.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd

down | 2020-10-06T01:48:25.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd2

up | 2020-10-06T01:50:29.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd3

up | 2020-10-06T01:50:29.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd4

up | 2020-10-06T01:50:30.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd5

up | 2020-10-06T01:50:30.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd6

up | 2020-10-06T01:50:30.000000 | - |

| cinder-volume | openstack-B-con01@ceph_hdd_multiattach up | 2020-10-06T01:50:29.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd

up | 2020-10-06T01:50:00.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd2

up | 2020-10-06T01:50:31.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd3

up | 2020-10-06T01:50:31.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd4

up | 2020-10-06T01:50:32.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd5

up | 2020-10-06T01:50:23.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd6

up | 2020-10-06T01:50:31.000000 | - |

| cinder-volume | openstack-B-con02@ceph_hdd_multiattach up | 2020-10-06T01:50:31.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd

down | 2020-10-06T01:49:14.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd2

up | 2020-10-06T01:50:24.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd3

up | 2020-10-06T01:50:24.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd4

up | 2020-10-06T01:50:25.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd5

up | 2020-10-06T01:50:25.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd6

up | 2020-10-06T01:50:24.000000 | - |

| cinder-volume | openstack-B-con03@ceph_hdd_multiattach up | 2020-10-06T01:50:24.000000 | - |

+------------------+----------------------------------------+------+---------

+-------+----------------------------+ +

我们增加了一个配置keyword_in_backend，这个参数主要是用来识别ceph后端的，也就是ceph后端包含的关键字，这是一个list，可以包含多个，默认配置是['ceph']

这里可以看到所有后端都包含ceph，可以在cinder配置中增加

[DEFAULT]

keyword_in_backend = ceph

因为默认值是ceph可以不配置，如果还有ceph后端包含rbd可以这样配置

[DEFAULT]

keyword_in_backend = ceph,rbd

修改cinder-scheduler代码（代码已经提交到20.3和tstack_master分支,这里只是列出优化点） cinder/scheduler/host_manager.py文件

+ 61

cfg.ListOpt('keyword_in_backend',

default=[

'ceph'

+ 506行左右

],

help='Keywords contained in the backend'),

backend = cluster_name or host backend_name = backend.split("@")[1] for keyword in CONF.keyword_in_backend:

if backend_name.find(keyword) != -1:

ctxt = cinder_context.get_admin_context()

volumes = objects.VolumeList.get_all(ctxt, filters={"host~": "@%s#" % backend_name})

provioned_capacity_gb = 0 for volume in volumes:

provisioned_capacity_gb += volume.size

capab_copy['provisioned_capacity_gb'] = provisioned_capacity_gb break

capab_old = self.service_states.get(backend, {"timestamp": 0})

+ 551

'cap': capab_copy,

修改cinder-volume代码 cinder/volume/drivers/rbd.py

+ 471

#total_gbi = self._get_usage_info() total_gbi = 0 stats['provisioned_capacity_gb'] = total_gbi

重启cinder-volume和cinder-scheduler

systemctl restart openstack-cinder-volume systemctl restart openstack-cinder-scheduler

并观察cinder-scheduler日志，查看有没有上报错误信息

1. 设置不从镜像下载，修改配置文件/etc/cinder/cinder.conf

• [DEFAULT]

allow_download_image_from_glance = false allow_copy_volume_over_backends = True

neutron

1. 设置neutron数据库连接数。修改配置文件/etc/neutron/neutron.conf

[DEFAULT]

ovs_vsctl_timeout = 30

[database] max_pool_size = 100

max_overflow = 100

max_retries = -1

1. 修改neutron ml2 /etc/neutron/plugins/ml2/ml2_conf.ini

[ovs]

vsdb_interface = vsctl

修改neutron l3配置

[DEFAULT]

ovs_use_veth=True ovs_vsctl_timeout = 30

注意事项

cinder禁止Ceph的一个池作为两个后端使用(多挂载的后端除外，因为比较少)

cinder中ceph后端的名字规范起来，最好包含ceph字样

0001-Add-NoopQuotaDriver.patch¹

1 https://iwiki.woa.com/download/attachments/377030353/0001-Add-NoopQuotaDriver.patch? api=v2&modificationDate=1602235427000&version=1