经常出现observer崩溃的问题，检查系统日志发现内存不足导致。
(base) [root@lnpg log]# grep Killed messages*
messages-20241027:Oct 21 13:00:44 lnpg kernel: Out of memory: Killed process 15609 (observer) total-vm:24040644kB, anon-rss:19108860kB, file-rss:0kB, shmem-rss:0kB, UID:54322 pgtables:45800kB oom_score_adj:0
messages-20241027:Oct 23 11:47:26 lnpg kernel: Out of memory: Killed process 32587 (observer) total-vm:23755968kB, anon-rss:18518008kB, file-rss:0kB, shmem-rss:0kB, UID:54322 pgtables:44744kB oom_score_adj:0
messages-20241027:Oct 25 11:13:44 lnpg kernel: Out of memory: Killed process 28870 (observer) total-vm:23473084kB, anon-rss:14872228kB, file-rss:0kB, shmem-rss:0kB, UID:54322 pgtables:42360kB oom_score_adj:0
messages-20241103:Nov 1 14:20:10 lnpg kernel: Out of memory: Killed process 5804 (observer) total-vm:24022976kB, anon-rss:18868760kB, file-rss:0kB, shmem-rss:0kB, UID:54322 pgtables:45296kB oom_score_adj:0
messages-20241110:Nov 8 17:49:06 lnpg kernel: Out of memory: Killed process 28386 (observer) total-vm:23475132kB, anon-rss:16365368kB, file-rss:0kB, shmem-rss:0kB, UID:54322 pgtables:42384kB oom_score_adj:0
处理办法：
限制ob内存的使用，需要使用root修改，sys和普通用户无权限。
memory_limit 和memory_limit_percentage 修改的前提下，优先以memory_limit 为准
(base) [root@lnpg log]# free -g
total used free shared buff/cache available
Mem: 31 20 9 0 1 7
Swap: 0 0 0
[admin@lnpg oceanbase]$ obclient -h127.0.0.1 -uroot -P2881 -pxxx
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 3221487621
Server version: OceanBase 3.2.4.1 (r101000052023010822-346aa35c32e99d1b82d713f75f0072c45bdf7aab) (Built Jan 8 2023 22:52:43)
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
obclient [(none)]> ALTER SYSTEM SET memory_limit = '16G';
Query OK, 0 rows affected (0.027 sec)
obclient [(none)]> SHOW PARAMETERS LIKE 'memory_limit';
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 192.168.207.143 | 2882 | memory_limit | NULL | 16G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
1 row in set (0.003 sec)
obclient [(none)]> SHOW PARAMETERS LIKE 'memory_limit_percentage';
+-------+----------+-----------------+----------+-------------------------+-----------+-------+----------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
+-------+----------+-----------------+----------+-------------------------+-----------+-------+----------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 192.168.207.143 | 2882 | memory_limit_percentage | NULL | 80 | the size of the memory reserved for internal use(for testing purpose). Range: [10, 90] | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-----------------+----------+-------------------------+-----------+-------+----------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
1 row in set (0.002 sec)
obclient [(none)]> ALTER SYSTEM SET memory_limit = '15G';
Query OK, 0 rows affected (0.023 sec)
obclient [(none)]> SHOW PARAMETERS LIKE 'memory_limit';
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 192.168.207.143 | 2882 | memory_limit | NULL | 15G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
1 row in set (0.002 sec)
obclient [GISTAR]> ALTER SYSTEM SET memory_limit = '15G';
ORA-00600: internal error code, arguments: -5044, Unknown system variable 'memory_limit'
obclient [GISTAR]> SHOW PARAMETERS LIKE 'memory_limit';
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| ZONE | SVR_TYPE | SVR_IP | SVR_PORT | NAME | DATA_TYPE | VALUE | INFO | SECTION | SCOPE | SOURCE | EDIT_LEVEL |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 192.168.207.143 | 2882 | memory_limit | NULL | 15G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
1 row in set (0.006 sec)
[admin@lnpg log]$ obclient -h127.0.0.1 -uroot -P2881 -pxxx
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 3221488207
Server version: OceanBase 3.2.4.1 (r101000052023010822-346aa35c32e99d1b82d713f75f0072c45bdf7aab) (Built Jan 8 2023 22:52:43)
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
obclient [(none)]> ALTER SYSTEM SET memory_limit_percentage = '60';
Query OK, 0 rows affected (0.028 sec)
obclient [(none)]> SHOW PARAMETERS LIKE 'memory_limit_percentage';
+-------+----------+-----------------+----------+-------------------------+-----------+-------+----------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
+-------+----------+-----------------+----------+-------------------------+-----------+-------+----------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 192.168.207.143 | 2882 | memory_limit_percentage | NULL | 60 | the size of the memory reserved for internal use(for testing purpose). Range: [10, 90] | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-----------------+----------+-------------------------+-----------+-------+----------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
1 row in set (0.002 sec)
obclient [(none)]> SHOW PARAMETERS LIKE 'memory_limit';
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 192.168.207.143 | 2882 | memory_limit | NULL | 15G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-----------------+----------+--------------+-----------+-------+-------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
1 row in set (0.002 sec)
obclient [(none)]>
ob的参数修改：
https://www.oceanbase.com/knowledge-base/oceanbase-database-20000000031
适用版本
OceanBase 数据库所有版本。
配置 OBServer 内存使用量
OceanBase 数据库通过系统配置项 memory_limit 与 memory_limit_percentage 限制 OBServer 的内存使用。
系统配置项 memory_limit 的默认值为 0。如果该值不为 0，系统会忽略 memory_limit_percentage 的设定而优先使用 memory_limit 参数配置的值。例如，一台物理机的内存为 100 GB，memory_limit_percentage 参数设置为 80，表示使用 80% 的内存，而 memory_limit 参数设置为 90 GB，则实际上物理机上 OceanBase 数据库可分配的内存使用 memory_limit 的配置，即 90 GB。
配置memory_limit 时需要注意以下事项。
-
memory_limit的默认单位为 MB。例如，
memory_limit='40G'表示将 observer 进程使用的内存上限设置为 40 GB；memory_limit=40960表示将 OceanBase 进程使用的内存上限设置为 40960 MB，与memory_limit='40G'相同。 -
memory_limit的上限为 2^64^，这可能会导致设置的 OBServer 内存的最大使用上限大于物理内存的总和。 -
如果希望限制运行中的 observer 进程的内存大小，可以直接修改
memory_limit的值，使其达到预期。后台 reload 线程会使其动态生效，无需重启。但是在设定时，需要保证memory_limit的值大于 OBServer 总的内存使用量，小于机器的物理内存。可以通过以下语句查询当前used_gb的大小。memory_limit参数值必须大于used_gb。obclient> SELECT value/1024/1024/1024 used_gb FROM v$sysstat WHERE name LIKE '%observer memory%' AND con_id = 1 AND stat_id=140008 AND name='observer memory used size'; +-----------------+ | used_gb | +-----------------+ | 10.582031250000 | +-----------------+ 1 row in set (0.03 sec)
(base) [root@lnpg ~]# find / -name observer.config.bin
/home/admin/oceanbase/etc/observer.config.bin
(base) [root@lnpg ~]# strings /home/admin/oceanbase/etc/observer.config.bin|grep mem
__min_full_resource_pool_memory=268435456
memory_limit_percentage=60
system_memory=1G
memory_limit=15G