金蝶云苍穹的简单学习

金蝶云苍穹的简单学习

简介

云苍穹 是一套管理控制台
云星瀚 是一套带各种app的ERP应用。

服务管理方式

虽然使用的是 ubuntu2204的系统. 
但是都是使用的systemd的管理
主要服务信息如下:

cloud-init-hotplugd.service
e2scrub_all.timer
e2scrub_all.service
cloud-init.service
cloud-init-local.service
cloud-final.service
cloud-config.service
redis-6379.service
zookeeper.service
nginx.service
postgresql.service
fileserver.service
mc.service
mservice.service
mservice-qing.service

简单分析

第一: 时间同步
systemd-timesyncd.service
简单查看一下状态:
root@cosmic:/etc/systemd# systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-12-05 05:45:55 UTC; 1 day 1h ago
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 800 (systemd-timesyn)
     Status: "Initial synchronization to time server 91.189.94.4:123 (ntp.ubuntu.com)."
      Tasks: 2 (limit: 38374)
     Memory: 1.4M
        CPU: 207ms
     CGroup: /system.slice/systemd-timesyncd.service
             └─800 /lib/systemd/systemd-timesyncd

Dec 05 05:45:55 cosmic systemd[1]: Starting Network Time Synchronization...
Dec 05 05:45:55 cosmic systemd[1]: Started Network Time Synchronization.
Dec 05 05:45:55 cosmic systemd-timesyncd[800]: Network configuration changed, trying to establish connection.
Dec 05 05:46:07 cosmic systemd-timesyncd[800]: Network configuration changed, trying to establish connection.
Dec 05 05:46:08 cosmic systemd-timesyncd[800]: Initial synchronization to time server 91.189.94.4:123 (ntp.ubuntu.com).

时间同步

其实可以通过
systemctl stop systemd-timesyncd.service
systemctl disable systemd-timesyncd.service

配置文件其实在
/etc/systemd/timesyncd.conf

苍穹的启动脚本

启动路径为.
systemd 调用 service配置文件
调用 kdservice.sh
然后调用 start.sh
然后调用 setenv.sh
最后调用 setappenv.sh
所以需要修改setappenv.sh 里面的 url 最终通过systemd的服务进行服务启动.

启动脚本

cat /lib/systemd/system/mc.service
[Unit]
Description=mc
After=syslog.target network.target 

[Service]
Type=forking
Environment="JAVA_HOME=/kingdee/jdk/jdk1.8.0_161"
ExecStart=/kingdee/cosmic/mc/mservice/bin/kdservice.sh start
ExecStop=/kingdee/cosmic/mc/mservice/bin/kdservice.sh stop

[Install]
WantedBy=multi-user.target

查看具体的服务启动脚本

#!/bin/bash
#rd_fuxiong_pan
:<<!
使用本地配置文件进行启动服务
!
#脚本只要发生错误，就终止执行
set -e
#在运行结果之前，先输出执行的那一行命令
set -x

#获取脚本所有的绝对路径
PRG="$0"
while [ -h "$PRG" ]; do
  ls=`ls -ld "$PRG"`
  link=`expr "$ls" : '.*-> \(.*\)$'`
  if expr "$link" : '/.*' > /dev/null; then
    PRG="$link"
  else
    PRG=`dirname "$PRG"`/"$link"
  fi
done

# Get standard environment variables
PRGDIR=`dirname "$PRG"`
#进入脚本所在的目录
cd $PRGDIR

#设置使用本地的应用配置文件
export ENABLE_APPENV=true
#export ENABLE_JVMGC=true
#export ENABLE_JMXOPS=true 
#export ENABLE_JETTY=true


start()
{
      source ./start.sh start -d
}

stop()
{
      source ./start.sh stop
}

status() {
       source ./start.sh status
}

debug() {
       export ENABLE_JVMDEBUG=true
       source ./start.sh start -d
}


restart() {
       source ./start.sh restart -d
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  debug)
    debug
    ;;  
  restart|reload|force-reload)
    restart
    ;;
  status)
    status
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|debug}"
    RETVAL=1
esac

exit $RETVAL

最后的脚本

• 其实简单看了下 这样的脚本比较优雅了
• 就是jdk的设置没有动态实现.
• 可以不同的脚本参数实现启动生产,测试还有debug模式.

#!/bin/bash
#rd_fuxiong_pan
:<<!
resolve links - $0 may be a softlink
# mservice执行脚本,启动参数为start|status|stop|restart
修改记录：
2018-03-02:
删除：
    1.去除debug参数，修改为环境变量控制
    2.去除第二个参数传递配置名，从../assembly/product/env/$2/复制配置文件到bin目录，使用独立脚本
    
变更：
    1.按照最新的目录结构加载classpath
    2.拆解不同的配置初始化脚本
    3.增加第二个参数
!
#脚本只要发生错误，就终止执行
set -e
#在运行结果之前，先输出执行的那一行命令
#set -x

#延时检查进程是否存在
if [ ! $KD_STARTUP_SLEEP_TIME ]; then
      KD_STARTUP_SLEEP_TIME=3
fi

#如果存在目录的环境变量，则不再进入脚本所在目录
#如果不存在，则进入脚本所在的目录
if [ ! $PRGDIR ]; then
    #获取脚本所有的绝对路径
    PRG="$0"
    while [ -h "$PRG" ]; do
      ls=`ls -ld "$PRG"`
      link=`expr "$ls" : '.*-> \(.*\)$'`
      if expr "$link" : '/.*' > /dev/null; then
        PRG="$link"
      else
        PRG=`dirname "$PRG"`/"$link"
      fi
    done

    # Get standard environment variables
    PRGDIR=`dirname "$PRG"`
    export PRGDIR=$PRGDIR
    #进入脚本所在的目录
    cd $PRGDIR
fi

#检查java运行环境
if [ -x "$JAVA_HOME/bin/java" ]; then
    JAVA="$JAVA_HOME/bin/java"
else
    JAVA=`which java`
fi

#检查java命令是否存在
if [ ! -x "$JAVA" ]; then
    echo "无法找到java运行文件，请安装jdk，或者设置PATH环境变量或者JAVA_HOME环境变量"
    echo "Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME"
    exit 1
fi

#加载应用环境变量
source setenv.sh

#判断是否存在参数-d或者--daemonize，有则后台运行
if [[ $* =~ "-d" || $* =~ "--daemonize" ]] ; then
    daemonized="true"
fi

start()
{
    #拉jar包
    #java -jar -DAPPSTORE_URL=http://172.18.2.144:88/kingdee/dev2.0/mservice/ -DdeployedAppIds=fi-er,fi-ai -DCUSLIBS=  kd-appstore-download-1.0.jar
    $JAVA -jar kd-appstore-download-1.0.jar
    #获取进程号,避免在start时,出现pid值仍然存在的情况
    pidlist=`ps -ef|grep "appName=$appName " |grep -v "grep"|awk '{print $2}'`
    #如果有进程存在，则直接跳出
    if [ "$pidlist" ]; then
        echo "ERROR:: $appName is still running,please confirm and use stop|restart!"
        exit 1
    fi
  
    #初始化环境变量
    #source执行子级命令后继续执行父级命令
    #同时子级设置的环境变量会影响到父级的环境变量
    source setclasspath.sh
    
    #判断是否存在classpath
    if [ ! $KD_CLASSPATH ]; then
      echo "ERROR:: 无法初始化环境变量CLASSPATH,异常退出"
      echo "ERROR:: can not setCLASSPATH,exit error!!!"
      exit 1
    fi

    echo -n "Starting $appName: "
    #如果参数长度为零，则运行在前端
    if [ -z "$daemonized" ]; then
        echo "start app in front end !!!"
        exec "$JAVA" $JVM_OPTS -DappName=$appName -DMONITOR_HTTP_PORT=9997 $JVM_DEBUG_OPTS $JVM_GC_OPTS $JVM_JMX_OPTS $JAVA_APP_OPTS $WEB_OPTS \
               -classpath "$KD_CLASSPATH" $BOOTCLASS
    else
        #后台运行
        nohup "$JAVA" $JVM_OPTS -D"appName=$appName "  -DMONITOR_HTTP_PORT=9997  $JVM_DEBUG_OPTS $JVM_GC_OPTS $JVM_JMX_OPTS $JAVA_APP_OPTS $WEB_OPTS \
               -classpath "$KD_CLASSPATH" $BOOTCLASS > ../logs/std.out  2>&1 &
        
        if [ ! -z "$KD_STARTUP_SLEEP_TIME" ]; then
          sleep $KD_STARTUP_SLEEP_TIME
        fi
        pidlist=`ps -ef|grep "appName=$appName " |grep -v "grep"|awk '{print $2}'`
        if [ "$pidlist" = "" ] ; then
           echo "ERROR:: Could not execute $appName java process，please check the error log"
           exit 1
        else
           echo "success to run $appName app，please check status!"
           exit 0
        fi
    fi
}


stop()
{
  echo -n $"Stopping $appName: "
  pidlist=`ps -ef|grep "appName=$appName " |grep -v "grep"|awk '{print $2}'`
  if [ "$pidlist" = "" ];then
      echo "no $appName pid alive"
  else
      for pid in ${pidlist}
      {
         kill -9 $pid
         echo "KILL $pid:"
      }
      echo "$appName stop success"
   fi
}

status()
{
       pidlist=`ps -ef|grep "appName=$appName " |grep -v "grep"|awk '{print $2}'`
       if [ "$pidlist" = "" ] ; then
          echo "$appName is stoped! "
       else
          echo "$appName is running! "
       fi
}

#此处有个问题,之前运行为正常start模式,重启后,也会变成debug模式
#已修改,不要使用pid,换为dpid,应该是因为pid变量在上面有使用:(,那不是局部变量么?
restart() {
      pidlist=`ps -ef|grep $BOOTCLASS |grep "appName=$appName " |grep -v "grep"|awk '{print $2}'`
      if [ "$pidlist" = "" ];then
          echo "no $appName pid alive"
          start
      else
          for pid in ${pidlist}
          {
             kill -9 $pid
             echo "KILL $pid:"
          }
          echo "$appName stop success"
          start
       fi
}


case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart|reload|force-reload)
    restart
    ;;
  status)
    status
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|debug}"
    RETVAL=1
esac

exit $RETVAL

设置环境变量的脚本

• 其实这里面我没太搞明白. 调用逻辑.
• 但是理论上一个启动脚本应该调用到了.
• 应该是:
• systemd->

#!/bin/bash
#rd_fuxiong_pan
#设置初始参数,最好全部用引号包含,避免出现特殊字符串导致的断片
#脚本只要发生错误，就终止执行
set -e

export configUrl="127.0.0.1:2181"
export clusterName="mc"

#在运行结果之前，先输出执行的那一行命令
export BOOTCLASS="kd.bos.service.webserver.JettyServer"
export appName="mc"
export appSplit=false
export appIds=
export WEB_PORT=8082
export dubboProtocolPort="21880"
export ENABLE_JETTY=true
export JETTY_WEBAPP_PATH="../webapp"
export JETTY_CONTEXT=ierp
export JPDA_ADDRESS=9800
export APPSTORE_URL=http://127.0.0.1:8090/appstore/mc
export JAVA_APP_OPTS=" -DclusterName=$clusterName -DappName=$appName -Dalgox.local.enable=false -DconfigUrl=$configUrl -Ddubbo.protocol.port=$dubboProtocolPort -Ddev.project.sync.enable=true"
export is_unzip_dm_before_upgrade=false

export webmserviceinone=true
export configAppName=mservice,web
export isMC=true
export domain_contextUrl=http://10.110.136.90:8090/mc
export ENABLE_JVMDEBUG=true
export JVM_OPTS="-server -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:NativeMemoryTracking=summary  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../logs/dump.%t.hprof -XX:+PrintGCApplicationStoppedTime  -XX:+UseG1GC -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:../logs/gc.%t.log -Duser.timezone=GMT+8 -Dfile.encoding=UTF-8 -Dxdb.enable=false -DdataCenterName=newmc"
export lightweightdeploy=false

云星瀚的启动脚本

其实发现云苍穹和云星瀚是一模一样的启动脚本
只不过分别选择了不同的环境变量
不同的配置文件.进行启动
脚本都是一套的. 
运维成本比较好控制一下. 

其他的不列举了

总体启动脚本

• 脚本位置 /lib/systemd/system/mservice.service

[Unit]
Description=mservice
After=syslog.target network.target 

[Service]
Type=forking
Environment="JAVA_HOME=/kingdee/jdk/jdk1.8.0_161"
ExecStart=/kingdee/cosmic/mservice/mservice/bin/kdservice.sh start
ExecStop=/kingdee/cosmic/mservice/mservice/bin/kdservice.sh stop

[Install]
WantedBy=multi-user.target

redis的启动脚本

• /lib/systemd/system/redis-6379.service

[Unit]
Description=redis-6379
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStart=/usr/local/bin/redis-server /kingdee/common/redis/conf/redis-6379.conf
ExecReload=/bin/kill -s HUP $MAINPID 
ExecStop=/usr/local/bin/redis-cli -p 6379 shutdown
PrivateTmp=true

[Install]
WantedBy=multi-user.target

postgresql的启动脚本

• /lib/systemd/system/postgresql.service
• 自己写了一个焦恩,感觉比较简单一些. 但是比较有效
• 可以魔方圆Ixia.

[Unit]
Description=PostgreSQL 12 database server
After=syslog.target network.target

[Service]
Type=forking
TimeoutSec=120
User=postgres
Environment=PGDATA=/home/postgres/pg_data
ExecStart=/var/postgresql/soft/pgsql/bin/pg_ctl start -w -D "/home/postgres/pg_data"
ExecStop=/var/postgresql/soft/pgsql/bin/pg_ctl stop -m fast -w -D "/home/postgres/pg_data"
ExecReload=/var/postgresql/soft/pgsql/bin/bin/pg_ctl reload -D "/home/postgres/pg_data"

[Install]
WantedBy=multi-user.target

zookeeper

• /lib/systemd/system/zookeeper.service
• 其实讨论都是一样,而且复用了 jdk挺好的.

[Unit]
Description=zookeeper
After=syslog.target network.target

[Service]
Type=forking
Environment=ZOO_LOG_DIR=/kingdee/common/zookeeper/data/zk_2181/logs
Environment=ZOO_LOG4J_PROP=WARN,ROLLINGFILE
Environment="PATH=/kingdee/jdk/jdk1.8.0_161/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin"
ExecStart=/kingdee/common/zookeeper/zookeeper-3.5.9/bin/zkServer.sh start
ExecStop=/kingdee/common/zookeeper/zookeeper-3.5.9/bin/zkServer.sh stop

[Install]
WantedBy=multi-user.target

zookeeper之二

/kingdee/common/zookeeper/zookeeper-3.5.9/bin/zkServer.sh   
好像是zookeeper 官方的 不放了..

Nginx

[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStartPre=/usr/local/nginx/sbin/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

nginx 使用的是 conf.d的模式

• 其实配置文件挺值得学习的

user  root;
worker_processes  4;
events {
    use epoll;
    multi_accept on;
    worker_connections  61024;
}
stream {
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    client_max_body_size   100m;
        log_format  main '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" "$upstream_response_time"';
    sendfile        on;
    #tcp_nopush     on;
    gzip    on;
    gzip_buffers   4 8k;
    gzip_comp_level 2;
    gzip_min_length 1000;
    gzip_types text/plain text/json text/css application/x-httpd-php application/json application/x-javascript application/javascript text/xml application/xml application/xml+rss text/javascript image/png image/jpg image/jpeg image/gif image/bmp;
    server_tokens off;
    keepalive_timeout  65;
        
    include conf.d/*.conf;
}

一个conf.d下面的样例

• /usr/local/nginx/conf/conf.d/mc.conf

server {
    listen 8090;   # mc虚拟主机端口号
    server_name localhost;  
    error_log /kingdee/nginx-logs/mc_error.log;
    access_log off;
    access_log /kingdee/nginx-logs/mc_access.log;

    underscores_in_headers on;
    client_max_body_size   2048m;

# 后端请求配置
    location ~(\.(do|jsp)$)|(/ierp/(kapi|kdctlres|attachment|excelpreview|kws)/)|(/ierp/?$)|(/ierp/(index\.html|mobile\.html|login\.html|login-mobile\.html)$)|(monitor/) {
        proxy_pass http://next-mc;              
        proxy_set_header Cookie $http_cookie;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;
        proxy_set_header tenantAlias mc;   
        client_body_buffer_size 256k;
        proxy_connect_timeout 1800;
        proxy_send_timeout 1800;
        proxy_read_timeout 1800;
        proxy_buffering on;
        proxy_buffer_size 256k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
        proxy_temp_file_write_size 256k;
        proxy_next_upstream error timeout invalid_header http_500 http_503 http_404;
        proxy_max_temp_file_size 128m;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        send_timeout 1800;
    }
    # 静态资源前端拦截
    location /ierp/ {
        alias /var/appstatic/static-files/mc/webapp/;    #静态资源存放地址
    }

    location ^~ /mc/ {
        proxy_pass http://127.0.0.1:8090/ierp/;
        proxy_cookie_path /mc/ /ierp/;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header https 1;
        proxy_connect_timeout 1800;
        proxy_send_timeout 1800;
        proxy_read_timeout 1800;
    }
    location /mq/ {
        proxy_pass http://mqserver/;
        proxy_connect_timeout    500s;
        proxy_read_timeout       500s;
        proxy_send_timeout       500s;
        proxy_set_header  Host  $host:$server_port;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    }
    location ^~/bos-olap-webserver/ {
        proxy_pass http://next-mdd/bos-olap-webserver/;
        proxy_connect_timeout    500s;
        proxy_read_timeout       500s;
        proxy_send_timeout       500s;
        proxy_set_header  Host  $host:$server_port;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    }
    
    location /appstore {
        root   /var/appstatic;
        client_max_body_size 4G;
        access_log off;
    }
    location /patchwarehouse {
        root   /var/appstatic;
        client_max_body_size 4G;
        access_log off;
    }

    location ^~/mc/msgwatch/ {
        proxy_pass http://next-mc/msgwatch/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;
    }

    location / {
        proxy_pass http://k8e;
        proxy_connect_timeout    500s;
        proxy_read_timeout       500s;
        proxy_send_timeout       500s;
        proxy_set_header  Host  $host:$server_port;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

统一的服务器资源

• /usr/local/nginx/conf/conf.d/upstream.conf

upstream next-ierp {
    server 127.0.0.1:8080;
    check interval=3000 rise=2 fall=5 timeout=1000 type=tcp port=8080;   
}
upstream next-mc {
    server 127.0.0.1:8082;
    check interval=3000 rise=2 fall=5 timeout=1000 type=tcp port=8082;
}
upstream next-fileserver {
    server 127.0.0.1:8100;
    check interval=3000 rise=2 fall=5 timeout=1000 type=tcp port=8100;
}
upstream mqserver {   
    server 127.0.0.1:15672;
}
upstream next-mdd {
    server 127.0.0.1:8888;
    #check interval=3000 rise=2 fall=5 timeout=1000 type=tcp port=8888;
}
upstream k8e {
    server 127.0.0.1:8081;
    #check interval=3000 rise=2 fall=5 timeout=1000 type=tcp port=8081;
}

fileserver

• /lib/systemd/system/fileserver.service
• 好像模型里面 是用jetty 自己写了一个很简单的filerserver.

[Unit]
Description=fileserver
After=syslog.target network.target 

[Service]
Type=forking
Environment="JAVA_HOME=/kingdee/jdk/jdk1.8.0_161"
ExecStart=/kingdee/cosmic/fileserver/node-debug-fileserver/bin/start.sh start
ExecStop=/kingdee/cosmic/fileserver/node-debug-fileserver/bin/start.sh stop

[Install]
WantedBy=multi-user.target