k8s Ingress和 Ingress Controller

一、Ingress和 Ingress Controller概述

关于service四层代理介绍链接点我跳转

1.1 Ingress

ingress是k8s中的资源，主要是管理ingress-controller这个代理的配置文件。

Ingress 是对集群中服务的外部访问进行管理的 API 对象，典型的访问方式是 HTTP，Ingress 可以提供负载均衡、SSL 终结和基于名称的虚拟托管。

 

使用ingress写规则也要依赖于四层代理service，由service代理才能找到后端代理的pod

kubectl explain ingress  可通过该命令查看ingress资源的字段

1.2 Ingress Controller

Ingress Controller是一个七层的负载均衡调度器，正常客户端访问的都是pod，访问pod的时候如果指定了Ingress Controller，那么请求会先到达Ingress Controller，然后Ingress Controller会根据Ingress写的配置代理到被访问pod前端的service里，然后由servic里面定义关联的标签选择器，最终将请求代理到所关联的pod。

 

Ingress Controller里面封装的是nginx，也可称为ingress-nginx-controller七层代理

1.3 Ingress和 Ingress Controller 总结

Ingress是一个代理，是真正用于处理请求的，如一个以pod形式运行的Ingress Controller，该pod里面运行的是nginx容器，

Ingress就是负责将Ingress Controller 的配置进行实时的更新，Ingress创建后，它里面定义的资源会同步到Ingress Controller的这个pod中

1.4 使用Ingress Controller代理pod流程

1、创建部署ingress controller，它使用的是nginx

2、创建pod，也可以通过控制器创建pod

3、创建servic，用来分组pod

4、创建ingress 写规则，可以写http或者https的规则，然后测试通过http/https访问应用

 

客户端访问的时候，请求的就是ingress controller，它通过ingress写的规则转发请求到servic，然后通过servic最终将客户端的请求转发到service所关联的pod。

二、Ingress Controller 高可用

创建两个Ingress Controller 高可用的pod

 

Ingress Controller是集群流量的接入层，对它做高可用非常重要，可以基于keepalive实现

Ingress Controller 官网的yanl文件参考：https://github.com/kubernetes/ingress-nginx

https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/baremetal

2.1 创建Ingress-controller

在ingress-nginx名称空间下创建两个名称Ingress-controller的pod，根据Deployment+ nodeSeletor+pod反亲和性方式部署在k8s指定的两个work节点，这个pod共享宿主机的IP，通过keepalive+nginx实现高可用

 

需要使用到如下两个镜像，可通过docker search 搜索下载后打包存到containerd或者使用containerd直接进行下载。

镜像：

kube-webhook-certgen

nginx-ingress-controller

#编写Ingress-controller的yaml文件
[root@k8s-master ~]# vi ingress-deploy.yaml   //加入如下内容

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
      - namespaces
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ipFamilyPolicy: SingleStack
  ipFamilies:
    - IPv4
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      hostNetwork: true
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: ingress-nginx
              topologyKey: kubernetes.io/hostname
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --controller-class=k8s.io/ingress-nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: nginx
  namespace: ingress-nginx
spec:
  controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-4.0.10
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 1.1.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-4.0.10
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 1.1.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000

#创建
[root@k8s-master ~]# kubectl apply -f ingress-deploy.yaml 
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

#查看ingress-nginx名称空间下创建的pod  （上面两个admission是用来生成证书的，主要关注下面两个pod（已创建ok且在不同的节点））
[root@k8s-master ~]# kubectl get pod -n ingress-nginx -owide
NAME                                        READY   STATUS      RESTARTS   AGE   IP               NODE          NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-8rgqz        0/1     Completed   0          24s   10.244.194.70    k8s-worker1   <none>           <none>
ingress-nginx-admission-patch-lj2dh         0/1     Completed   1          24s   10.244.194.71    k8s-worker1   <none>           <none>
ingress-nginx-controller-64bdc78c96-brd5v   1/1     Running     0          24s   192.168.57.132   k8s-worker1   <none>           <none>
ingress-nginx-controller-64bdc78c96-wkfln   1/1     Running     0          24s   192.168.57.133   k8s-worker2   <none>           <none>

2.2 为pod做代理实现高可用

上面创建好的pod还没有做相关配置，如下内容将基于keepalived和nginx为两个pod做负载均衡及反向代理。

#192.168.57.132  是主

#192.168.57.133  是备

#192.168.57.134  是虚拟ip

2.2.1 安装keepalived和nginx及修改nginx配置文件

#在两个工作节点上安装nginx及keepalived
[root@k8s-worker1 ~]# yum install  epel-release  nginx keepalived nginx-mod-stream  -y

#修改两个工作节点的nginx配置文件
#修改内容如下  (最好使用vi打开文件复制，vim可能会出现缩进问题)
[root@k8s-worker1 ~]# vi /etc/nginx/nginx.conf   //注意，文件中ip地址要修改为自己工作节点的IP

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-ingress-controller {
       server 192.168.57.132:80 weight=5 max_fails=3 fail_timeout=30s;   # xianchaonode1 IP:PORT
       server 192.168.57.133:80 weight=5 max_fails=3 fail_timeout=30s;   # xianchaonode2 IP:PORT
    }
    
    server {
       listen 30080; 
       proxy_pass k8s-ingress-controller;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

}

2.2.2 编写nginx状态检查脚本

#在两个节点上使用nginx 检查脚本
#脚本内容及路径如下
#[root@k8s-worker1 ~]# vim /etc/keepalived/check_nginx.sh
#[root@k8s-worker1 ~]# chmod +x /etc/keepalived/check_nginx.sh  //给脚本赋予执行权限

#!/bin/bash
#1、判断Nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
if [ $counter -eq 0 ]; then
    #2、如果不存活则尝试启动Nginx
    service nginx start
    sleep 2
    #3、等待2秒后再次获取一次Nginx状态
    counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
    #4、再次进行判断，如Nginx还不存活则停止Keepalived，让地址进行漂移
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi

2.2.3 修改主节点keepalived配置文件

#在两个工作节点中修改keepalived配置文件
[root@k8s-worker1 ~]# vim /etc/keepalived/keepalived.conf    //主节点修改配置文件内容如下
#(注意虚拟地址要与自己主备节点的ip地址网段一样且不被占用)

global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens33      # 修改为实际网卡名
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 100         # 优先级，备服务器设置 90 
    advert_int 1         # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.57.134/24
    } 
    track_script {
        check_nginx
    } 
}

#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
#virtual_ipaddress：虚拟IP（VIP）

2.2.4 修改备节点keepalived配置文件

#备节点修改配置如下，只需要修改如下两个点
[root@k8s-worker2 ~]# cat /etc/keepalived/keepalived.conf | grep -Ei "state|90"
    state BACKUP         # 将MASTER 修改为 BACKUP
    priority 90          # 优先级，备服务器设置 90 主服务器为100

2.2.5 启动服务

启动顺序：

1、启动主节点

2、启动备节点

#主节点启动服务
[root@k8s-worker1 ~]# systemctl daemon-reload
[root@k8s-worker1 ~]# systemctl enable nginx keepalived
[root@k8s-worker1 ~]# systemctl start nginx
[root@k8s-worker1 ~]# systemctl start keepalived

#启动后查看主节点IP  虚拟IP已经绑定
[root@k8s-worker1 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.57.132/24 brd 192.168.57.255 scope global noprefixroute ens33
    inet 192.168.57.134/24 scope global secondary ens33

#备节点启动服务
[root@k8s-worker2 ~]# systemctl daemon-reload
[root@k8s-worker2 ~]# systemctl enable nginx keepalived
[root@k8s-worker2 ~]# systemctl start nginx
[root@k8s-worker2 ~]# systemctl start keepalived

#测试虚拟ip漂移
#停掉主节点的keepalived
[root@k8s-worker1 ~]# systemctl stop keepalived

#主节点停掉keepalived服务之后，虚拟ip自动漂移到备节点上
[root@k8s-worker2 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.57.133/24 brd 192.168.57.255 scope global noprefixroute ens33
    inet 192.168.57.134/24 scope global secondary ens33


#由于主节点的优先级设置的高，主机点启动keepalived之后，虚拟IP还会自动漂移回来
[root@k8s-worker1 ~]# systemctl start keepalived

[root@k8s-worker1 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.57.132/24 brd 192.168.57.255 scope global noprefixroute ens33
    inet 192.168.57.134/24 scope global secondary ens33

三、测试ingress基于HTTP代理k8s内部pod

#k8s master节点编写创建一个service四层代理pod的yaml
[root@k8s-master ~]# vim ingress-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: canary
  ports:
  - name: http
    targetPort: 8080
    port: 8080
  - name: aaa
    targetPort: 8009
    port: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dep-tomcat
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tomcat
      release: canary
  template:
    metadata:
      labels:
        app: tomcat
        release: canary
    spec:
      containers:
      - name: mytomcat
        image: tomcat:8.5-jre8-alpine
        imagePullPolicy: IfNotPresent  
        ports:
        - name: http
          containerPort: 8080
          name: aaa
          containerPort: 8009


[root@k8s-master ~]# kubectl apply -f ingress-svc.yaml 
service/tomcat created
deployment.apps/dep-tomcat created

[root@k8s-master ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP             2d3h
tomcat       ClusterIP   10.103.110.10   <none>        8080/TCP,8009/TCP   8s

[root@k8s-master ~]# kubectl get pods -owide 
NAME                          READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
dep-tomcat-5b6b7b648c-5994q   1/1     Running   0          25s   10.244.126.3    k8s-worker2   <none>           <none>
dep-tomcat-5b6b7b648c-vm4zn   1/1     Running   0          25s   10.244.194.72   k8s-worker1   <none>           <none>

#创建ingress 资源的yaml
[root@k8s-master ~]# vim ingress-01.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-name
spec:
  ingressClassName: nginx #添加注解，指定创建的这个ingress规则往哪个ingress controller里面传，这里指定nginx表示往创建在ingress-nginx名称空间中的那两个ingress-nginx-contriller里面
  rules:
  - host: tomcat.abababc.com            #定义域名
    http:
      paths:
      - backend:
          service:
            name: tomcat                #这里定义创建的service的名称，表示要关联它
            port:
              number: 8080              #这里定义service的端口，kubectl get svc 查看（有两个端口 前面的外部使用，后面的内部使用）
        path: /
        pathType: Prefix


[root@k8s-master ~]# kubectl apply -f ingress-01.yaml 
ingress.networking.k8s.io/ingress-name created

[root@k8s-master ~]# kubectl get ingress
NAME           CLASS   HOSTS            ADDRESS                         PORTS   AGE
ingress-name   nginx   tomcat-001.com   192.168.57.132,192.168.57.133   80      60s

#验证ingress controller的pod是否注入了创建的ingress的配置
#进入ingress controller其中的一个pod
[root@k8s-master ~]# kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-8rgqz        0/1     Completed   0          24h
ingress-nginx-admission-patch-lj2dh         0/1     Completed   1          24h
ingress-nginx-controller-64bdc78c96-brd5v   1/1     Running     0          24h
ingress-nginx-controller-64bdc78c96-wkfln   1/1     Running     0          24h

[root@k8s-master ~]# kubectl exec -it ingress-nginx-controller-64bdc78c96-brd5v -n ingress-nginx -- /bin/bash
bash-5.1$ cat nginx.conf | grep "server_name tomcat.abababc.com"
		server_name tomcat.abababc.com ;             //查看nginx配置文件，域名已注入
    
bash-5.1$ cat nginx.conf | grep -C 2 'set $service_name'   //验证如下，相关ingress配置已注入
			set $namespace      "default";
			set $ingress_name   "ingress-name";
			set $service_name   "tomcat";
			set $service_port   "8080";
			set $location_path  "/";
    
#测试外部浏览器访问，首先添加域名至电脑的hosts文件中
#hosts路径：C:\Windows\System32\drivers\etc  （这里添加的是虚拟IP 及域名）如下图一所示
#然后进行浏览器访问域名  如下图二所示

ping 域名也能得到虚拟IP的地址。

PS：访问域名加端口，它会将请求代理给192.168.57.134:30080这个虚拟IP，而这个虚拟IP会以keepalived中配置的网卡信息找到在nginx配置文件中配置的192.168.57.132:30080及192.168.57.133:30080，这两个IP就是ingresscontroller的pod，然后它会根据ingress的规则将请求交给service，最终根据service找到tomcat这个pod。

四、测试ingress基于HTTPS代理k8s内部pod

基于HTTPS代理的话需要使用到证书，所以要先生成证书

4.1 生成证书并创建secret加密

#生成私钥   （在master节点执行）
[root@k8s-master ~]# cd /root/
[root@k8s-master ~]# openssl genrsa -out aaa.key 2048
Generating RSA private key, 2048 bit long modulus
.......+++
.................................................................................................+++
e is 65537 (0x10001)

#基于私钥生成根证书，指定证书域名为aaabbbccc.com
[root@k8s-master ~]# openssl req -new -x509 -key aaa.key -out aaa.crt -subj /C=CN/ST=ShangHai/L=ShangHai/O=DevOps/CN=tomcat.aaabbbcc.com

#创建secret 对配置进行加密
[root@k8s-master ~]# kubectl create secret tls tomcat-ingress-secret --cert=aaa.crt --key=aaa.key
secret/tomcat-ingress-secret created

[root@k8s-master ~]# kubectl get secret
NAME                    TYPE                DATA   AGE
tomcat-ingress-secret   kubernetes.io/tls   2      6s

[root@k8s-master ~]# kubectl describe secret tomcat-ingress-secret
Name:         tomcat-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1285 bytes
tls.key:  1679 bytes

4.2 创建ingress资源

[root@k8s-master ~]# vim ingress-tomcat.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-tomcat
  namespace: default
spec:
  ingressClassName: nginx               #指定此Ingress应由Nginx Ingress控制器处理
  tls:
  - hosts:                              #指定创建证书时被签发的域名
    -  tomcat.aaabbbccc.com
    secretName: tomcat-ingress-secret   #指定创建的secret的名称
  rules:
  - host: tomcat.aaabbbccc.com          #指定Ingress监听的主机名。
    http:
      paths:
      - path: /
        pathType:  Prefix
        backend:
         service:
           name: tomcat
           port:
            number: 8080

#访问tomcat.aaabbbccc.com这个域名会将请求代理到tomcat:8080


[root@k8s-master ~]# kubectl apply -f ingress-tomcat.yaml 
ingress.networking.k8s.io/ingress-tomcat created

[root@k8s-master ~]# kubectl get ingress
NAME             CLASS   HOSTS                  ADDRESS                         PORTS     AGE
ingress-tomcat   nginx   tomcat.aaabbbccc.com   192.168.57.132,192.168.57.133   80, 443   39s

#添加域名信息至宿主机hosts文件，如下图一：
#浏览器访问验证，如下图二、图三：

ps：访问 tomcat.aaabbbccc.com该域名时，它会访问service，然后由servic配置中指定的8080端口找到它所关联的后端pod。