k8s 1.28.2 集群部署 docker registry 接入 MinIO 存储

• docker registry 部署
  • 生成 htpasswd 文件
    • 生成 secret 文件
  • 生成 registry 配置文件
  • 创建 service
  • 创建 statefulset
  • 创建 ingress
  • 验证 docker registry
• docker registry 监控
• docker registry ui

docker registry dockerfile

docker registry 配置文件

S3 storage driver

registry:2.8.3 Image hierarchy

docker registry 部署

生成 htpasswd 文件

<username> <password> 改成自己想配置的，如果密码有特殊字符，要用单引号包起来

docker run --rm \
  docker.m.daocloud.io/httpd:latest \
  htpasswd -Bbn <username> <password> > htpasswd

生成 secret 文件

kubectl create secret generic docker-registry-auth \
  -n registry \
  --from-file=htpasswd

生成 registry 配置文件

因为涉及到 MinIO 的 accesskey 和 secretkey，这里采用 secret 的方式来生成配置文件

---
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-cm
  namespace: registry
stringData:
  config.yml: |-
    version: 0.1
    log:
      level: info
      fields:
        service: registry
    storage:
      delete:
        enabled: true
      cache:
          blobdescriptor: inmemory
      s3:
        accesskey: wJpkHB8rznvZBRLfKmBz
        secretkey: ZHIyklv5tktYvGR0iFqBiL9NKh7JKbhyDR9SNAYp
        region: default
        regionendpoint: http://minio.api.devops.icu
        forcepathstyle: true
        accelerate: false
        bucket: docker-registry
        encrypt: false
        secure: false
        v4auth: true
        chunksize: 5242880
        multipartcopymaxconcurrency: 10
    http:
      addr: :5000
      debug:
        addr: :5001
        prometheus:
            enabled: true
            path: /metrics
      headers:
        X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3
    auth:
      htpasswd:
        realm: basic-realm
        path: /auth/htpasswd
type: Opaque

创建 service

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: docker-registry
  name: docker-registry-svc
  namespace: registry
spec:
  ports:
  - name: http
    port: 5000
    targetPort: http
  - name: http-metrics
    port: 5001
    targetPort: http-metrics
  selector:
    app.kubernetes.io/name: docker-registry
  type: ClusterIP

创建 statefulset

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: docker-registry
  name: docker-registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: docker-registry
  serviceName: docker-registry-svc
  template:
    metadata:
      labels:
        app.kubernetes.io/name: docker-registry
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: docker-registry
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - image: docker.m.daocloud.io/registry:2.8.3
        livenessProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        name: docker-registry
        ports:
        - containerPort: 5000
          name: http
        - containerPort: 5001
          name: http-metrics
        readinessProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 2000m
            memory: 2.5Gi
          requests:
            cpu: 100m
            memory: 100Mi
        startupProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        volumeMounts:
        - mountPath: /etc/docker/registry
          name: config
        - mountPath: /auth
          name: auth
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        secret:
          secretName: docker-registry-cm
      - name: auth
        secret:
          secretName: docker-registry-auth

创建 ingress

没有 ingress 可以开 nodeport 来实现

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 5G
  name: docker-registry
  namespace: registry
spec:
  ingressClassName: nginx
  rules:
  - host: registry.devops.icu
    http:
      paths:
      - backend:
          service:
            name: docker-registry-svc
            port:
              number: 5000
        path: /
        pathType: Prefix

验证 docker registry

/etc/docker/daemon.json 增加 registry 地址

"insecure-registries": ["ip:端口"]
# 或者
"insecure-registries": ["域名"]

登录 docker registry

docker login http://registry.devops.icu

修改 tag

docker tag docker.m.daocloud.io/registry:2.8.3 registry.devops.icu/registry:2.8.3

上传镜像

docker push registry.devops.icu/registry:2.8.3

docker registry 监控

grafana id：9621

prometheus 配置文件参考

    - job_name: docker-registry
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        regex: registry;docker-registry-svc
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:5001
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

docker registry ui

Github 项目地址：Joxit/docker-registry-ui-2.5.7

相关的变量和参数详见：available-options

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: docker-registry-ui
  name: docker-registry-ui-svc
  namespace: registry
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: docker-registry-ui
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: docker-registry-ui
  name: docker-registry-ui
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: docker-registry-ui
  template:
    metadata:
      labels:
        app.kubernetes.io/name: docker-registry-ui
    spec:
      containers:
      - env:
        - name: SINGLE_REGISTRY
          value: "true"
        - name: SHOW_CATALOG_NB_TAGS
          value: "true"
        - name: REGISTRY_SECURED
          value: "true"
        - name: NGINX_PROXY_PASS_URL
          value: http://docker-registry-svc.registry.svc.cluster.local:5000
        - name: NGINX_PROXY_HEADER_Authorization
          value: $http_authorization
        image: joxit/docker-registry-ui:2.5.7
        imagePullPolicy: IfNotPresent
        name: docker-registry-ui
      securityContext:
        fsGroup: 101
        runAsUser: 101
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry-ui
  namespace: registry
spec:
  ingressClassName: nginx
  rules:
  - host: registry.ui.devops.icu
    http:
      paths:
      - backend:
          service:
            name: docker-registry-ui-svc
            port:
              number: 8080
        path: /
        pathType: Prefix