首页 > 其他分享 >实现流程可控的镜像下载和存储(一)

实现流程可控的镜像下载和存储(一)

时间:2024-02-06 16:12:11浏览次数:31  
标签:存储 vnd 可控 v2 application json 镜像 docker sha256

基于https实现镜像所有相关元信息的获取

在弱网环境下,下载镜像很慢且容易出错,基于这个原因需要开发更加可靠且支持断点续传的镜像下载程序
由于Docker Hub在国内无法访问,用自己的阿里云镜像加速替代来进行测试
下面以下载linux/amd64的ubuntu22.04镜像为例

Authentication

例中的阿里云镜像加速无需校验,如果镜像源需要校验,那么首先获取token,然后加入header

    -H "Authorization: Bearer <Token>"

获取Manifest

curl -H "Accept: application/vnd.oci.image.index.v1+json" \
     -H "Accept: application/vnd.oci.image.manifest.v1+json" \
     -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
     -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
     -s https://no8ju4hi.mirror.aliyuncs.com/v2/library/ubuntu/manifests/22.04

对于多个架构的镜像仓库,会获取到所有架构的Digest[1]

ubuntu:22.04->Manifests
{
    "manifests": [
        {
            "digest": "sha256:3c3de9608507804525ff4303874525760ea36d62606e8105f515adaa761b80cb",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 529
        },
        {
            "digest": "sha256:77e15e9408b4057b6cb6b66fb004a488d42496b60d2fdc02ac73c38e3206d886",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            },
            "size": 529
        },
        {
            "digest": "sha256:c0c089b5b5b1ea9e860a12a35042d832dcb1733f69fe57089b3152577d96ea35",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            },
            "size": 529
        },
        {
            "digest": "sha256:b49ca9c2c0555ed157127cd429996e11900085b8b982414cc48483a1ca5c046b",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            },
            "size": 529
        },
        {
            "digest": "sha256:dce81b3b890c5f5637821a97f45ea9d42dda08c06a3069ae9559ce32385af4e0",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "riscv64",
                "os": "linux"
            },
            "size": 529
        },
        {
            "digest": "sha256:e45f2138cf3a542f701097c780c96a4a7b8a77d76e27f9ffc1986fda63debd43",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "s390x",
                "os": "linux"
            },
            "size": 529
        }
    ],
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "schemaVersion": 2
}

根据上面得到的Manifest,可以获取linux/amd64的镜像对应的信息

  • digest: "sha256:3c3de9608507804525ff4303874525760ea36d62606e8105f515adaa761b80cb"
  • architecture : "amd64"
  • os : "linux"
  • size : 424
curl -H "Accept: application/vnd.oci.image.index.v1+json" \
     -H "Accept: application/vnd.oci.image.manifest.v1+json" \
     -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
     -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
     -s https://no8ju4hi.mirror.aliyuncs.com/v2/library/ubuntu/manifests/sha256:3c3de9608507804525ff4303874525760ea36d62606e8105f515adaa761b80cb

对于只有一个架构的镜像,执行第一段命令行,会跳过步骤[1],直接得到下面的结果

ubuntu:22.04->sha256:3c3de9608507804525ff4303874525760ea36d62606e8105f515adaa761b80cb
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1462,
      "digest": "sha256:9d28ccdc1fc782ec635c98e55ff68b05e6de1df2c7fcbbb4385f023368eec716"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 29886454,
         "digest": "sha256:aee1767db0dd7da5c30ffaef2282976fe8730cdb0a7b1ecda17cd3c85374fa57"
      }
   ]
}

获取Metadata

根据上面的Manifest,可以得到镜像信息文件的路径
另外可以知道文件的sha256值为9d28ccdc1fc782ec635c98e55ff68b05e6de1df2c7fcbbb4385f023368eec716,大小为1462

wget https://no8ju4hi.mirror.aliyuncs.com/v2/library/ubuntu/blobs/sha256:9d28ccdc1fc782ec635c98e55ff68b05e6de1df2c7fcbbb4385f023368eec716
linux/amd64 ubuntu:22.04镜像信息
{
    "architecture": "amd64",
    "config": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "bash"
        ],
        "Image": "sha256:c0ffde4acd3f60443d26c66a72e3f18f9da32f570a45ab54ef0d56a301a50150",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": null
    },
    "container": "a2b4f5c93fe1e35e98c8624c70be1f9f61147f6568992d1036640ed69e40de6c",
    "container_config": {
        "Hostname": "a2b4f5c93fe1",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/bin/sh",
            "-c",
            "#(nop) ",
            "CMD [\"bash\"]"
        ],
        "Image": "sha256:c0ffde4acd3f60443d26c66a72e3f18f9da32f570a45ab54ef0d56a301a50150",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {}
    },
    "created": "2021-12-04T02:21:12.903923136Z",
    "docker_version": "20.10.7",
    "history": [
        {
            "created": "2021-12-04T02:21:12.507294562Z",
            "created_by": "/bin/sh -c #(nop) ADD file:1e60bfbe5a32672bfcd507fc81d927e28eb8b8b12907819058ba9bed8d6b8ac1 in / "
        },
        {
            "created": "2021-12-04T02:21:12.903923136Z",
            "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
            "empty_layer": true
        }
    ],
    "os": "linux",
    "rootfs": {
        "type": "layers",
        "diff_ids": [
            "sha256:950d1cd211572857ec798f05767ad7614025ec505d4e85df9d5e62662ed4fea9"
        ]
    }
}

下载镜像的所有blob

根据Manifest,可以知道当前镜像只有一个Layer,其对应的blob为sha256:aee1767db0dd7da5c30ffaef2282976fe8730cdb0a7b1ecda17cd3c85374fa57,大小为29886454
下载对应blob

wget https://no8ju4hi.mirror.aliyuncs.com/v2/library/ubuntu/blobs/sha256:aee1767db0dd7da5c30ffaef2282976fe8730cdb0a7b1ecda17cd3c85374fa57

元信息校验

所有的元信息都有sha256和size两个参数,当下载完成时,可以比较文件大小和sha256sum值,完全一致则下载的文件无误,否则需要重新下载

标签:存储,vnd,可控,v2,application,json,镜像,docker,sha256
From: https://www.cnblogs.com/umichan0621/p/18009675

相关文章

  • GO镜像
    UNIX#启用GoModules功能goenv-wGO111MODULE=on#配置GOPROXY环境变量,以下三选一#1.七牛CDNgoenv-wGOPROXY=https://goproxy.cn,direct#2.阿里云goenv-wGOPROXY=https://mirrors.aliyun.com/goproxy/,direct#3.官方goenv-wGOPROXY=https:......
  • 云计算 - 对象存储服务OSS技术全解
    本文全面深入地探讨了对象存储服务(OSS)的核心技术、基础知识和高级功能。从媒体存储到数据备份,再到数据仓库与数据湖,我们不仅解析了OSS在各种应用场景下的关键角色,还深入讨论了其与机器学习、多媒体处理以及日志和监控等多个开发场景的结合。关注【TechLeadCloud】,分享互联网架......
  • docker设置国内镜像源
    一、国内加速地址1、阿里云镜像站:(需登录,免费)https://<your_code>.mirror.aliyuncs.com2、网易云镜像站:http://hub-mirror.c.163.com3、百度云镜像站:https://mirror.baidubce.com4、上海交大镜像站:https://docker.mirrors.sjtug.sjtu.edu.cn5、南京大学镜像站:https://doc......
  • 联系信息的存储
    之前说过平台的用户信息是集中存储的。和用户相关的联系信息,包括手机号、电子信箱等等如何存储?是否和用户信息一样集中存储?    经过几次反复,最终决定还是分布存储。理由如下:    基本信息是分布存储的,例如人力资源系统存储内部人员信息,供应商系统存储供应商人员信息。......
  • MySQL存储引擎-InnoDB行格式
    MySQL存储引擎-InnoDB行格式mysql作为一款主流的关系型数据库,是以记录为单位向表中插入数据的。目前为止,Innodb共支持COMPACT、REDUNDANT、DYNAMIC、和COMMPRESSED四种行格式。在MySQL5.7及以上版本,默认采用DYNAMIC格式。DYNAMIC与COMPACT格式基本一致,下文中我们会介绍区别。因......
  • Docker网络与存储
    网络:bridge模式:当Docker进程启动后,会在主机上创建一个名为docker0的虚拟网桥,主机上启动的docker容器会连接到这个虚拟网桥上.从docker0子网中分配一个ip给容器使用,并设置docker0的IP地址为容器的默认网关.在主机上创建一堆虚拟网卡设备vethpair设备,Docker将vethpair设......
  • 配置镜像仓库
    配置镜像仓库安装好Docker后,其registryserver是默认指向https://hub.docker.com的。在国内该hub源访问速度异常慢,尤其是大一点的镜像经常出现timeout。我们可以通过切换至国内镜像仓库来解决这一问题配置镜像仓库1.修改配置文件/etc/docker/daemon.json:sudovim/etc......
  • VMware vSAN 8.0 U2 - 存储虚拟化软件
    VMwarevSAN8.0U2-存储虚拟化软件vSAN8withExpressStorageArchitecture请访问原文链接:https://sysin.org/blog/vmware-vsan/,查看最新版。原创作品,转载请保留出处。作者主页:sysin.orgVMwarevSAN存储虚拟化软件vSAN利用企业级存储虚拟化软件VMwarevSAN降低存......
  • 可控概率抽奖算法
    说明本文PHP语言去实现,只实现核心可控概率引擎,库存判断等其它业务需要其它代码配合实现。代码/***@function封装可控概率的抽奖功能*@param$arrarray数据集合*@param$weight_keystring权重字段*@returnarray被选中的元素*/funct......
  • containerd 镜像层分析
    meta.db找到d842e8e2623636b8fb0d070a2dd9592c1eb0ebfa975c6a283960bc1f710feab4mediatype:application/vnd.docker.distribution.manifest.v2+json多种类型application/vnd.docker.distribution.manifest.list.v2+json===================================================......