containerd的客户端工具ctr命令
类似docker为docker-shim容器运行时的客户端工具，ctr 是 containerd 的客户端工具，安装containerd作为容器运行时后，会自动安装ctr。
root@master1:~# ctr -v
ctr github.com/containerd/containerd 1.7.12
1. 帮助文档
ctr帮助文档
root@master1:~# ctr --help
containerd CLI
USAGE:
ctr [global options] command [command options] [arguments...]
VERSION:
1.7.12
DESCRIPTION:
ctr is an unsupported debug and administrative client for interacting
with the containerd daemon. Because it is unsupported, the commands,
options, and operations are not guaranteed to be backward compatible or
stable from release to release of the containerd project.
COMMANDS:
plugins, plugin Provides information about containerd plugins
version Print the client and server versions
containers, c, container Manage containers
content Manage content
events, event Display containerd events
images, image, i Manage images
leases Manage leases
namespaces, namespace, ns Manage namespaces
pprof Provide golang pprof outputs for containerd
run Run a container
snapshots, snapshot Manage snapshots
tasks, t, task Manage tasks
install Install a new package
oci OCI tools
sandboxes, sandbox, sb, s Manage sandboxes
info Print the server info
deprecations
shim Interact with a shim directly
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--debug Enable debug output in logs
--address value, -a value Address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS]
--timeout value Total timeout for ctr commands (default: 0s)
--connect-timeout value Timeout for connecting to containerd (default: 0s)
--namespace value, -n value Namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE]
--help, -h show help
--version, -v print the version
注意：
ctr与docker较大的一个区别再用ctr有命名空间的概念，下面的命令执行时候如果没加-n参数，表示default命名空间。当containerd结合k8s使用时，相关镜像一般存放到k8s.io，相关操作需要加
-n k8s.io，且将参数放到最前面。
子命令帮助文档，通过ctr+子命令查看帮助信息：
root@master1:~# ctr plugins
NAME:
ctr plugins - Provides information about containerd plugins
USAGE:
ctr plugins command [command options] [arguments...]
COMMANDS:
list, ls Lists containerd plugins
OPTIONS:
--help, -h show help
2. 镜像操作
帮助信息：
root@master1:~# ctr images
NAME:
ctr images - Manage images
USAGE:
ctr images command [command options] [arguments...]
COMMANDS:
check Check existing images to ensure all content is available locally
export Export images
import Import images
list, ls List images known to containerd
mount Mount an image to a target path
unmount Unmount the image from the target
pull Pull an image from a remote
push Push an image to a remote
prune Remove unused images
delete, del, remove, rm Remove one or more images by reference
tag Tag an image
label Set and clear labels for an image
convert Convert an image
usage Display usage of snapshots for a given image ref
OPTIONS:
--help, -h show help
拉取镜像:
# 帮助信息
[root@k8s ~]# ctr images pull --help
NAME:
ctr images pull - pull an image from a remote
USAGE:
ctr images pull [command options] [flags] <ref>
DESCRIPTION:
Fetch and prepare an image for use in containerd.
After pulling an image, it should be ready to use the same reference in a run
command. As part of this process, we do the following:
1. Fetch all resources into containerd.
2. Prepare the snapshot filesystem with the pulled resources.
3. Register metadata for the image.
OPTIONS:
--skip-verify, -k skip SSL certificate validation
--plain-http allow connections using plain HTTP
--user value, -u value user[:password] Registry user and password
--refresh value refresh token for authorization server
--hosts-dir value Custom hosts configuration directory
--tlscacert value path to TLS root CA
--tlscert value path to TLS client certificate
--tlskey value path to TLS client key
--http-dump dump all HTTP request/responses when interacting with container registry
--http-trace enable HTTP tracing for registry interactions
--snapshotter value snapshotter name. Empty value stands for the default value. [$CONTAINERD_SNAPSHOTTER]
--label value labels to attach to the image
--platform value Pull content from a specific platform
--all-platforms pull content and metadata from all platforms
--all-metadata Pull metadata for all platforms
--print-chainid Print the resulting image's chain ID
--max-concurrent-downloads value Set the max concurrent downloads for each pull (default: 0)
# 拉取nginx镜像，注意与docker拉取镜像不同，镜像名不能简写。
[root@k8s ~]# ctr i pull docker.io/library/nginx:latest
docker.io/library/nginx:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:9c367186df9a6b18c6735357b8eb7f407347e84aea09beb184961cb83543d46e: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:4e02e85a6f060a8406978fa53aafd2d828d0cedf5259275d191bab9afc33249e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a06b6fd631e8e2091ce18db1a1b063f14f06a63d0513cafc51500ce7cb1ae2f4: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:e0c9858e10ed8be697dc2809db78c57357ffc82de88c69a3dee5d148354679ef: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1018f2b8dba8ffec0afc981d5bae673e38ecac6ca29a861d12cccbd820d53f8d: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b831e78d8e20641c11c527baff6e2c7bae6a43cc1ab4316a5532885d6461b1e7: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3ab22521e91957c19caeb12eadcad2823fdca853477acb38cd5e9a1ebe57e96e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5112bf42775bbb6a896ccd3ad63cbf864976fb1c047a4c56f316cf78d3edd756: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:cbdaf9e4ee2d8507bf2e162c560cfb0b37567db3870235b8940aeb157d628327: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 17.1s total: 67.7 M (4.0 MiB/s)
unpacking linux/amd64 sha256:9c367186df9a6b18c6735357b8eb7f407347e84aea09beb184961cb83543d46e...
done: 1.164220475s
备注：
测试发现ctr拉取镜像如果要配置代理的话，走的是linux全局代理配置方式，而不是docker或者containerd单独的代理配置方式。参考方式：
export proxy=http://192.168.0.210:808 export all_proxy=$proxy export http_proxy=$proxy export https_proxy=$proxy export no_proxy="localhost, 127.0.0.1, ::1"
打印镜像列表:
# ctr images ls 等同于 ctr -n default images ls;同时images可以简写为image或i
root@master1:~# ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/redis:6.0.5 application/vnd.docker.distribution.manifest.list.v2+json sha256:800f2587bf3376cb01e6307afe599ddce9439deafbd4fb8562829da96085c9c5 36.4 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x - -
# -q，只打印镜像名称
root@master1:~# ctr images ls -q
docker.io/library/redis:6.0.5
检测本地镜像:
# 检查本地镜像的完整性
root@master1:~# ctr images check
...
root@master1:~# ctr -n k8s.io images check
...
镜像重新打标签:
root@master1:~# ctr i tag docker.io/library/redis:6.0.5 docker.io/lldhsds/redis:6.0.5
docker.io/lldhsds/redis:6.0.5
root@master1:~# ctr i ls -q
docker.io/library/redis:6.0.5
docker.io/lldhsds/redis:6.0.5
删除镜像:
root@master1:~# ctr i rm docker.io/lldhsds/redis:6.0.5
docker.io/lldhsds/redis:6.0.5
将镜像导出为压缩包:
# 指定命名空间，指定--platform参数
root@worker1:~# ctr -n k8s.io image export --platform=linux/amd64 guestbook.tar docker.io/library/redis:6.0.5
root@worker1:~# ls
guestbook.tar guestbook.tar.gz
从压缩包导入镜像:
# 导入到default命名空间下
root@master1:~# ctr image import --platform=linux/amd64 guestbook.tar
unpacking docker.io/library/redis:6.0.5 (sha256:800f2587bf3376cb01e6307afe599ddce9439deafbd4fb8562829da96085c9c5)...done
# 导入到k8s.io命名空间下
root@master1:~# ctr -n k8s.io image import --platform=linux/amd64 guestbook.tar
unpacking docker.io/library/redis:6.0.5 (sha256:800f2587bf3376cb01e6307afe599ddce9439deafbd4fb8562829da96085c9c5)...done
备注：
- 使用ctr导出、导入镜像时一般都需要加上
--platform=linux/amd64参数(x86-64架构)，否则会出现类似问题ctr: content digest sha256:a630f3b108d2562d7a6747a0d3b5181b8f5a9aa489b8915fccd04de94cdc96bf: not found- 如果是从docker侧打包的镜像，使用ctr打包，测试可以不加
--platform=linux/amd64参数
将镜像挂载到主机目录:
root@worker1:~# ctr i mount docker.io/library/redis:6.0.5 /mnt
ctr: image "docker.io/library/redis:6.0.5": not found
root@worker1:~# ctr -n k8s.io i mount docker.io/library/redis:6.0.5 /mnt
sha256:852691351e76013456bccc5ca476ea5998cd4ef829ff88f36aa6152d26752a9f
/mnt
root@worker1:~# ls /mnt -alh
total 88K
drwxr-xr-x 1 root root 4.0K Jun 21 21:27 .
drwxr-xr-x 23 root root 4.0K Jun 1 01:04 ..
drwxr-xr-x 2 root root 4.0K Jun 7 2020 bin
drwxr-xr-x 2 root root 4.0K May 3 2020 boot
drwxr-xr-x 2 999 systemd-journal 4.0K Jun 10 2020 data
drwxr-xr-x 2 root root 4.0K Jun 7 2020 dev
drwxr-xr-x 1 root root 4.0K Jun 10 2020 etc
drwxr-xr-x 2 root root 4.0K May 3 2020 home
drwxr-xr-x 1 root root 4.0K Jun 10 2020 lib
drwxr-xr-x 2 root root 4.0K Jun 7 2020 lib64
drwxr-xr-x 2 root root 4.0K Jun 7 2020 media
drwxr-xr-x 2 root root 4.0K Jun 7 2020 mnt
drwxr-xr-x 2 root root 4.0K Jun 7 2020 opt
drwxr-xr-x 2 root root 4.0K May 3 2020 proc
drwx------ 1 root root 4.0K Jun 10 2020 root
drwxr-xr-x 3 root root 4.0K Jun 7 2020 run
drwxr-xr-x 2 root root 4.0K Jun 7 2020 sbin
drwxr-xr-x 2 root root 4.0K Jun 7 2020 srv
drwxr-xr-x 2 root root 4.0K May 3 2020 sys
drwxrwxrwt 1 root root 4.0K Jun 10 2020 tmp
drwxr-xr-x 1 root root 4.0K Jun 7 2020 usr
drwxr-xr-x 1 root root 4.0K Jun 7 2020 var
将镜像从主机目录上卸载:
root@worker1:~# ctr i unmount /mnt
/mnt
root@worker1:~# ls /mnt
3. 容器操作
帮助信息：
root@worker1:~# ctr container
NAME:
ctr containers - Manage containers
USAGE:
ctr containers command [command options] [arguments...]
COMMANDS:
create Create container
delete, del, remove, rm Delete one or more existing containers
info Get info about a container
list, ls List containers
label Set and clear labels for a container
checkpoint Checkpoint a container
restore Restore a container from checkpoint
OPTIONS:
--help, -h show help
创建容器：
# 创建一个nginx容器
root@master1:~# ctr c create docker.io/library/nginx:1.27.0 nginx
# 打印容器列表
root@master1:~# ctr c ls
CONTAINER IMAGE RUNTIME
nginx docker.io/library/nginx:1.27.0 io.containerd.runc.v2
查看容器详细信息：
root@master1:~# ctr c info nginx
{
"ID": "nginx",
"Labels": {
"io.containerd.image.config.stop-signal": "SIGQUIT",
"maintainer": "NGINX Docker Maintainers \[email protected]\u003e"
},
"Image": "docker.io/library/nginx:1.27.0",
"Runtime": {
"Name": "io.containerd.runc.v2",
"Options": {
"type_url": "containerd.runc.v1.Options"
}
},
"SnapshotKey": "nginx",
"Snapshotter": "overlayfs",
"CreatedAt": "2024-06-22T07:08:10.924977879Z",
"UpdatedAt": "2024-06-22T07:08:10.924977879Z",
"Extensions": {},
...
删除容器：
root@master1:~# ctr c ls
CONTAINER IMAGE RUNTIME
nginx docker.io/library/nginx:1.27.0 io.containerd.runc.v2
root@master1:~# ctr c rm nginx
root@master1:~# ctr c ls
CONTAINER IMAGE RUNTIME
4. 任务
通过 container create 命令创建的容器，并没有处于运行状态，仅仅只是一个创建容器的声明。一个 container 对象包含了运行一个容器所需的资源及相关配置数据，表示 namespaces、rootfs 和容器的配置都已经初始化成功了，只是用户进程还没有启动。
启动并进入容器：
root@master1:~# ctr c create docker.io/library/nginx:1.27.0 nginx
root@master1:~# ctr c ls
CONTAINER IMAGE RUNTIME
nginx docker.io/library/nginx:1.27.0 io.containerd.runc.v2
# 启动一个容器
root@master1:~# ctr task start -d nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
# 查看task列表，即运行的容器列表
root@master1:~# ctr task ls
TASK PID STATUS
nginx 63588 RUNNING
# 进入容器里面，需要指定--exec-id参数，值可以随意。
root@master1:~# ctr task exec --exec-id 0 -t nginx sh
# ls
bin dev docker-entrypoint.sh home lib64 mnt proc run srv tmp var
boot docker-entrypoint.d etc lib media opt root sbin sys usr
#
暂停/恢复容器：
root@master1:~# ctr task ls
TASK PID STATUS
nginx 63588 RUNNING
# 暂停容器
root@master1:~# ctr task pause nginx
root@master1:~# ctr task ls
TASK PID STATUS
nginx 63588 PAUSED
# 恢复容器
root@master1:~# ctr t resume nginx
root@master1:~# ctr task ls
TASK PID STATUS
nginx 63588 RUNNING
杀死容器，同停止容器：
root@master1:~# ctr t kill nginx
root@master1:~# ctr task ls
TASK PID STATUS
nginx 63588 STOPPED
删除task：
root@master1:~# ctr t ls
TASK PID STATUS
nginx 63588 STOPPED
root@master1:~# ctr t rm nginx
# 运行状态下删除task：
root@master1:~# ctr task ls
TASK PID STATUS
nginx 67604 RUNNING
root@master1:~# ctr t rm nginx
ERRO[0000] unable to delete nginx error="task must be stopped before deletion: running: failed precondition"
ctr: task must be stopped before deletion: running: failed precondition
说明：
删除task之前必须停止或kill容器。
获取容器的 cgroup 相关信息：
root@master1:~# ctr t metrics nginx
ID TIMESTAMP
nginx seconds:1719041526 nanos:710831637
METRIC VALUE
pids.current 5
pids.limit 18446744073709551615
cpu.usage_usec 24082
cpu.user_usec 6337
cpu.system_usec 17745
cpu.nr_periods 0
cpu.nr_throttled 0
cpu.throttled_usec 0
memory.usage 4849664
memory.usage_limit 18446744073709551615
memory.swap_usage 0
memory.swap_limit 18446744073709551615
查看容器中所有进程在宿主机中的 PID：
root@master1:~# ctr t ps nginx
PID INFO
67604 -
67644 -
67645 -
67646 -
67647 -
# 与ps命令查到的进程ID匹配。
root@master1:~# ps -elf | grep nginx | grep -v grep
0 S root 67583 1 0 80 0 - 309523 futex_ 15:29 ? 00:00:00 /usr/bin/containerd-shim-runc-v2 -namespace default -id nginx -address /run/containerd/containerd.sock
4 S root 67604 67583 0 80 0 - 2851 sigsus 15:29 ? 00:00:00 nginx: master process nginx -g daemon off;
5 S message+ 67644 67604 0 80 0 - 2975 ep_pol 15:29 ? 00:00:00 nginx: worker process
5 S message+ 67645 67604 0 80 0 - 2975 ep_pol 15:29 ? 00:00:00 nginx: worker process
5 S message+ 67646 67604 0 80 0 - 2975 ep_pol 15:29 ? 00:00:00 nginx: worker process
5 S message+ 67647 67604 0 80 0 - 2975 ep_pol 15:29 ? 00:00:00 nginx: worker process
5. 命名空间
Containerd中是用命名空间进行资源区分的。
命名空间生命周期管理：
# 查看ns列表
root@master1:~# ctr ns ls
NAME LABELS
default
k8s.io
# 创建命名空间
root@master1:~# ctr ns create test
root@master1:~# ctr ns ls
NAME LABELS
default
k8s.io
test
# 删除命名空间
root@master1:~# ctr ns rm test
test
指定命名空间选项：
root@master1:~# ctr i ls -q
docker.io/library/nginx:1.27.0
docker.io/library/redis:6.0.5
us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5
us-docker.pkg.dev/google-samples/containers/gke/gb-redis-follower:v2
root@master1:~# ctr -n k8s.io i ls -q
docker.io/flannel/flannel-cni-plugin:v1.4.1-flannel1
docker.io/flannel/flannel-cni-plugin@sha256:e88c0d84fa89679eb6cb6a28bc257d652ced8d1b2e44d54a592f0a2cd85dba53
docker.io/flannel/flannel:v0.25.4
docker.io/flannel/flannel@sha256:17415d91743e53fc4b852676a30a08915f131a2b6848d891ba5786eacd447076
docker.io/library/redis:6.0.5
quay.io/tigera/operator:v1.34.0
quay.io/tigera/operator@sha256:479ddc7ff9ab095058b96f6710bbf070abada86332e267d6e5dcc1df36ba2cc5
registry.aliyuncs.com/google_containers/coredns:v1.10.1
备注：
在使用ctr指定命名空间进行操作时，将
-n xxx选项放到命令的前面。
6. ctr与docker
ctr 是 containerd 自带的工具， 有 namespaces 的概念，对于上层编排系统的支持，ctr 客户端 主要区分了 3 个命名空间分别是k8s.io、moby和default。
较新版本的Docker 底层默认调用的 也是containerd。 Docker 使用 containerd 下面的命名空间默认 moby，而不是 default，所以假如我们有用 docker 启动容器，那么我们也可以通过 ctr -n moby 来定位下面的容器：
[root@k8s ~]# docker run -d --name nginx nginx:latest
b6f9e425cc80ef576a1878ac5f94727cdee6397b845d2f0101263cd0ab00996c
[root@k8s ~]# ctr -n moby c ls | grep b6f9e425cc
b6f9e425cc80ef576a1878ac5f94727cdee6397b845d2f0101263cd0ab00996c - io.containerd.runc.v2
遗留问题：
按照上面的说法类推，测试发现docker的镜像与ctr的镜像在moby命名空间下是对不上的，目前没有查到相关解释，作为遗留问题记录。测试版本：
docker client:26.1.4，docker server:20.10.6，containerd.io 1.6.33。
7. ctr与k8s、ctrictl
Kubernetes 下使用的 containerd 默认命名空间是 k8s.io。所以在导入镜像时需要指定命令空间为 k8s.io，否则使用 crictl images 无法查询到。
#docker导出镜像
docker save -o kube-controllers.tar docker.io/calico/kube-controllers:v3.26.1
# ctr导出镜像(也支持多个镜像一起打包)
ctr -n k8s.io images export k8s-v1.28.2.tar registry.k8s.io/kube-apiserver:v1.28.2 registry.k8s.io/kube-controller-manager:v1.28.2 registry.k8s.io/kube-proxy:v1.28.2 registry.k8s.io/kube-scheduler:v1.28.2 registry.k8s.io/coredns/coredns:v1.10.1 registry.k8s.io/etcd:3.5.9-0
#ctr导入镜像
ctr -n k8s.io image import kube-controllers.tar
#查看镜像是否导入
crictl images
我们用crictl操作的均在k8s.io命名空间，使用ctr 看镜像列表就需要加上-n 参数。crictl 是只有一个k8s.io命名空间，但是没有-n 参数。
ctr images pull 拉取的镜像默认放在default，而 crictl pull 和 kubelet 默认拉取的镜像都在 k8s.io 命名空间下。所以通过ctr导入镜像的时候特别注意一点，最好指定命名空间。
# 注意-n不能放在命令最后面，下面几行查看的镜像是一样的
ctr -n=k8s.io image ls
ctr -n k8s.io image ls
# crictl 没有-n参数，操作都在k8s.io命名空间下。
crictl images
# crictl image list/ls 等同于 ctr -n=k8s.io image list/ls
使用ctr命令指定命名空间导入镜像
ctr -n=k8s.io image import dashboard.tar
#查看镜像，可以看到可以查询到了
crictl images