当我们的k8s没有使用docker的容器运行时的时候，管理查询镜像就需要用到其他Containerd 客户端工具
containerd 的客户端工具有ctr,crictl和 nerdctl
ctr和crictl
ctr 是由 containerd 提供的一个客户端工具。
crictl 是 CRI 兼容的容器运行时命令行接口，和containerd无关，由Kubernetes提供，可以使用它来检查和调试 k8s 节点上的容器运行时和应用程序。
| 命令 | docker | ctr（containerd） | crictl（kubernetes） |
|---|---|---|---|
| 查看镜像 | docker images | ctr image ls | crictl images |
| 拉取镜像 | docker pull | ctr image pull | ctictl pull |
| 推送镜像 | docker push | ctr image push | 无 |
| 删除镜像 | docker rmi | ctr image rm | crictl rmi |
| 导入镜像 | docker load | ctr image import | 无 |
| 导出镜像 | docker save | ctr image export | 无 |
| 修改镜像标签 | docker tag | ctr image tag | 无 |
| 创建一个新的容器 | docker create | ctr container create | crictl create |
| 运行一个新的容器 | docker run | ctr run | 无（最小单元为Pod） |
| 删除容器 | docker rm | ctr container rm | crictl rm |
| 查看运行的容器 | docker ps | ctr task ls ctr container ls |
crictl ps |
| 启动已有的容器 | docker start | ctr task start | crictl start |
| 关闭已有的容器 | docker stop | ctr task kill | crictl stop |
| 在容器内部执行命令 | docker exec | 无 | crictl exec |
| 查看容器数信息 | docker inspect | ctr container info | crictl inspect |
| 查看容器日志 | docker logs | 无 | crictl logs |
| 查看容器资源 | docker stats | 无 | crictl stats |
注意：
- 要用ctr操作或查看k8s的相关镜像或者容器，需要在前面加上
-n k8s.io- 例如，如果要把打包好的镜像加载到k8s中
ctr -n=k8s.io images import k8s_1.25.0.tar.gz - 要查看k8s中的镜像
ctr -n=k8s.io image ls
- 例如，如果要把打包好的镜像加载到k8s中
ctr导出镜像
$ ctr -n k8s.io image export es.v8.5.0.tar.gz docker.elastic.co/elasticsearch/elasticsearch:8.5.0
ctr导入镜像
$ ctr -n k8s.io image import es.v8.5.0.tar.gz
nerdctl
nerdctl 介绍
nerdctl 是 与 Docker 兼容的CLI for Containerd，其支持Compose
nerdctl 和 docker命令行语法很相似，学习比较容易
项目地址：https://github.com/containerd/nerdctl
nerdctl 官方发布包包含两个安装版本:
- Minimal: 仅包含 nerdctl 二进制文件以及 rootless 模式下的辅助安装脚本
- Full: 看名字就能知道是个全量包，其包含了 Containerd、CNI、runc、BuildKit 等完整组件
下载地址https://github.com/containerd/nerdctl/releases
nerdctl 安装和使用
在单机使用 nerdctl 代替 Docker
#在新主机使用nerdctl/代替docker
$ wget https://github.com/containerd/nerdctl/releases/download/v0.23.0/nerdctl-full-0.23.0-linux-amd64.tar.gz
#查看文件内容
$ tar tf nerdctl-full-0.23.0-linux-amd64.tar.gz
bin/
bin/buildctl
bin/buildg
bin/buildg.sh
bin/buildkitd
bin/bypass4netns
bin/bypass4netnsd
bin/containerd
bin/containerd-fuse-overlayfs-grpc
bin/containerd-rootless-setuptool.sh
bin/containerd-rootless.sh
bin/containerd-shim-runc-v2
bin/containerd-stargz-grpc
bin/ctd-decoder
bin/ctr
bin/ctr-enc
bin/ctr-remote
bin/fuse-overlayfs
bin/ipfs
bin/nerdctl
bin/rootlessctl
bin/rootlesskit
bin/runc
bin/slirp4netns
bin/tini
lib/
lib/systemd/
lib/systemd/system/
lib/systemd/system/buildkit.service
lib/systemd/system/containerd.service
lib/systemd/system/stargz-snapshotter.service
libexec/
libexec/cni/
libexec/cni/bandwidth
libexec/cni/bridge
libexec/cni/dhcp
libexec/cni/firewall
libexec/cni/host-device
libexec/cni/host-local
libexec/cni/ipvlan
libexec/cni/loopback
libexec/cni/macvlan
libexec/cni/portmap
libexec/cni/ptp
libexec/cni/sbr
libexec/cni/static
libexec/cni/tuning
libexec/cni/vlan
libexec/cni/vrf
share/
share/doc/
share/doc/nerdctl/
share/doc/nerdctl/README.md
share/doc/nerdctl/docs/
share/doc/nerdctl/docs/build.md
share/doc/nerdctl/docs/builder-debug.md
share/doc/nerdctl/docs/cni.md
share/doc/nerdctl/docs/compose.md
share/doc/nerdctl/docs/config.md
share/doc/nerdctl/docs/cosign.md
share/doc/nerdctl/docs/dir.md
share/doc/nerdctl/docs/experimental.md
share/doc/nerdctl/docs/faq.md
share/doc/nerdctl/docs/freebsd.md
share/doc/nerdctl/docs/gpu.md
share/doc/nerdctl/docs/ipfs.md
share/doc/nerdctl/docs/multi-platform.md
share/doc/nerdctl/docs/nydus.md
share/doc/nerdctl/docs/ocicrypt.md
share/doc/nerdctl/docs/overlaybd.md
share/doc/nerdctl/docs/registry.md
share/doc/nerdctl/docs/rootless.md
share/doc/nerdctl/docs/stargz.md
share/doc/nerdctl-full/
share/doc/nerdctl-full/README.md
share/doc/nerdctl-full/SHA256SUMS
$ tar xf nerdctl-full-0.23.0-linux-amd64.tar.gz -C /usr/local
$ systemctl enable containerd
#查看用法
$ nerdctl --help
nerdctl is a command line interface for containerd
Config file ($NERDCTL_TOML): /etc/nerdctl/nerdctl.toml
Usage: nerdctl [flags]
Management commands:
apparmor Manage AppArmor profiles
builder Manage builds
container Manage containers
image Manage images
ipfs Distributing images on IPFS
namespace Manage containerd namespaces
network Manage networks
system Manage containerd
volume Manage volumes
Commands:
build Build an image from a Dockerfile. Needs buildkitd to be running.
commit Create a new image from a container's changes
completion Generate the autocompletion script for the specified shell
compose Compose
cp Copy files/folders between a running container and the local filesystem.
create Create a new container. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
events Get real time events from the server
exec Run a command in a running container
help Help about any command
history Show the history of an image
images List images
info Display system-wide information
inspect Return low-level information on objects.
internal DO NOT EXECUTE MANUALLY
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container. Currently, only containers created with `nerdctl run -d` are supported.
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image from a registry. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
push Push an image or a repository to a registry. Optionally specify "ipfs://" or "ipns://" scheme to push image to IPFS.
rename rename a container
restart Restart one or more running containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
save Save one or more images to a tar archive (streamed to STDOUT by default)
start Start one or more running containers
stats Display a live stream of container(s) resource usage statistics.
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update one or more running containers
version Show the nerdctl version information
wait Block until one or more containers stop, then print their exit codes.
Flags:
-H, --H string Alias of --address (default "/run/containerd/containerd.sock")
-a, --a string Alias of --address (default "/run/containerd/containerd.sock")
--address string containerd address, optionally with "unix://" prefix [$CONTAINERD_ADDRESS] (default "/run/containerd/containerd.sock")
--cgroup-manager string Cgroup manager to use ("cgroupfs"|"systemd") (default "cgroupfs")
--cni-netconfpath string cni config directory [$NETCONFPATH] (default "/etc/cni/net.d")
--cni-path string cni plugins binary directory [$CNI_PATH] (default "/usr/local/libexec/cni")
--data-root string Root directory of persistent nerdctl state (managed by nerdctl, not by containerd) (default "/var/lib/nerdctl")
--debug debug mode
--debug-full debug mode (with full output)
--experimental Control experimental: https://github.com/containerd/nerdctl/blob/master/docs/experimental.md [$NERDCTL_EXPERIMENTAL] (default true)
-h, --help help for nerdctl
--host string Alias of --address (default "/run/containerd/containerd.sock")
--hosts-dir strings A directory that contains <HOST:PORT>/hosts.toml (containerd style) or <HOST:PORT>/{ca.cert, cert.pem, key.pem} (docker style) (default [/etc/containerd/certs.d,/etc/docker/certs.d])
--insecure-registry skips verifying HTTPS certs, and allows falling back to plain HTTP
-n, --n string Alias of --namespace (default "default")
--namespace string containerd namespace, such as "moby" for Docker, "k8s.io" for Kubernetes [$CONTAINERD_NAMESPACE] (default "default")
--snapshotter string containerd snapshotter [$CONTAINERD_SNAPSHOTTER] (default "overlayfs")
--storage-driver string Alias of --snapshotter (default "overlayfs")
-v, --version version for nerdctl
Run 'nerdctl COMMAND --help' for more information on a command.
测试
$ nerdctl run -d --name nginx -p 80:80 nginx:alpine
$ nerdctl ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
4dc4a4e4d872 docker.io/library/nginx:alpine "/docker-entrypoint...." 5
seconds ago Up 0.0.0.0:80->80/tcp nginx
$ curl 127.0.0.1
在 Kubernetes 集群中使用 nerdctl
#在使用containerd的kubernetes环境中安装nerdctl
$ wget https://github.com/containerd/nerdctl/releases/download/v0.23.0/nerdctl-0.23.0-linux-amd64.tar.gz
#查看文件内容
$ tar tf nerdctl-0.23.0-linux-amd64.tar.gz
nerdctl
containerd-rootless-setuptool.sh
containerd-rootless.sh
#安装至指定目录
$ tar xf nerdctl-0.23.0-linux-amd64.tar.gz -C /usr/local/bin/
$ ls /usr/local/bin/
containerd-rootless-setuptool.sh containerd-rootless.sh nerdctl
注意：加-n k8s.io 选项才能查看到k8s的名称空间的镜像和容器
#查看k8s名称空间的镜像
$ nerdctl -n k8s.io images
#查看k8s名称空间的容器
$ nerdctl -n k8s.io ps
#查看默认名称空间default的镜像
$ nerdctl images
#查看默认名称空间default的容器
$ nerdctl ps
如果是使用apt安装的cni插件创建容器时会出下面错误
$ nerdctl run -d --name nginx -p 80:80 nginx:alpine
FATA[0010] failed to create shim: OCI runtime create failed: runc create failed:
unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2022-09-10T12:56:43Z" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\"failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports[\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/206d99263af985df7dce896e29451d8ee31234fd0a55b19eb3bde39d2b1bfdd9/oci-hook.createRuntime.log: file already closed:unknown
$ mv /opt/cni/bin/* /srv
#下载cni插件
$ wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
$ tar xf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin/
#启动成功
$ nerdctl start nginx
#或者删除再启动成功
$ nerdctl run -d --name nginx -p 80:80 nginx:alpine
FATA[0000] name "nginx" is already used by ID "206d99263af985df7dce896e29451d8ee31234fd0a55b19eb3bde39d2b1bfdd9"
$ nerdctl rm -f nginx
nginx
$ nerdctl run -d --name nginx -p 80:80 nginx:alpine
6b7ea18022d3e1293a5149f316742af093e061f3a968eb80762d5fd976dec595
$ nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
94bdff6354c0 docker.io/library/nginx:alpine "/docker-entrypoint...." 4 seconds ago Up 0.0.0.0:80->80/tcp nginx
nerdctl的命令补全
在/etc/profile里添加source <(nerdctl completion bash)
$ vim /etc/profile
source <(nerdctl completion bash)
$ . /etc/profile
坑
在查询与k8s相关的镜像或容器时一定要加上-n k8s.io
例如
#查看k8s名称空间的镜像
$ nerdctl -n k8s.io images
#查看k8s名称空间的容器
$ nerdctl -n k8s.io ps