移植docker容器到一种CPU架构

docker包含组件 docker/            docker-cli/        docker-compose/    docker-containerd/ docker-engine/     docker-proxy/ 编译打包 1、.stamp_built' failed，要看前面的错误，一般都是前面有错误停止了导致.stamp_built文件没有生成，如果前面没有错误，尝试make clean后重新make。 2、menuconfig使能需要在go.mk添加csky      else ifeq ($(BR2_csky),y)      GO_GOARCH = csky 3、更换golang直接将交叉工具链放去install，跳过golang的build步骤。  编译CLI和contained 4、提示SP错误，修改gcc_cs.S（关闭CGO后没有此问题） 5、 解决类型未定义_CPU_SETSIZE问题。 将output\host\lib\go\src\cmd\vendor\golang.org\x\sys\unix\中的 ./zerrors_linux_csky.go ./zsyscall_linux_csky.go ./syscall_linux_csky.go ./asm_linux_csky.s ./ztypes_linux_csky.go ./zsysnum_linux_csky.go 拷贝到output\build\docker-cli-19.03.8\vendor\golang.org\x\sys\unix。 cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/unix/* output/build/docker-cli-19.03.8/vendor/golang.org/x/sys/unix/ cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/unix/* output/build/docker-containerd-1.2.13/vendor/golang.org/x/sys/unix/ 6、解决提示缺少internal目录下的unsafeheader文件： cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/internal/ output/build/docker-cli-19.03.8/gopath/src/github.com/docker/cli/vendor/golang.org/x/sys/ -r cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/internal/ output/build/docker-containerd-1.2.13/vendor/golang.org/x/sys/ -r 7、提示错误： vendor/go.etcd.io/bbolt/db.go:109:13: undeclared name maxMapSize for array length vendor/go.etcd.io/bbolt/db.go:376:12: undefined: maxMapSize vendor/go.etcd.io/bbolt/db.go:394:10: undefined: maxMapSize 解决方法： cp output/build/docker-containerd-1.2.13/vendor/go.etcd.io/bbolt/bolt_arm.go output/build/docker-containerd-1.2.13/vendor/go.etcd.io/bbolt/bolt_csky.go   8、编译contained除了和cli类似错误，新增提示unknown reloc to io.ErrUnexpectedEOF: 67 (R_CSKY_ADDR32_HILO) 解决方法：CGO_ENABLE = 0;     编译docker-engine 9、提示错误：vendor/go.etcd.io/bbolt/db.go:400:12: undefined: maxMapSize 解决方法： cp output/build/docker-engine-19.03.8/vendor/go.etcd.io/bbolt/bolt_arm.go output/build/docker-engine-19.03.8/vendor/go.etcd.io/bbolt/bolt_csky.go 10、提示错误： vendor/golang.org/x/sys/unix/affinity_linux.go:13:20: undefined: _CPU_SETSIZE vendor/golang.org/x/sys/unix/affinity_linux.go:13:35: undefined: _NCPUBITS vendor/golang.org/x/sys/unix/affinity_linux.go:16:25: undefined: cpuMask vendor/golang.org/x/sys/unix/affinity_linux.go:49:27: undefined: cpuMask 解决方法： cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/unix/* output/build/docker-engine-19.03.8/vendor/golang.org/x/sys/unix/ 11、提示错误cannot find package "golang.org/x/sys/internal/unsafeheader" 解决方法：cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/internal/ output/build/docker-engine-19.03.8/vendor/golang.org/x/sys/ -r   12、编译docker-engine，提示错误undefined: cmsghdr     解决方法：cp output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/zsys_linux_arm.go output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/zsys_linux_csky.go 13、编译提示 vendor/golang.org/x/net/internal/socket/msghdr_linux.go:11:9: vs[i].set undefined (type iovec has no field or method set) vendor/golang.org/x/net/internal/socket/msghdr_linux.go:13:4: h.setIov undefined (type *msghdr has no field or method setIov) vendor/golang.org/x/net/internal/socket/msghdr_linux.go:15:5: h.setControl undefined (type *msghdr has no field or method setControl) vendor/golang.org/x/net/internal/socket/socket.go:143:4: h.set undefined (type *cmsghdr has no field or method set) vendor/golang.org/x/net/internal/socket/socket.go:166:4: h.set undefined (type *cmsghdr has no field or method set) 解决方法：添加csky支持 gedit output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/cmsghdr_linux_32bit.go  :// +build arm mips mipsle 386 csky gedit output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/msghdr_linux_32bit.go  :// +build arm mips mipsle 386 csky gedit output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/iovec_32bit.go     :// +build arm mips mipsle 386 csky 14、提示错误：undefined: sysRECVMMSG 解决方法：cp output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/sys_linux_arm.go output/build/docker-engine-19.03.8/vendor/golang.org/x/net/internal/socket/sys_linux_csky.go 注意这个文件不同arch的配置数值不一样，和zsysnum_linux_csky.go的定义有关，因此复制完后还需要修改sys_linux_csky.go文件，  sysRECVMMSG = 243  sysSENDMMSG = 269   15、提示错误undefined: icmpFilter， undefined: icmpv6Filter 解决方法： cp output/build/docker-engine-19.03.8/vendor/golang.org/x/net/ipv4/zsys_linux_arm.go output/build/docker-engine-19.03.8/vendor/golang.org/x/net/ipv4/zsys_linux_csky.go cp output/build/docker-engine-19.03.8/vendor/golang.org/x/net/ipv6/zsys_linux_arm.go output/build/docker-engine-19.03.8/vendor/golang.org/x/net/ipv6/zsys_linux_csky.go 16、提示 undefined: quota.NewControl 问题原因：因为禁止了CGO，导致projectquota.go文件中依赖了import "C"之后的代码没有编译，其中就有NewControl的实现。 解决方法： docker-engine.mk配置文件中添加exclude_disk_quota，关闭此项功能。 17、提示错误：cannot use ifrDataByte(hw[i]) (value of type int8) as type uint8 in assignment 解决方法：找到代码添加强制转换： gedit output/build/docker-engine-19.03.8/vendor/github.com/docker/libnetwork/drivers/bridge/netlink_deprecated_linux.go uint8(ifrDataByte(hw[i])) 18、执行docker run报错： docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2020-02-07T16:00:32Z\\\\\\\" level=fatal msg=\\\\\\\"failed to create a netlink handle: failed to set into network namespace 31 while creating netlink socket: bad address\\\\\\\"\\\\n\\\"\"": unknown. 分析：netns_linux.go文件中的func Setns(ns NsHandle, nstype int) (err error)做系统调用没有成功调用到zsyscall_linux.go中的代码。 解决方法：netns_linux.go中的var SYS_SETNS = map[string]uintptr 增加 "csky":     268,     编译runc 18、提示：lookup.go:15:41: undefined: current       解决方法：run.mk中删除CGO 19、提示：vendor/golang.org/x/sys/unix/affinity_linux.go:13:20: undefined: _CPU_SETSIZE  解决：cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/unix/* output/build/runc-1.0.0-rc10/vendor/golang.org/x/sys/unix/ 20、提示：cannot find package "golang.org/x/sys/internal/unsafeheader" in any of cp output/host/lib/go/src/cmd/vendor/golang.org/x/sys/internal/ output/build/runc-1.0.0-rc10/vendor/golang.org/x/sys/ -r 21、提示： libcontainer/init_linux.go:315:19: undefined: system.Setgid libcontainer/init_linux.go:318:19: undefined: system.Setuid 解决：编辑的gedit output/build/runc-1.0.0-rc10/libcontainer/system/syscall_linux_32.go，增加// +build 386 arm csky 22、提示： libcontainer/system/syscall_linux_32.go:12:35: undefined: unix.SYS_SETUID32 libcontainer/system/syscall_linux_32.go:21:35: undefined: unix.SYS_SETGID32 解决：gedit output/build/runc-1.0.0-rc10/vendor/golang.org/x/sys/unix/zsysnum_linux_csky.go，增加定义  SYS_SETUID32 = 447  SYS_SETGID32 = 448 23、tags中打开cgo标志，出现system.GetClockTicks为定义情况，go中没有实现，cgo中有实现，但是因为cgo_enable=0,所以其中import “C“的实现没有参与编译。 24、使能CGO编译docker-engine提示错误：/opt/csky-toolchain/bin/../lib/gcc/csky-linux-gnuabiv2/6.3.0/../../../../csky-linux-gnuabiv2/bin/ld: cannot find -lsystemd没有解决，临时在docker-engine.mk中注释了systemd等。     部署镜像 1、使用busybox文件系统打包成rootfs.tar 2、编写dockerfile： FROM scratch MAINTAINER cql ADD rootfs.tar / 3、板卡上cd到包含rootfs.tar和dockerfile的工作目录，制作docker镜像：docker build -t busyboxbase:latest . 4、查看docker镜像：docker image ls 4、运行docker镜像：docker run --rm -it busyboxbase sh     运行调试 1、docker ps 测试，出现错误提示 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? docker启动失败。 查看启动日志或者使用dockerd --debug，提示错误：failed to start daemon: error initializing graphdriver: invalid argument 原因是没有使能graphdriver。menuconfig中打开一个即可。   3、原因是没有部署socker镜像，下面通过dockers build建立镜像并部署到系统。 6、查看启动日志，dockerd运行一直报错： failed to set to initial namespace, net:[4026531877], initns fd 14: bad address 代码分析： InitOSContext—>SetNamespace（vendor/github.com/docker/libnetwork/ns/init_linux.go）—>Setns(vendor/github.com/vishvananda/netns/netns_linux.go) —>syscall.Syscall(SYS_SETNS)—>Syscall（vendor/golang.org/x/sys/unix/gccgo.go）—>realSyscall(vendor/golang.org/x/sys/unix/gccgo_c.c) 4、执行docker run报错： docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2020-02-07T16:00:32Z\\\\\\\" level=fatal msg=\\\\\\\"failed to create a netlink handle: failed to set into network namespace 31 while creating netlink socket: bad address\\\\\\\"\\\\n\\\"\"": unknown. 分析：netns_linux.go文件中的func Setns(ns NsHandle, nstype int) (err error)做系统调用没有成功调用到zsyscall_linux.go中的代码。 解决方法：netns_linux.go中的var SYS_SETNS = map[string]uintptr 增加 "csky":     268, 5、执行docker run报错： docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown. 分析： container_linux.go打印确认，但是添加打印有点奇怪，通过编写测试程序stat系统调用正常，确认docker文件系统镜像中是包含了/bin/sh目录，怀疑是文件系统的路径不对。使用docker run --rm -it busyboxbase /rootfs/bin/sh测试可以成功进入容器。 解决方法：重新支持容器文件系统，将bin等目录置于压缩包rootfs.tar的一级目录，剔除rootfs目录。