一、Quad Sensor MIPI CSI Video Capture and HDMI Display案例介绍
总体框架如下图所示(不包含HDMI)：
图中ISP部分：
四个AR0231 sensor进来经过AMZ9286将思路图像汇总成一路CSI-2接到pl端的mipi接口，通过一个“AXI Stream Switch”将四个虚拟通道分别送到四路ISP处理，最后送到buffer中。
四个isp分别为：
Sensor Demasaic：去马赛克，也就是通过插值算法把RAW图像转换为RGB图像
GamamaLUT：伽马矫正
Video Processing Subsystem：第一个做rgb颜色校正，第二个将rgb转yuv。
二、vivado和petalinux工程搭建
构建vivado工程和petalinux工程参考：https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010368517/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Run+and+Build+Flow 的3 Build Flow章节
2.1下载官方案例
下载地址：https://www.xilinx.com/cgi-bin/docs/ctdoc?cid=bigfile;d=rdf0428-zcu106-vcu-trd-2020-2.zip
2.2、搭建vivado工程
1、将下载的案例rdf0428-zcu106-vcu-trd-2020.2复制到虚拟机
2、设置环境变量
export TRD_HOME=</path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020-2
3、source vivado命令：
source /opt/xilinx/vitis_2020/Vivado/2020.2/settings64.sh
4、进入rdf0428-zcu106-vcu-trd-2020.2/pl目录(不能用secureCRT，要用虚拟机中的终端，否则无法自动打开vivado)
cd </path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020.2/pl
5、输入命令导入tcl文件：
vivado -source ./designs/zcu106_Quad_Sensor/project.tcl
注：如果遇到project.tcl无法执行，则执行chmod 777 project.tcl
vivado工程构建完成后，会生成如下所示的工程：
新建的工程默认保存在</path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020-2/pl/build 目录中
2.3 创建petalinux 工程
1、到处petalinx环境变量：source /opt/xilinx/petalinux/settings.sh
2、进入到rdf0428-zcu106-vcu-trd-2020.1/apu/vcu_petalinux_bsp目录
3、用rdf0428-zcu106-vcu-trd-2020.1/apu/vcu_petalinux_bsp/xilinx-vcu-zcu106-v2020.1-final.bsp建立petalinux工程:
petalinux-create -t project -s ./
4、导入rdf0428-zcu106-vcu-trd-2020.1\pl\prebuild\zcu106_Quad_Sensor\zcu106_Quad_Sensor_wrapper.xsa 文件
petalinux-config --get-hw-description ../../pl/prebuild/zcu106_Quad_Sensor/
3、将vcu_quad_sensor.dtsi软件\\软连接成system-user.dtsi
ln -s xilinx-vcu-zcu106-v2020.1-final/project-spec/meta-user/recipes-bsp/device-tree/files/vcu_quad_sensor.dtsi system-user.dtsi
4、petalinux-bulid
5、用qemu虚拟机运行：petalinux-boot --qemu --prebuilt 3
退出qemu：Ctrl+A，然后放掉再按X
编译过程中如果出现qemu无法获取失败，参考https://forums.xilinx.com/t5/Embedded-Linux/qemu-fails-in-petalinux-buld-due-to-Fetcher-failure/m-p/1168900
三、设备树解析：
Quad Sensor的设备树是petalinux_prj/project-spec/meta-user/recipes-bsp/device-tree/files/vcu_quad_sensor.dtsi。vcu_quad_sensor.dtsi主要是配置pl端的各个模块，包过mipi，buffer、isp模块，hdmi等，vcu_quad_sensor.dtsi中还包含了一个配置AR0231 sensor和AMZ9286的设备树
“avnet-ar0231-multicam-fmc.dtsi”，如下所示：
设备树中各个模块的连接就是按上面总框架图的连接。
四、gstreamer命令
的第四小节4 Appendix B - CSI-2 Rx/HDMI-Tx Link-up and GStreamer Commands的gstreamer命令。
1、配置pl：sh /media/card/quad_sensor_media_graph_setting.sh
2、用gst-launch-1.0命令显示raw数据
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>" v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>" v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -v
这条命令显示四个摄像头的图像，四个摄像头的取流节点分别为/dev/video0、/dev/video1、/dev/video2、/dev/video3。可以把这条命名分为四行，每行对应一个摄像头和一个显示画面，如下所示：
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" \
v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>" \
v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>" \
v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -v
fpsdisplaysink是个渲染插件可以理解为送显，a0270000.v_mix是个video mixer，这个是xilinx的一个视频混合器ip，这条视频管道为v4l2src -->queue-->fpsdisplaysink
3、capture → encode → decode → display
下面是视频捕获-->vcu编码-->vcu解码-->hdmi显示的管道
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>" v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>" v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -v
同样把这个长命令分为4行，每行代表一个显示管道
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>"
v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>"
v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>"
v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -
omxh265enc 和omxh265dec 分别是xilinx的vcu编码和解码插件，详情见<<ug1449 Multimedia User Guide>>的GStreamer Multimedia Framework章节。
这条视频管道为：v4l2src-->omxh265enc-->queue-->omxh265dec-->fpsdisplaysink