首页 > 系统相关 >Windows平台如何实现RTSP拉流添加动态水印|视频处理后转推RTMP或轻量级RTSP服务

Windows平台如何实现RTSP拉流添加动态水印|视频处理后转推RTMP或轻量级RTSP服务

时间:2023-12-27 17:35:15浏览次数:40  
标签:publisher Windows RTSP wrapper _. video btn NT 轻量级


 技术背景

我们在做Windows平台流数据转发的时候,除了常规的RTSP转RTMP推送外,还有个场景就是,好多开发者希望拉取的RTSP流,做二次视频分析,或者加动态水印等,处理后的数据,再二次编码推送到RTMP服务或轻量级RTSP服务。

技术实现

本文就以Windows平台拉取RTSP流,回调yuv数据到上层,处理后的数据,二次投递到RTMP服务和轻量级RTSP服务,然后叠加动态水印,并实现处理后的数据实时录像功能,废话不多说,先上图:

Windows平台如何实现RTSP拉流添加动态水印|视频处理后转推RTMP或轻量级RTSP服务_RTSP添加水印

上图拉取了RTSP流,然后左侧窗体显示,添加动态水印后,再在右侧预览,并把数据重新投递到推送端,考虑到编码性能,我们可选硬编码。

先说RTSP拉流,其他接口不表,这里主要是设置下video frame callback:

video_frame_call_back_ = new SP_SDKVideoFrameCallBack(SetVideoFrameCallBack);
NTSmartPlayerSDK.NT_SP_SetVideoFrameCallBack(player_handle_, (Int32)NT.NTSmartPlayerDefine.NT_SP_E_VIDEO_FRAME_FORMAT.NT_SP_E_VIDEO_FRAME_FROMAT_I420, IntPtr.Zero, video_frame_call_back_);

回调上来的video数据,投递到推送端,当然如果需要二次处理的话,处理后再丢给推送端:

public void SetVideoFrameCallBack(IntPtr handle, IntPtr userData, UInt32 status, IntPtr frame)
        {
            if (frame == IntPtr.Zero)
                return;

            NT_SP_VideoFrame video_frame = (NT_SP_VideoFrame)Marshal.PtrToStructure(frame, typeof(NT_SP_VideoFrame));

            if (publisher_wrapper_ != null) {
                         publisher_wrapper_.post_i420_layer_image(publisher_wrapper_.get_external_video_layer_index(), video_frame.plane0_, video_frame.stride0_, video_frame.plane1_, video_frame.stride1_,
                            video_frame.plane2_, video_frame.stride2_,
                            video_frame.width_, video_frame.height_);
            }
        }

audio的也是:

audio_pcm_frame_call_back_ = new SP_SDKAudioPCMFrameCallBack(SetAudioPCMFrameCallBack);
NTSmartPlayerSDK.NT_SP_SetAudioPCMFrameCallBack(player_handle_, IntPtr.Zero, audio_pcm_frame_call_back_);

回调上来的audio数据:

public void SetAudioPCMFrameCallBack(IntPtr handle, IntPtr user_data,
             UInt32 status, IntPtr data, UInt32 size,
             Int32 sample_rate, Int32 channel, Int32 per_channel_sample_number)
        {
            if (data == IntPtr.Zero || size == 0)
                return;

            if (publisher_wrapper_ != null)
                publisher_wrapper_.post_audio_pcm_data(data, size, 0, sample_rate, channel, per_channel_sample_number);
        }

由于处理后的数据,需要重新编码,我们播放端会把原始视频宽高回调上来:

video_size_call_back_ = new SP_SDKVideoSizeCallBack(SP_SDKVideoSizeHandle);
NTSmartPlayerSDK.NT_SP_SetVideoSizeCallBack(player_handle_, IntPtr.Zero, video_size_call_back_);

拿到视频宽高后,我们可以把宽高投递到推送端,便于推送端推送相同分辨率出去,当然,二次编码,也可以设置其他期望的分辨率:

private void PlaybackWindowResized(Int32 width, Int32 height)
        {
            width_ = width;
            height_ = height;

            if (publisher_wrapper_ != null)
                publisher_wrapper_.SetResolution(width, height);
        }

设置文字水印字体、字号:

private void btn_set_font_Click(object sender, EventArgs e)
        {
            FontDialog font_dlg = new FontDialog();
            DialogResult result = font_dlg.ShowDialog();

            if (result == DialogResult.OK)
            {
                // 获取用户所选字体
                Font selectedFont = font_dlg.Font;
                btn_set_font.Text = "" + selectedFont.Name + ", " + selectedFont.Size + "pt";

                selected_osd_font_ = new Font(selectedFont.Name, selectedFont.Size, FontStyle.Regular, GraphicsUnit.Point);
            }
        }

如果需要添加文字水印:

private async void btn_text_osd_Click(object sender, EventArgs e) {
            Bitmap bitmap = null;

            try
            {
                string format = "yyyy-MM-dd HH:mm:ss.fff";

                StringBuilder sb = new StringBuilder();
                sb.Append("施工单位:上海视沃信息科技有限公司(daniusdk.com)");
                sb.Append("\r\n");
                sb.Append("施工时间:");
                sb.Append(DateTime.Now.DayOfWeek.ToString());
                sb.Append(" ");
                sb.Append(DateTime.Now.ToString(format));
                sb.Append("\r\n");
                sb.Append("当前位置:上海市");
                string str = sb.ToString();
                bitmap = GenerateBitmap(str);
            }
            catch (Exception )
            {
                return;
            }

            if (null == bitmap)
                return;

            int x = 0;
            int y = 200;
            UpdateLayerRegion(publisher_wrapper_.get_text_layer_index(), x, y, bitmap);
            publisher_wrapper_.enable_layer(publisher_wrapper_.get_text_layer_index(), true);

            await Task.Delay(30);
            publisher_wrapper_.post_argb8888_layer_image(publisher_wrapper_.get_text_layer_index(), bitmap);
        }

开始推送RTMP:

private void btn_publish_rtmp_Click(object sender, EventArgs e)
        {
            if (!OpenPublisherHandle())
                return;

            SetCommonOptionToPublisherSDK();

            String url = "rtmp://192.168.0.108:1935/hls/stream1";
            //String url = "rtmp://192.168.2.154:1935/live/stream1";

            if (url.Length < 8)
            {
                publisher_wrapper_.try_close_handle();
                MessageBox.Show("请输入推送地址");
                return;
            }

            if (!publisher_wrapper_.StartPublisher(url))
            {
                MessageBox.Show("调用StartPublisher失败..");
                return;
            }

            btn_publish_rtmp.Enabled = false;
            btn_stop_publish_rtmp.Enabled = true;
        }

publisher_wrapper接口封装:

public bool StartPublisher(String url)
        {
            if (is_empty_handle() || is_rtmp_publishing())
                return false;

            if (!String.IsNullOrEmpty(url))
                NTSmartPublisherSDK.NT_PB_SetURL(handle_, url, IntPtr.Zero);

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPublisher(handle_, IntPtr.Zero))
            {
                try_close_handle();
                return false;
            }

            shared_lock_.EnterWriteLock();
            try
            {
                handle_reference_count_++;
                is_rtmp_publishing_ = true;
            }
            finally
            {
                shared_lock_.ExitWriteLock();
            }

            return true;
        }

        public void StopPublisher()
        {
            if (is_empty_handle() || !is_rtmp_publishing())
                return;

            shared_lock_.EnterWriteLock();
            try
            {
                is_rtmp_publishing_ = false;
                handle_reference_count_--;
            }
            finally
            {
                shared_lock_.ExitWriteLock();
            }

            NTSmartPublisherSDK.NT_PB_StopPublisher(handle_);
            try_close_handle();
        }

对应的SetCommonOptionToPublisherSDK()实现:

private void SetCommonOptionToPublisherSDK()
        {
            if (publisher_wrapper_.is_empty_handle())
                return;

            if (publisher_wrapper_.handle_reference_count() > 0)
                return;

            publisher_wrapper_.config_layers(true);

            publisher_wrapper_.SetFrameRate((uint)video_fps_);

            int cur_video_codec_id = (int)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264;

            bool is_h264_encoder = true;

            bool is_hw_encoder = false;

            if (btn_check_video_hardware_encoder_.Checked)
            {
                is_hw_encoder = true;
            }

            Int32 cur_sel_encoder_id = 0;
            Int32 cur_sel_gpu = 0;

            if (is_hw_encoder)
            {
                int cur_sel_hw = combobox_video_encoders_.SelectedIndex;
                if (cur_sel_hw >= 0)
                {
                    cur_sel_encoder_id = Convert.ToInt32(combobox_video_encoders_.SelectedValue);
                    cur_sel_gpu = -1;

                    int cur_sel_hw_dev = combobox_video_hardware_encoder_devices_.SelectedIndex;
                    if (cur_sel_hw_dev >= 0)
                    {
                        cur_sel_gpu = Convert.ToInt32(combobox_video_hardware_encoder_devices_.SelectedValue);
                    }
                }
                else
                {
                    is_hw_encoder = false;
                }
            }

            if (!is_hw_encoder)
            {
                if ((int)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264 == cur_video_codec_id)
                {
                    cur_sel_encoder_id = btn_check_openh264_encoder_.Checked ? 1 : 0;
                }
            }

            publisher_wrapper_.SetVideoEncoder((int)(is_hw_encoder ? 1 : 0), (int)cur_sel_encoder_id, (uint)cur_video_codec_id, (int)cur_sel_gpu);

            publisher_wrapper_.SetVideoQualityV2(publisher_wrapper_.CalVideoQuality(width_, height_, is_h264_encoder));

            publisher_wrapper_.SetVideoBitRate(publisher_wrapper_.CalBitRate(video_fps_, width_, height_));

            publisher_wrapper_.SetVideoMaxBitRate(publisher_wrapper_.CalMaxKBitRate(video_fps_, width_, height_, false));

            publisher_wrapper_.SetVideoKeyFrameInterval(key_frame_interval_);

            if (is_h264_encoder)
            {
                publisher_wrapper_.SetVideoEncoderProfile(3);
            }

            publisher_wrapper_.SetVideoEncoderSpeed(publisher_wrapper_.CalVideoEncoderSpeed(width_, height_, is_h264_encoder));

            publisher_wrapper_.SetPublisherAudioCodecType(1);   //1: AAC 2: Speex
        }

图层配置实现:

public bool config_layers(bool is_add_rgbx_zero_layer)
        {
            if (video_option_ != (uint)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_LAYER)
                return false;

            if (is_empty_handle())
                return false;

            int w = video_width_;
            int h = video_height_;

            if ((w & 0x1) != 0)
                --w;

            if ((h & 0x1) != 0)
                --h;

            if (w < 2 || h < 2)
                return false;

            NTSmartPublisherSDK.NT_PB_ClearLayersConfig(handle_, 0, 0, IntPtr.Zero);

            int type, index = 0;
            if (is_add_rgbx_zero_layer)
            {
                NT_PB_RGBARectangleLayerConfig rgba_layer = new NT_PB_RGBARectangleLayerConfig();
                type = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE;
                fill_layer_base(rgba_layer, out rgba_layer.base_, type, index, true, 0, 0, w, h);
                rgba_layer.red_ = 0;
                rgba_layer.green_ = 0;
                rgba_layer.blue_ = 0;
                rgba_layer.alpha_ = 255;
                if (add_layer_config(rgba_layer, type))
                    index++;
            }

            NT_PB_ExternalVideoFrameLayerConfig external_video_layer = new NT_PB_ExternalVideoFrameLayerConfig();
            type = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_EXTERNAL_VIDEO_FRAME;
            fill_layer_base(external_video_layer, out external_video_layer.base_, type, index, true, 0, 0, w, h);
            if (add_layer_config(external_video_layer, type))
                external_video_layer_index_ = index++;


            //叠加的文本层
            NT_PB_ExternalVideoFrameLayerConfig text_layer = new NT_PB_ExternalVideoFrameLayerConfig();
            type = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_EXTERNAL_VIDEO_FRAME;
            fill_layer_base(text_layer, out text_layer.base_, type, index, false, 0, 0, 64, 64);
            if (add_layer_config(text_layer, type))
                text_layer_index_ = index++;

            return index > 0;
        }

推送端二次录像:

private void btn_start_recorder_Click(object sender, EventArgs e)
        {
            if (!OpenPublisherHandle())
                return;

           SetCommonOptionToPublisherSDK();

            if (!publisher_wrapper_.StartRecorder())
            {
                MessageBox.Show("调用StartRecorder失败..");
                return;
            }

            btn_start_recorder.Enabled = false;
            btn_stop_recorder.Enabled = true;
        }

        private void btn_pause_recorder_Click(object sender, EventArgs e)
        {
            String btn_pause_rec_text = btn_pause_recorder.Text;

            if ("暂停录像" == btn_pause_rec_text)
            {
                UInt32 ret = publisher_wrapper_.PauseRecorder(true);

                if ((UInt32)NT.NTSmartPublisherDefine.NT_PB_E_ERROR_CODE.NT_ERC_PB_NEED_RETRY == ret)
                {
                    MessageBox.Show("暂停录像失败, 请重新尝试!");
                    return;
                }
                else if (NTBaseCodeDefine.NT_ERC_OK == ret)
                {
                    btn_pause_recorder.Text = "恢复录像";
                }
            }
            else
            {
                UInt32 ret = publisher_wrapper_.PauseRecorder(false);
                if ((UInt32)NT.NTSmartPublisherDefine.NT_PB_E_ERROR_CODE.NT_ERC_PB_NEED_RETRY == ret)
                {
                    MessageBox.Show("恢复录像失败, 请重新尝试!");
                    return;
                }
                else if (NTBaseCodeDefine.NT_ERC_OK == ret)
                {
                    btn_pause_recorder.Text = "暂停录像";
                }
            }
        }

        private void btn_stop_recorder_Click(object sender, EventArgs e)
        {
            if (publisher_wrapper_.is_recording()) {
                publisher_wrapper_.StopRecorder();

                btn_start_recorder.Enabled = true;
                btn_stop_recorder.Enabled = false;
            }
        }

启动轻量级RTSP服务:

private void btn_rtsp_service_Click(object sender, EventArgs e)
        {
            if(publisher_wrapper_.IsRTSPSerivceRunning())
            {
                publisher_wrapper_.StopRtspService();
                btn_rtsp_service.Text = "启动RTSP服务";
                btn_rtsp_stream.Enabled = false;
            }
            else
            {
                if(publisher_wrapper_.StartRtspService())
                {
                    btn_rtsp_service.Text = "停止RTSP服务";
                    btn_rtsp_stream.Enabled = true;
                }
            }
        }

发布RTSP流:

private void btn_rtsp_stream_Click(object sender, EventArgs e)
        {
            if (publisher_wrapper_.is_rtsp_publishing())
            {
                publisher_wrapper_.StopRtspStream();
                btn_rtsp_stream.Text = "发布RTSP流";
                btn_get_rtsp_session_numbers.Enabled = false;
                btn_rtsp_service.Enabled = true;
            }
            else
            {
                if (!OpenPublisherHandle())
                    return;

                SetCommonOptionToPublisherSDK();

                if (!publisher_wrapper_.StartRtspStream())
                {
                    MessageBox.Show("调用StartRtspStream失败..");
                    return;
                }

                btn_rtsp_stream.Text = "停止RTSP流";
                btn_get_rtsp_session_numbers.Enabled = true;
                btn_rtsp_service.Enabled = false;
            }
        }

获取RTSP session会话数:

private void btn_get_rtsp_session_numbers_Click(object sender, EventArgs e)
        {
            if (publisher_wrapper_.is_rtsp_publishing())
            {
               int session_numbers = publisher_wrapper_.GetRtspSessionNumbers();

               MessageBox.Show(session_numbers.ToString(), "当前RTSP连接会话数");
            }
        }

叠加后的数据,本地预览:

private void btn_preview_Click(object sender, EventArgs e)
        {
            if (publisher_wrapper_.is_previewing())
            {
                publisher_wrapper_.StopPreview();

                btn_preview.Text = "开始预览";
            }
            else
            {
               if (!OpenPublisherHandle())
                   return;

                SetCommonOptionToPublisherSDK();
                       
                if (!publisher_wrapper_.StartPreview())
                {
                    MessageBox.Show("调用StartPreview失败..");
                    return;
                }

                btn_preview.Text = "停止预览";
            }
        }

对应的预览封装实现:

public bool StartPreview()
        {
            if (is_empty_handle() || is_previewing())
                return false;

            video_preview_image_callback_ = new NT_PB_SDKVideoPreviewImageCallBack(SDKVideoPreviewImageCallBack);
            NTSmartPublisherSDK.NT_PB_SetVideoPreviewImageCallBack(handle_, (int)NTSmartPublisherDefine.NT_PB_E_IMAGE_FORMAT.NT_PB_E_IMAGE_FORMAT_RGB32, IntPtr.Zero, video_preview_image_callback_);

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPreview(handle_, 0x800000, IntPtr.Zero))
            {
                try_close_handle();
                return false;
            }

            shared_lock_.EnterWriteLock();
            try
            {
                handle_reference_count_++;
                is_previewing_ = true;
            }
            finally
            {
                shared_lock_.ExitWriteLock();
            }

            return true;
        }

        public void StopPreview()
        {
            if (is_empty_handle() || !is_previewing())
                return;

            shared_lock_.EnterWriteLock();
            try
            {
                is_previewing_ = false;
                handle_reference_count_--;
            }
            finally
            {
                shared_lock_.ExitWriteLock();
            }

            NTSmartPublisherSDK.NT_PB_StopPreview(handle_);
            try_close_handle();

            if (render_wnd_ != null)
                render_wnd_.Invalidate();
        }

预览数据回调:

//预览数据回调
        public void SDKVideoPreviewImageCallBack(IntPtr handle, IntPtr user_data, IntPtr image)
        {
            NT_PB_Image pb_image = (NT_PB_Image)Marshal.PtrToStructure(image, typeof(NT_PB_Image));

            NT_VideoFrame pVideoFrame = new NT_VideoFrame();

            pVideoFrame.width_ = pb_image.width_;
            pVideoFrame.height_ = pb_image.height_;

            pVideoFrame.stride_ = pb_image.stride_[0];

            Int32 argb_size = pb_image.stride_[0] * pb_image.height_;

            pVideoFrame.plane_ = Marshal.AllocHGlobal(argb_size);

            CopyMemory(pVideoFrame.plane_, pb_image.plane_[0], (UInt32)argb_size);

            if (sync_invoke_ != null)
            {
                System.ComponentModel.ISynchronizeInvoke sync_invoke_target = sync_invoke_.Target as System.ComponentModel.ISynchronizeInvoke;

                if (sync_invoke_target != null)
                {

                    if (sync_invoke_target.InvokeRequired)
                    {
                        sync_invoke_target.BeginInvoke(set_video_preview_image_callback_, new object[] { pVideoFrame });
                    }
                    else
                    {
                        set_video_preview_image_callback_(pVideoFrame);
                    }
                }
            }
        }

总结

以上就是RTSP流二次编辑(如增加动态水印)或视频分析(视觉算法处理)后,再录像、转推至RTMP或轻量级RTSP服务流程,经过二次处理后的流数据,配合我们的SmartPlayer,依然可以整体毫秒级的延迟体验,感兴趣的开发者,可以和我探讨。


标签:publisher,Windows,RTSP,wrapper,_.,video,btn,NT,轻量级
From: https://blog.51cto.com/daniusdk/9002021

相关文章

  • windows 网络适配器
     添加虚拟网卡:            https://blog.51cto.com/elasticsearch/5488949如何区分虚拟网卡和物理网卡:           参考:https://blog.csdn.net/EDDJH_31/article/details/82694205 ......
  • Ubuntu访问Windows共享
    要在Ubuntu上访问Windows共享,可以通过以下步骤进行设置:确保Windows共享设置正确:共享文件夹:在Windows上选择要共享的文件夹,并确保它已共享。右键点击文件夹,选择“属性”,然后进入“共享”选项卡,设置共享选项和权限。网络发现和共享:确保Windows上的网络发现和文件共享已打开。......
  • Python windows下subprocess模块 cwd 参数不支持相对路径
    前言全局说明Pythonwindows下subprocess模块cwd参数不支持相对路径一、问题程序要执行命令,用到了subprocess模块,并指定了cwd运行路径,在MAC系统下运行正常,在Windows下运行报错。经过查询,是系统差异导致,所以为了方便,在windows下获取当前路径后拼接再生成绝对路径......
  • 在windows下安装mysql 8.1
    1、下载并解压官网下载mysql8,https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.11-winx64.zip解压到D:\mysql,以下称为根目录2、编写配置文件在根目录下新建my.ini文件,配置以下内容[mysqld]#设置3306端口port=3306#设置mysql的安装目录,一定要与上面的安装路......
  • Ping不通问题解决 windows 查看对端MAC地址 ARP -a
    Ping不通问题解决   Linux查看ARP信息指南(linux查看arp) ARP(地址解析协议)是TCP/IP协议提供的网络层协议,通过ARP可以查看网络层面上当前可连接的本地网络内每个主机的MAC地址。 ##查看系统的ARP信息 Linux系统中查看ARP信息的方法有很多,下面简单介绍几种常见的查......
  • go-carbon v2.3.0 圣诞特别版发布,轻量级、语义化、对开发者友好的 Golang 时间处理库
    go-carbonv2.3.0圣诞节特别版发布,这应该是2023年的最后一个版本,祝大家圣诞节快乐!carbon是一个轻量级、语义化、对开发者友好的golang时间处理库,支持链式调用。目前已被awesome-go收录,如果您觉得不错,请给个star吧github.com/golang-module/carbongitee.com/golang-m......
  • Windows 安装 Rust 并设置镜像加速
    目录下载rustup-init.exe(Rust安装工具)使用镜像加速rustup安装安装Rust安装标准库源码使用镜像加速cargo包下载安装结果确认更新、卸载和文档查看参考文档下载rustup-init.exe(Rust安装工具)下载安装程序https://www.rust-lang.org/tools/install,我选的是64位:使用镜像加速rustu......
  • Windows Server 2019-Powershell之客户端加域
    将本地计算机添加到域或工作组,可通过Add-Computer命令操作,具体信息如下:语法:Add-Computer[-DomainName][-ComputerName<String[]>][-Confirm]-Credential[-Force][-LocalCredential][-NewName][-OUPath][-Options{AccountCreate|Win9XUpgrade|UnsecuredJoi......
  • 超轻量级MP4封装方法介绍
    liwen012023.12.17前言MP4是目前非常常用的一种视频封装格式,关于MP4的介绍资料也非常多。我们常用的封装库或工具有:ffmpeg,libmp4v2,GPAC,MP4.js,它们的优点是功能基本上都是比较全面,缺点就是它们占用的资源相对来说也是非常多的。在嵌入式系统中,不管是RAM还是FLASH空间,一般都是非......
  • Windows11 win11提示这台电脑不符合安装此版本的Windows所需的最低系统要求怎么解决?
    Windows11win11提示这台电脑不符合安装此版本的Windows所需的最低系统要求怎么解决?  现在很多用户都会选择用U盘来安装系统,最新有用户在使用U盘安装Win11系统的时候,结果安装到第一步就提示这台电脑无法运行Windows11,这台电脑不符合安装此版本的Windows所需的最低系统要求。......