0%

ros图像处理相关

1. image_transport

ros规定了多种基本的数据结构,用于node之间的传输。图像也是一种常用的数据,image_transport package就是用来处理图像数据传输的,它本身是个框架,只提供最基本的原始图像数据(raw),如果需要降低传输时的带宽,还需要压缩格式,由框架下集成的各种插件来完成。

image_transport 会发布 sensor_msgs/Image 格式的数据,其他压缩格式由插件提供。


ROS中原始图像格式”sensor_msgs/Image”如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
                        # This message contains an uncompressed image
# (0, 0) is at top-left corner of image
#
Header header # std_msgs/Header

uint32 height # image height, that is, number of rows
uint32 width # image width, that is, number of columns

# The legal values for encoding are in file src/image_encodings.cpp
# If you want to standardize a new string format, join
# ros-users@lists.sourceforge.net and send an email proposing a new encoding.
string encoding # Encoding of pixels -- channel meaning, ordering, size
# taken from the list of strings in include/sensor_msgs/image_encodings.h

uint8 is_bigendian # is this data bigendian?
uint32 step # Full row length in bytes
uint8[] data # actual matrix data, size is (step * rows)

其中 header 是标准结构:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
                        # Standard metadata for higher-level stamped data types.
# This is generally used to communicate timestamped data
# in a particular coordinate frame.
#
# sequence ID: consecutively increasing ID
uint32 seq

#Two-integer timestamp that is expressed as:
# * stamp.sec: seconds (stamp_secs) since epoch (in Python the variable is called 'secs')
# * stamp.nsec: nanoseconds since stamp_secs (in Python the variable is called 'nsecs')
# time-handling sugar is provided by the client library
time stamp

#Frame this data is associated with
string frame_id

一个具体例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw --noarr
header:
seq: 293 # 当前帧序号
stamp: # 获取图像的时间戳
secs: 1636868180
nsecs: 53476744
frame_id: "camera" # frame_id
height: 480 # 行数
width: 640 # 列数
encoding: "bgr8" # 像素点的压缩格式,通道数3,顺序是BGR,深度是24。按照这个可以知道data数组中值的意义
is_bigendian: 0 # 是否大端排序
step: 1920 # 步长,一行的字节数,宽度(640) x 像素点字节数(3) = 1920bytes。图像总共 480行,也就是 1920x480=921600字节,也是下边data数组长度
data: "<array type: uint8, length: 921600>"

使用上的例子
不使用 image_transport 的订阅发布:

1
2
3
4
5
6
7
8
9
10
11
// Do not communicate images this way!
#include <ros/ros.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
// ...
}

ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe("in_image_topic", 1, imageCallback);
ros::Publisher pub = nh.advertise<sensor_msgs::Image>("out_image_topic", 1);

使用 image_transport 的订阅发布:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Use the image_transport classes instead.
#include <ros/ros.h>
#include <image_transport/image_transport.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
// ...
}

ros::NodeHandle nh;
image_transport::ImageTransport it(nh); // 相当于将原始的nh封装了一次
image_transport::Subscriber sub = it.subscribe("in_image_base_topic", 1, imageCallback);
image_transport::Publisher pub = it.advertise("out_image_base_topic", 1);

提供的东西:

API

publisher

image_transport::Publisher
image_transport本身只在基础topic: <base topic> 上发布 sensor_msgs/Image 原始格式的图像,如果有其他插件,会自动在基础topic下的其他topic发布:<base topic>/<transport name>,比如基础topic是 “/cv_camera”,
image_transport 会发布 “/cv_camera/image_raw” 原始格式的topic,插件compressed_image_transport 会在 “/cv_camera/image_raw/compressed” 发布压缩格式的图像。

subscriber
image_transport::Subscriber
订阅基础topic后,会自动订阅其他相关topic。

parameter

image_transport 本身不发布参数,但插件会发布各种参数,比如帧率、压缩等级等,需要看插件本身。
发布者应该使用动态参数机制,让参数更容易被订阅者使用。
参数格式一般是: <base topic>/<transport name>/<parameter name>
比如: /cv_camera/image_raw/compressed/parameter_descriptions

node

1
$ rosrun image_transport republish [in_transport] in:=<in_base_topic> [out_transport] out:=<out_base_topic>

比如将 theora 格式的topic图像解压,重新发布到另一个topic,可以这样写:

1
$ rosrun image_transport republish theora in:=camera/image raw out:=camera/image_decompressed

如果一个node只发布原始的图像格式,可以用 image_transport 重新发布成多种格式的topic。

命令行工具

1
$ rosrun image_transport list_transports

可以查看当前所有的package,看看是否有可用的图像处理插件,以及插件状态。


下边的插件介绍
使用 image_transport 提供的一个node命令: list_transports,可以查看当前有的topic和对应的插件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
root@suntus:~# rosrun image_transport list_transports
Declared transports:
image_transport/compressed
image_transport/compressedDepth
image_transport/ffmpeg (*): Not available. Try 'catkin_make --pkg ffmpeg_image_transport'.
image_transport/raw
image_transport/theora

Details:
----------
"image_transport/compressed"
- Provided by package: compressed_image_transport
- Publisher:
This plugin publishes a CompressedImage using either JPEG or PNG compression.

- Subscriber:
This plugin decompresses a CompressedImage topic.

----------
"image_transport/compressedDepth"
- Provided by package: compressed_depth_image_transport
- Publisher:
This plugin publishes a compressed depth images using PNG compression.

- Subscriber:
This plugin decodes a compressed depth images.


----------
"image_transport/ffmpeg"
*** Plugins are not built. ***
- Provided by package: ffmpeg_image_transport
- Publisher:
This plugin encodes frame into ffmpeg compressed packets

- Subscriber:
This plugin decodes frame from ffmpeg compressed packets

----------
"image_transport/raw"
- Provided by package: image_transport
- Publisher:
This is the default publisher. It publishes the Image as-is on the base topic.

- Subscriber:
This is the default pass-through subscriber for topics of type sensor_msgs/Image.

----------
"image_transport/theora"
- Provided by package: theora_image_transport
- Publisher:
This plugin publishes a video packet stream encoded using Theora.

- Subscriber:
This plugin decodes a video packet stream encoded using Theora.

compressed_image_transport(“compressed”):
发布和订阅 JPEG 或 PNG 压缩格式的图像,支持运行时改变图片质量。

压缩格式:

1
2
3
4
5
6
                    # This message contains a compressed image
Header header # std_msgs/Header
string format # Specifies the format of the data
# Acceptable values:
# jpeg, png
uint8[] data # Compressed image buffer

一个例子:

1
2
3
4
5
6
7
8
9
10
root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw/compressed --noarr
header:
seq: 0
stamp:
secs: 1636880329
nsecs: 751462995
frame_id: "camera"
format: "bgr8; jpeg compressed bgr8"
data: "<array type: uint8, length: 43548>"
---

提供的东西:

发布插件

发布topic:

<base_topic>/compressed (sensor_msgs/CompressedImage)

参数

  • <base_topic>/compressed/format (string, default: jpeg) Compression format to use, “jpeg” or “png”.
  • <base_topic>/compressed/jpeg_quality (int, default: 80) JPEG quality percentile, in the range [1, 100]. Lower values trade image quality for space savings.
  • <base_topic>/compressed/png_level (int, default: 9) PNG compression level, in the range [1, 9]. Higher values trade computation time for space savings.

订阅插件

订阅topic:
<base_topic>/compressed (sensor_msgs/CompressedImage)


compressed_depth_image_transport
提供深度图像(depth image),主要用于3D领域,带距离的图像格式。


theora_image_transport
发布成视频流
theora视频编码格式的视频流数据
theora是个视频编码格式,类似H264这种
对应的声音编码技术是vorbis
对应的文件封装格式是ogg

发布topic
<base topic>/theora (theora_image_transport/Packet)

参数

  • <base topic>/theora/optimize_for (int, default: Quality (1))
    Controls whether to use constant bitrate (CBR) encoding, aiming for ~theora/target_bitrate; or variable bitrate (VBR) encoding, aiming for ~theora/quality. Values are Bitrate (0) and Quality (1).
  • <base topic>/theora/target_bitrate (int, default: 800000)
    Target bitrate. Used if optimize_for is set to Bitrate.
  • <base topic>/theora/quality (int, default: 31)
    Encoding quality level, in the range [0, 63]. Used if optimize_for is set to Quality.
  • <base topic>/theora/keyframe_frequency (int, default: 64)
    Maximum distance between key frames. If set to 1, every frame is a keyframe.

订阅topic:
<base topic>/theora (theora_image_transport/Packet)

提供有一个node: ogg_saver,可以保存成 .ogv 格式的封装文件,用其他播放器播放。


ffmpeg_image_transport
使用ffmpe将图像转换成h264或h265格式,并且支持nvidia GPU硬件加速。

https://github.com/daniilidis-group/ffmpeg_image_transport


cv_bridge
在ROS Image messages 和 OpenCV images 格式之间进行转换。

ros图像格式基础的是 sensor_msgs/Image, 还有 sensor_msgs/CompressedImage, opencv中用到的格式为 cv::Mat 矩阵,需要进行转换,才能放到opencv中使用。

opencv中支持的像素编码格式有:
8UC[1-4]
8SC[1-4]
16UC[1-4]
16SC[1-4]
32SC[1-4]
32FC[1-4]
64FC[1-4]

cv_bridge有时候会进行必要的像素格式转换,可以使用如下的字符串来表示目标格式:

  • mono8: CV_8UC1, grayscale image
  • mono16: CV_16UC1, 16-bit grayscale image
  • bgr8: CV_8UC3, color image with blue-green-red color order
  • rgb8: CV_8UC3, color image with red-green-blue color order
  • bgra8: CV_8UC4, BGR color image with an alpha channel
  • rgba8: CV_8UC4, RGB color image with an alpha channel

其中 mono8和bgr8是opencv中常用的格式。


roscpp

初始化和停止
ros::init(argc, argv, “my_node_name”);
ros::shutdown();
ros::ok() 用于检查node是否shutdown

timer

1
2
3
4
5
6
7
8
9
10
11
12
13
// 定时器,周期性执行回调。
ros::Timer ros::NodeHandle::createTimer(ros::Duration period, <callback>, bool oneshot = false);
ros::Timer timer = nh.createTimer(ros::Duration(0.1), timerCallback); // 创建timer

// 回调签名:
void callback(const ros::TimerEvent&);
// 其中 ros::TimerEvent:
// ros::Time last_expected -- 理想中前一个回调应该被执行的时间
// ros::Time last_real -- 实际上前一个回调执行的时间
// ros::Time current_expected -- 理想中当前回调应该被执行的时间
// ros::Time current_real -- 实际上当前回调执行的时间
// ros::WallTime profile.last_duration -- 前一个回调执行的耗时

线程

单线程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// 单线程调用
ros::init(argc, argv, "my_node");
ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe(...);
...
ros::spin();

// 自己实现 spin()
#include <ros/callback_queue.h>
ros::NodeHandle n;
while (ros::ok())
{
ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0.1));
}


// 另一种形式的单线程调用
ros::Rate r(10); // 10 hz
while (should_continue)
{
... do some work, publish some messages, etc. ...
ros::spinOnce();
r.sleep();
}

// 自己实现 spinOnce()
#include <ros/callback_queue.h>
ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0));

多线程

1
2
3
4
5
6
7
8
// 同步spin,阻塞
ros::MultiThreadedSpinner spinner(4); // Use 4 threads
spinner.spin(); // spin() will not return until the node has been shutdown

// 异步spin
ros::AsyncSpinner spinner(4); // Use 4 threads
spinner.start();
ros::waitForShutdown(); // 这里会阻塞

调用队列

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// ros有个全局默认队列, 所有回调都会挂到这个上边
ros::getGlobalCallbackQueue()

// 创建自己的调用队列,有两层次:
#include <ros/callback_queue.h>
ros::CallbackQueue my_queue;

// 层次1: 每个 subscribe(), advertise(), advertiseService()
// 使用 Options 结构体传入自己的队列

// 层次2: 每个 NodeHandle,更常用. ros::spin() 不会自动触发这些回调,需要手动去触发
ros::NodeHandle nh;
nh.setCallbackQueue(&my_callback_queue);

// 一次调用所有的回调
callAvailable()

// 一次调用一个回调
callOne()