Add entry point for converter plugins (#47)

* GH-101 Add converter interface

* GH-102 Create format converter factory

* GH-103 Write documentation for converter plugin authors

* GH-16 Adjust rosbag2 message type

* GH-16 Change name of converter interface to include "serialization"

- Easier to differentiate between storage format (e.g. sqlite)
  and serialization format (e.g. cdr)
- Closer to naming in ros middleware

* GH-16 Improve plugin development documentation

- Also adapt to name changes

* GH-16 Fix naming of SerializationFormatConverterFactory

docs/converter_plugin_development.md

@@ -0,0 +1,56 @@
+# Converter plugins
+
+Converter plugins are used to convert between different storage serialization formats.
+A storage for rosbags in ROS 2 obtains and stores its data using a serialization format.
+By default, it will store a bag in the serialization format of the middleware it was recorded with (two middlewares may share the same serialization format).
+If messages should be read by a middleware with a different serialization format, the messages must be converted to that format.
+To simplify conversion between all formats each plugin provides functions to convert to and from generic ROS 2 messages.
+
+Rosbag2 is shipped with a default converter plugin to convert between ROS 2 messages and serialized messages in the CDR format (this is the general serialization format specified by DDS).
+
+## Writing converter plugins
+
+To write a plugin `MyConverter`, implement the interface `rosbag2::SerializationFormatConverterInterface`.
+
+The plugin interface provides two functions: 
+
+- The function `serialize` takes an initialized SerializedBagMessage and a ROS 2 message together with information about its topic and type and should fill the SerializedBagMessage with the ROS 2 message contents.
+- The other function `deserialize` does the reverse, taking a full SerializedBagMessage to fill a preallocated ROS 2 message using the provided typesupport (which has to match the actual type of the SerializedBagMessage).
+
+In order to find the plugin at runtime, it needs to be exported to the pluginlib. 
+Add the following lines in the `my_converter.cpp`:
+
+```
+#include "pluginlib/class_list_macros.hpp"
+PLUGINLIB_EXPORT_CLASS(MyConverter, rosbag2::SerializationFormatConverterInterface)
+```
+
+Furthermore, we need some meta-information in the form of a `plugin_description.xml` file.
+In the case of `MyConverter` this would look like:
+```
+<library path="my_converter_library">
+  <class
+    name="my_storage_format_converter"
+    type="MyConverter"
+    base_class_type="rosbag2::SerializationFormatConverterInterface"
+  >
+    <description>This is a converter plugin for my storage format.</description>
+  </class>
+</library>
+```
+where `my_converter_library` is the name of the library, while `my_storage_format_converter` is the identifier used by the pluginlib to load it.
+It **must** have the format `<rmw storage format>_converter` as described in the section "How to choose conversion at runtime".
+
+In addition, in the `CMakeLists.txt` the `plugin_description.xml` file needs to be added to the index to be found at runtime:
+```
+pluginlib_export_plugin_description_file(rosbag2 plugin_description.xml)
+```
+
+The first argument `rosbag2` denotes the ament index key we add our plugin to (this will always be `rosbag2` for converter plugins), while the second argument is the path to the plugin description file.
+
+## How to choose conversion at runtime
+
+The conversion will be chosen automatically according to the storage format specified in the bagfile or specified by the user.
+To achieve this, rosbag2 searches for a converter plugin with name `<storage format>_converter`. 
+For instance, the default plugin for conversion between CDR messages is named `cdr_converter` in its `pluginlib.txt` file.
+There should be no need for different converters for the same storage format hence this is not supported.

rosbag2/CMakeLists.txt

@@ -17,6 +17,7 @@ endif()
 
 find_package(ament_cmake REQUIRED)
 find_package(ament_index_cpp REQUIRED)
+find_package(pluginlib REQUIRED)
 find_package(poco_vendor)
 find_package(Poco COMPONENTS Foundation)
 find_package(rcutils REQUIRED)
@@ -25,11 +26,13 @@ find_package(rosidl_generator_cpp REQUIRED)
 
 add_library(${PROJECT_NAME} SHARED
   src/rosbag2/sequential_reader.cpp
+  src/rosbag2/serialization_format_converter_factory.cpp
   src/rosbag2/typesupport_helpers.cpp
   src/rosbag2/writer.cpp)
 
 ament_target_dependencies(${PROJECT_NAME}
   ament_index_cpp
+  pluginlib
   Poco
   rcutils
   rosbag2_storage
@@ -46,6 +49,9 @@ target_include_directories(${PROJECT_NAME}
 # which is appropriate when building the dll but not consuming it.
 target_compile_definitions(${PROJECT_NAME} PRIVATE "ROSBAG2_BUILDING_DLL")
 
+# prevent pluginlib from using boost
+target_compile_definitions(${PROJECT_NAME} PUBLIC "PLUGINLIB__DISABLE_BOOST_FUNCTIONS")
+
 install(
   DIRECTORY include/
   DESTINATION include)
@@ -71,6 +77,25 @@ if(BUILD_TESTING)
   if(TARGET test_typesupport_helpers)
     target_link_libraries(test_typesupport_helpers rosbag2)
   endif()
+
+  add_library(
+    converter_test_plugin
+    SHARED
+    test/rosbag2/converter_test_plugin.cpp)
+  target_link_libraries(converter_test_plugin rosbag2)
+  install(
+    TARGETS converter_test_plugin
+    ARCHIVE DESTINATION lib
+    LIBRARY DESTINATION lib
+    RUNTIME DESTINATION bin)
+  pluginlib_export_plugin_description_file(rosbag2 test/rosbag2/converter_test_plugin.xml)
+
+  ament_add_gmock(test_converter_factory
+    test/rosbag2/test_converter_factory.cpp)
+  if(TARGET test_converter_factory)
+    target_include_directories(test_converter_factory PRIVATE include)
+    target_link_libraries(test_converter_factory rosbag2)
+  endif()
 endif()
 
 ament_package()

rosbag2/include/rosbag2/serialization_format_converter_factory.hpp

@@ -0,0 +1,66 @@
+// Copyright 2018 Open Source Robotics Foundation, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_FACTORY_HPP_
+#define ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_FACTORY_HPP_
+
+#include "rosbag2/serialization_format_converter_factory_interface.hpp"
+
+#include <memory>
+#include <string>
+
+#include "rosbag2/serialization_format_converter_interface.hpp"
+#include "rosbag2/visibility_control.hpp"
+
+// This is necessary because of using stl types here. It is completely safe, because
+// a) the member is not accessible from the outside
+// b) there are no inline functions.
+#ifdef _WIN32
+# pragma warning(push)
+# pragma warning(disable:4251)
+#endif
+
+namespace pluginlib
+{
+
+template<class T>
+class ClassLoader;
+
+}  // namespace pluginlib
+
+namespace rosbag2
+{
+
+class ROSBAG2_PUBLIC SerializationFormatConverterFactory
+  : public SerializationFormatConverterFactoryInterface
+{
+public:
+  SerializationFormatConverterFactory();
+
+  ~SerializationFormatConverterFactory() override;
+
+  std::shared_ptr<SerializationFormatConverterInterface>
+  load_converter(const std::string & format) override;
+
+private:
+  std::unique_ptr<pluginlib::ClassLoader<SerializationFormatConverterInterface>> class_loader_;
+};
+
+}  // namespace rosbag2
+
+#ifdef _WIN32
+# pragma warning(pop)
+#endif
+
+#endif  // ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_FACTORY_HPP_

rosbag2/include/rosbag2/serialization_format_converter_factory_interface.hpp

@@ -0,0 +1,38 @@
+// Copyright 2018 Open Source Robotics Foundation, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_FACTORY_INTERFACE_HPP_
+#define ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_FACTORY_INTERFACE_HPP_
+
+#include <memory>
+#include <string>
+
+#include "rosbag2/serialization_format_converter_interface.hpp"
+#include "rosbag2/visibility_control.hpp"
+
+namespace rosbag2
+{
+
+class ROSBAG2_PUBLIC SerializationFormatConverterFactoryInterface
+{
+public:
+  virtual ~SerializationFormatConverterFactoryInterface() = default;
+
+  virtual std::shared_ptr<SerializationFormatConverterInterface>
+  load_converter(const std::string & format) = 0;
+};
+
+}  // namespace rosbag2
+
+#endif  // ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_FACTORY_INTERFACE_HPP_

rosbag2/include/rosbag2/serialization_format_converter_interface.hpp

@@ -0,0 +1,48 @@
+// Copyright 2018, Bosch Software Innovations GmbH.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_INTERFACE_HPP_
+#define ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_INTERFACE_HPP_
+
+#include <memory>
+#include <string>
+
+#include "rosbag2/types/ros2_message.hpp"
+#include "rosbag2/types.hpp"
+#include "rcutils/types.h"
+#include "rosbag2_storage/serialized_bag_message.hpp"
+#include "rosidl_typesupport_cpp/message_type_support.hpp"
+
+namespace rosbag2
+{
+
+class SerializationFormatConverterInterface
+{
+public:
+  virtual ~SerializationFormatConverterInterface() = default;
+
+  virtual void deserialize(
+    std::shared_ptr<rosbag2_ros2_message_t> ros_message,
+    std::shared_ptr<const SerializedBagMessage> serialized_message,
+    const rosidl_message_type_support_t * type_support) = 0;
+
+  virtual void serialize(
+    std::shared_ptr<SerializedBagMessage> serialized_message,
+    std::shared_ptr<const rosbag2_ros2_message_t> ros_message,
+    const rosidl_message_type_support_t * type_support) = 0;
+};
+
+}  // namespace rosbag2
+
+#endif  // ROSBAG2__SERIALIZATION_FORMAT_CONVERTER_INTERFACE_HPP_

rosbag2/include/rosbag2/types/ros2_message.hpp

@@ -0,0 +1,29 @@
+// Copyright 2018, Bosch Software Innovations GmbH.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2__TYPES__ROS2_MESSAGE_HPP_
+#define ROSBAG2__TYPES__ROS2_MESSAGE_HPP_
+
+#include "rcutils/time.h"
+#include "rcutils/allocator.h"
+
+typedef struct rosbag2_ros2_message_t
+{
+  void * message;
+  const char * topic_name;
+  rcutils_time_point_value_t timestamp;
+  rcutils_allocator_t allocator;
+} rosbag2_ros2_message_t;
+
+#endif  // ROSBAG2__TYPES__ROS2_MESSAGE_HPP_

rosbag2/package.xml

@@ -1,5 +1,4 @@
 <?xml version="1.0"?>
-<?xml-model href="http://download.ros.org/schema/package_format2.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="2">
   <name>rosbag2</name>
   <version>0.0.0</version>
@@ -10,6 +9,7 @@
   <buildtool_depend>ament_cmake</buildtool_depend>
 
   <depend>ament_index_cpp</depend>
+  <depend>pluginlib</depend>
   <depend>poco_vendor</depend>
   <depend>rcutils</depend>
   <depend>rosbag2_storage</depend>

rosbag2/src/rosbag2/serialization_format_converter_factory.cpp

@@ -0,0 +1,60 @@
+// Copyright 2018 Open Source Robotics Foundation, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "rosbag2/serialization_format_converter_factory.hpp"
+
+#include <memory>
+#include <string>
+
+#include "pluginlib/class_loader.hpp"
+#include "rosbag2/logging.hpp"
+
+namespace rosbag2
+{
+
+SerializationFormatConverterFactory::SerializationFormatConverterFactory()
+{
+  try {
+    class_loader_ = std::make_unique<pluginlib::ClassLoader<SerializationFormatConverterInterface>>(
+      "rosbag2", "rosbag2::SerializationFormatConverterInterface");
+  } catch (const std::exception & e) {
+    ROSBAG2_LOG_ERROR_STREAM("Unable to create class loader instance: " << e.what());
+    throw e;
+  }
+}
+
+SerializationFormatConverterFactory::~SerializationFormatConverterFactory() = default;
+
+std::shared_ptr<SerializationFormatConverterInterface>
+SerializationFormatConverterFactory::load_converter(const std::string & format)
+{
+  auto converter_id = format + "_converter";
+
+  const auto & registered_classes = class_loader_->getDeclaredClasses();
+  auto class_exists = std::find(registered_classes.begin(), registered_classes.end(), converter_id);
+  if (class_exists == registered_classes.end()) {
+    ROSBAG2_LOG_ERROR_STREAM("Requested converter id '" << converter_id << "' does not exist");
+    return nullptr;
+  }
+
+  try {
+    return std::shared_ptr<SerializationFormatConverterInterface>(
+      class_loader_->createUnmanagedInstance(converter_id));
+  } catch (const std::runtime_error & ex) {
+    ROSBAG2_LOG_ERROR_STREAM("Unable to load instance of converter interface: " << ex.what());
+    return nullptr;
+  }
+}
+
+}  // namespace rosbag2