(demo, sqlite3) First working rosbag2 implementation (ros2#6)

* First implementation of writer

* Extract storage interface and sqlite3 implementation

* Add test for sqlite storage

* Split main() and rosbag2::record()

* Add close() method to Storage

* Add getMessage() method and refactor test

* Refactor SqliteStorage constructor and open()

* Add linters

* Fix uncrustify

* Fix cpplint

* Specify test working directory

* Better error handling

* Use gmock matchers for assertions

* Add test fixture for SqliteStorage tests

* Extract message retrieval in tests into separate method

* Add integration test for rosbag2::record()

* Add ignore files for empty packages

* Introduce create() method and refactor open()

* Use shared pointer of Storage instead of SqliteStorage

* Remove getDatabaseHandle() method

* Fix uncrustify

* Improve storage interface and add storage factory

* Remove need of sleep() from integration test by usage of std::future

* Move deletion of test database from fixture constructor to destructor

* Use sqlite3 directly in intergration test instead of own sqlite wrapper

* Move rosbag2::record() into Rosbag2 class

* Use the test name as database file name

* Add build instructions to README

* ros2GH-37 Rename camelCase methods to snake_case

* Use common test fixture

* Add RAII wrapper for sqlite API

* Mock away sqlite from sqlite_storage test

* Use more reasonable assert

* Add test

* Add virtual destructor to WritableStorage

* Use file_name instead of database_name in StorageFactory

* Implement saving of test files in a tmp directory for linux/Mac

* Try to implement saving of test files in a tmp directory for Windows

* Write and use proper gmock SqliteWrappe mock

* Refactor integration test and get rid of promise/future where possible

* Throw exception in resource aquisition constructors

* Make SqliteWrapper destructor virtual

* Refactor test fixture and update SqliteWrapper mock

* Fix warning when moving a temporary object

* ros2GH-38 Refactor integration test

* ros2GH-38 Get rid of superfluous string constructor in emplace_back()

* ros2GH-38 Assert also execute_query() argument in sqlite_storage_test

* ros2GH-38 add StorageFactory test

* ros2GH-38 Refactor rosbag2 Test Fixture

* ros2GH-40 Add first implementation of a rosbag reader and publisher

* ros2GH-40 Add StorageFactory test when reading non-existing file

* ros2GH-40 Fix uncrustify

* ros2GH-40 Minor cleanup of CMakeLists

* ros2GH-40 Wrap sqlite statements

* ros2GH-40 Remove superfluous import

* ros2GH-40 Use better include

* ros2GH-40 Add play integration test

* ros2GH-40 Fix Warning when moving a temporary object in reading

* ros2GH-40 Initialize database pointer to nullpointer

* ros2GH-40 Fix reader integration test

* ros2GH-40 Polish storage wrapper

* Revert "ros2GH-40: Wrap sqlite statements"

* ros2GH-38 Fix Test Fixture after rebase

* ros2GH-38 Refactor read_integration_test and refix Windows conversion warning

* ros2GH-38 Add StorageFactory test

* Simplify storage factory test

* ros2GH-38 Try to fix flaky test

* ros2GH-38 Move rclcpp::shutdown() at the end

* ros2GH-41 Fix windows warning due to virtual explicit operator bool

* ros2GH-41 Use sqlite3 vendor package in rosbag2

* ros2GH-41 Stop linking tests to sqlite

* ros2GH-41 Fix test fixture on Windows

* ros2GH-41 Cleanup test fixture includes

* ros2GH-41 Print test database name

* ros2GH-41 Correctly determine temp dir on Windows

* ros2GH-41 Show error message on sqlite_open failure

* ros2GH-41 Actually create temp dir on Windows

* ros2GH-41 Fix bool conversion warning in VS2015 build

* Fix CMakeLists.txt after rebase

* ros2GH-40 Implement workouround to fix flaky test

* Update package.xml

* Add gtest test dependencies to package.xml

* ros2GH-40 Move to sqlite3_storage_plugin folder

- The separation into the intended structure and plugin apis is not
  there yet. However, most code will stay in the storage plugin for
  sqlite3 file.
- Proper separation of this code into storage plugin and rosbag layer
  will be done in bosch-io/aos-rosbag2#5.

* ros2GH-40 Add TODO comments and small cleanup

README.md

@@ -3,3 +3,37 @@
 THIS IS WORK IN PROGRESS AND NOT READY TO BE USED YET
 
 Repository for implementing ROSBag2 as described in its corresponding [design article](https://github.com/ros2/design/blob/f69fbbd11848e3dd6866b71a158a1902e31e92f1/articles/rosbags.md)
+
+## Build instructions
+
+Create a new workspace:
+
+```
+$ mkdir -p ~/rosbag_ws/src
+$ cd ~/rosbag_wsk/src
+```
+
+Clone this repository into the source folder:
+
+```
+$ git clone https://github.com/ros2/rosbag2.git
+```
+
+Then build all the packages with this command:
+
+```
+$ colcon build --merge-install
+```
+
+The `--merge-install` flag is optional but ensures a cleaner environment which is helpful for development.
+
+#### Executing tests
+
+The tests can be run using the following commands:
+
+```
+$ colcon test --merge-install
+$ colcon test-result --verbose
+```
+
+The first command executes the test and the second command displays the errors (if any).

sqlite3_storage_plugin/CMakeLists.txt

@@ -0,0 +1,93 @@
+cmake_minimum_required(VERSION 3.5)
+project(sqlite3_storage_plugin)
+
+# Default to C99
+if(NOT CMAKE_C_STANDARD)
+  set(CMAKE_C_STANDARD 99)
+endif()
+
+# Default to C++14
+if(NOT CMAKE_CXX_STANDARD)
+  set(CMAKE_CXX_STANDARD 14)
+endif()
+
+if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+  add_compile_options(-Wall -Wextra -Wpedantic -Werror)
+endif()
+
+find_package(ament_cmake REQUIRED)
+find_package(rclcpp REQUIRED)
+find_package(std_msgs REQUIRED)
+find_package(sqlite3_vendor REQUIRED)
+
+find_package(SQLite3 REQUIRED)  # provided by sqlite3_vendor
+
+add_library(librosbag
+  include/rosbag2/rosbag2.hpp
+  src/rosbag2/rosbag2.cpp
+  src/rosbag2/storage/readable_storage.hpp
+  src/rosbag2/storage/writable_storage.hpp
+  src/rosbag2/storage/storage_factory.hpp
+  src/rosbag2/storage/storage_factory.cpp
+  src/rosbag2/storage/sqlite/sqlite_wrapper.hpp
+  src/rosbag2/storage/sqlite/sqlite_wrapper.cpp
+  src/rosbag2/storage/sqlite/sqlite_storage.hpp
+  src/rosbag2/storage/sqlite/sqlite_storage.cpp)
+ament_target_dependencies(librosbag rclcpp std_msgs SQLite3)
+target_link_libraries(librosbag)
+target_include_directories(librosbag
+  PUBLIC
+  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+  $<INSTALL_INTERFACE:include>
+)
+
+add_executable(${PROJECT_NAME}_record src/rosbag2/demo_record.cpp)
+target_link_libraries(${PROJECT_NAME}_record librosbag)
+
+add_executable(${PROJECT_NAME}_play src/rosbag2/demo_play.cpp)
+target_link_libraries(${PROJECT_NAME}_play librosbag)
+# TODO(anhosi): Add install for library and headers (GH-5)
+
+if(BUILD_TESTING)
+  # TODO(Martin-Idel-SI): Add copyright linter or use ament_lint_auto() once available
+  find_package(ament_cmake_cppcheck REQUIRED)
+  find_package(ament_cmake_cpplint REQUIRED)
+  find_package(ament_cmake_lint_cmake REQUIRED)
+  find_package(ament_cmake_uncrustify REQUIRED)
+  ament_cppcheck()
+  ament_cpplint()
+  ament_lint_cmake()
+  ament_uncrustify()
+
+  find_package(ament_cmake_gtest REQUIRED)
+  find_package(ament_cmake_gmock REQUIRED)
+
+  ament_add_gmock(sqlite_storage_test test/rosbag2/storage/sqlite_storage_test.cpp
+    test/rosbag2/rosbag2_test_fixture.hpp
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+  if(TARGET sqlite_storage_test)
+    target_link_libraries(sqlite_storage_test librosbag)
+  endif()
+
+  ament_add_gmock(storage_factory_test test/rosbag2/storage/storage_factory_test.cpp
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+  if(TARGET storage_factory_test)
+    target_link_libraries(storage_factory_test librosbag)
+  endif()
+
+  ament_add_gmock(rosbag2_write_integration_test test/rosbag2/rosbag2_write_integration_test.cpp
+    test/rosbag2/rosbag2_test_fixture.hpp
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+  if(TARGET rosbag2_write_integration_test)
+    target_link_libraries(rosbag2_write_integration_test librosbag)
+  endif()
+
+  ament_add_gmock(rosbag2_read_integration_test test/rosbag2/rosbag2_read_integration_test.cpp
+    test/rosbag2/rosbag2_test_fixture.hpp
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+  if(TARGET rosbag2_read_integration_test)
+    target_link_libraries(rosbag2_read_integration_test librosbag)
+  endif()
+endif()
+
+ament_package()

sqlite3_storage_plugin/include/rosbag2/rosbag2.hpp

@@ -0,0 +1,38 @@
+/*
+ *  Copyright (c) 2018,  Bosch Software Innovations GmbH.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+#ifndef ROSBAG2__ROSBAG2_HPP_
+#define ROSBAG2__ROSBAG2_HPP_
+
+#include <functional>
+#include <string>
+
+namespace rosbag2
+{
+
+class Rosbag2
+{
+public:
+  void record(
+    const std::string & file_name,
+    const std::string & topic_name,
+    std::function<void(void)> after_write_action = nullptr);
+  void play(const std::string & file_name, const std::string & topic_name);
+};
+
+}  // namespace rosbag2
+
+#endif  // ROSBAG2__ROSBAG2_HPP_

sqlite3_storage_plugin/package.xml

@@ -0,0 +1,30 @@
+<?xml version="1.0"?>
+<?xml-model href="http://download.ros.org/schema/package_format2.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
+<package format="2">
+  <name>sqlite3_storage_plugin</name>
+  <version>0.0.0</version>
+  <description>ROSBag2 SQLite3 storage plugin</description>
+  <maintainer email="karsten.knese@googlemail.com">Karsten Knese</maintainer>
+  <license>Apache License 2.0</license>
+
+  <buildtool_depend>ament_cmake</buildtool_depend>
+
+  <build_depend>sqlite3_vendor</build_depend>
+  <build_depend>rclcpp</build_depend>
+  <build_depend>std_msgs</build_depend>
+
+  <exec_depend>sqlite3_vendor</exec_depend>
+  <exec_depend>rclcpp</exec_depend>
+  <exec_depend>std_msgs</exec_depend>
+
+  <test_depend>ament_lint_auto</test_depend>
+  <test_depend>ament_lint_common</test_depend>
+  <test_depend>ament_cmake_gmock</test_depend>
+  <test_depend>ament_cmake_gtest</test_depend>
+  <test_depend>rclcpp</test_depend>
+  <test_depend>std_msgs</test_depend>
+
+  <export>
+    <build_type>ament_cmake</build_type>
+  </export>
+</package>

sqlite3_storage_plugin/src/rosbag2/demo_play.cpp

@@ -0,0 +1,31 @@
+/*
+ *  Copyright (c) 2018,  Bosch Software Innovations GmbH.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+#include "rclcpp/rclcpp.hpp"
+
+#include "rosbag2/rosbag2.hpp"
+
+int main(int argc, const char ** argv)
+{
+  rclcpp::init(argc, argv);
+
+  rosbag2::Rosbag2 rosbag2;
+  rosbag2.play("test.bag", "string_topic");
+
+  rclcpp::shutdown();
+
+  return 0;
+}

sqlite3_storage_plugin/src/rosbag2/demo_record.cpp

@@ -0,0 +1,39 @@
+/*
+ *  Copyright (c) 2018,  Bosch Software Innovations GmbH.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+#include <cstdio>
+#include <string>
+
+#include "rclcpp/rclcpp.hpp"
+
+#include "rosbag2/rosbag2.hpp"
+
+int main(int argc, const char ** argv)
+{
+  // TODO(anhosi): allow output file to be specified by cli argument and do proper checking if
+  // file already exists
+  std::string filename("test.bag");
+  std::remove(filename.c_str());
+
+  rclcpp::init(argc, argv);
+
+  rosbag2::Rosbag2 rosbag2;
+  rosbag2.record(filename, "string_topic");
+
+  rclcpp::shutdown();
+
+  return 0;
+}

sqlite3_storage_plugin/src/rosbag2/rosbag2.cpp

@@ -0,0 +1,75 @@
+/*
+ *  Copyright (c) 2018,  Bosch Software Innovations GmbH.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+#include "rosbag2/rosbag2.hpp"
+
+#include <memory>
+#include <string>
+
+#include "rclcpp/rclcpp.hpp"
+#include "std_msgs/msg/string.hpp"
+
+#include "storage/storage_factory.hpp"
+#include "storage/sqlite/sqlite_storage.hpp"
+
+namespace rosbag2
+{
+
+void Rosbag2::record(
+  const std::string & file_name,
+  const std::string & topic_name,
+  std::function<void(void)> after_write_action)
+{
+  StorageFactory factory;
+  std::unique_ptr<WritableStorage> storage = factory.get_for_writing(file_name);
+
+  if (storage) {
+    auto node = std::make_shared<rclcpp::Node>("rosbag_node");
+    auto subscription = node->create_subscription<std_msgs::msg::String>(
+      topic_name,
+      [&storage, after_write_action](std_msgs::msg::String::ConstSharedPtr msg) {
+        storage->write(msg->data);
+        if (after_write_action) {
+          after_write_action();
+        }
+      });
+
+    // TODO(anhosi): use proper logging from rcutils
+    std::cout << "Waiting for messages..." << std::endl;
+    rclcpp::spin(node);
+  }
+}
+
+void Rosbag2::play(const std::string & file_name, const std::string & topic_name)
+{
+  StorageFactory factory;
+  std::unique_ptr<ReadableStorage> storage = factory.get_for_reading(file_name);
+
+  if (storage) {
+    auto messages = storage->get_messages();
+    auto node = std::make_shared<rclcpp::Node>("rosbag_publisher_node");
+    auto publisher = node->create_publisher<std_msgs::msg::String>(topic_name);
+    for (const auto & message : messages) {
+      auto string_msg = std_msgs::msg::String();
+      string_msg.data = message;
+      // without the sleep_for() many messages are lost.
+      std::this_thread::sleep_for(std::chrono::milliseconds(500));
+      publisher->publish(string_msg);
+    }
+    rclcpp::spin_some(node);
+  }
+}
+
+}  // namespace rosbag2

sqlite3_storage_plugin/src/rosbag2/storage/readable_storage.hpp

@@ -0,0 +1,36 @@
+/*
+ *  Copyright (c) 2018,  Bosch Software Innovations GmbH.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+#ifndef ROSBAG2__STORAGE__READABLE_STORAGE_HPP_
+#define ROSBAG2__STORAGE__READABLE_STORAGE_HPP_
+
+#include <string>
+#include <vector>
+
+namespace rosbag2
+{
+
+class ReadableStorage
+{
+public:
+  virtual ~ReadableStorage() = default;
+
+  virtual std::vector<std::string> get_messages() = 0;
+};
+
+}  // namespace rosbag2
+
+#endif  // ROSBAG2__STORAGE__READABLE_STORAGE_HPP_