Introduce rosbag2_transport layer and CLI #38

Martin-Idel-SI · 2018-09-20T15:28:50Z

This PR does a number of things:

Introduce rosbag2_transport to make rosbag2 independent of any subscription/publishing code and therefore rclcpp or rcl
Make rosbag2 a thin wrapper around rosbag2_storage (for now). Later on, rosbag2 can be used by other tooling
Introduce first CLI versions: ros2 bag play <filename> and ros2 bag record <topic names> as well as ros2 bag record --all. The two verbs and the given options are fully functional.

This PR improves testing:

Added a new package rosbag2_tests containing e2e-testing: We start ros2 bag record/play commands in separate processes
Make tests in rosbag2_transport not rely on particular storage, which decreases dependencies
Conform to naming conventions: All tests start with test, all other files don't.

This PR should also improve test stability to basically as much as we can

We always start any publishers or subscribers before the real test starts to make discovery easier
We start publishers and subscribers single threaded, we only ever use another thread for spinning
Any test failures which we have now seen (very rare as in <1% of all runs) must be related to discovery being too slow. There is nothing much we can do about it instead of wait longer.

To discuss:
There are a few duplicate files in testing (they are named the same to make it easy to spot). How should we go about this problem?

Should we create additional testing libraries in each package which provides headers/objects which are potentially interesting in other tests?
Or should we create another package rosbag2_test_commons which all packages might depend on as a pure testing dependency?

CMake already prepends libraries with `lib`, so the old name resulted in `liblibrosbag2`

- not referenced in header, therefore unnecessary

- Don't link against rosbag2_storage anymore

Martin-Idel-SI · 2018-09-26T15:35:27Z

I deleted all duplicate files and put them into the header-only library rosbag2_test_commons.

Karsten1987 · 2018-09-27T08:01:04Z

There is one problem I'd like to have fixed on this PR:

✔  09:58:54  ~/workspace/osrf/rosbag2_ws
 ➭ ros2 bag record -a
DISCLAIMER
ros2 bag is currently under development and not ready to use yet
[ERROR] [rosbag2_storage]: Could not open 'test.bag' with 'sqlite3'. Error: SQL error when preparing statement 'CREATE TABLE topics(id INTEGER PRIMARY KEY,name TEXT NOT NULL,type TEXT NOT NULL);' with return code: 1
[ERROR] [rosbag2_storage]: Could not load/open plugin with storage id 'sqlite3'.
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: No storage could be initialized. Abort
Abort trap: 6

This error occurs when trying to record the bag file twice in a row.

greimela-si · 2018-09-27T08:06:32Z

This error occurs when trying to record the bag file twice in a row.

Yes, that is the error message when trying to overwrite a bagfile.

For this PR we could add the behavior that bagfiles are overwritten by default, since the bagname is fixed.
In the future, we should probably offer a better error message if the user tries to overwrite a bagfile.

Karsten1987 · 2018-09-27T08:09:17Z

I am in favor of either overwriting it or gracefully failing if the bag file already exists. But somehow it shouldn't be terminating and the user should be informed about what's happening.

another comment is to rename rosbag2_test_commons to rosbag2_test_common. See here:
ros2/rmw_connext#240

Looks like it's a German thing ;-)

Karsten1987

a few more comments, sorry :)

Karsten1987 · 2018-09-27T08:13:46Z

rosbag2_test_commons/CMakeLists.txt

+endif()
+
+find_package(ament_cmake REQUIRED)
+find_package(ament_index_cpp REQUIRED)


can that line be inside the if(BUILD_TESTING) ?

rosbag2_transport/src/rosbag2_transport/rosbag2_transport_python.cpp

rosbag2_transport/src/rosbag2_transport/rosbag2_transport.cpp

Karsten1987 · 2018-09-27T08:38:06Z

rosbag2_transport/src/rosbag2_transport/rosbag2_transport.cpp

+  subscriptions_.clear();
+}
+
+std::shared_ptr<Rosbag2Node> Rosbag2Transport::setup_node()


why is the node not setup directly in the constructor? If so, then we don't need that check whether it's already set up or not.

the same for init and shutdown btw

Same reason as above for init. The setup_node may only be called after rclcpp::init has happended.

so it looks like you can't really call rclpy.init() and then spin your nodes in c++.
I guess with that we can leave the code as it is.

rosbag2_transport/src/rosbag2_transport/rosbag2_transport.cpp

rosbag2_transport/src/rosbag2_transport/player.cpp

rosbag2_transport/src/rosbag2_transport/player.hpp

This is a temporary solution and will be handled properly once a file path can be passed via the cli.

- Log every subscription - move all dependencies onside BUILD_TESTING for rosbag2_test_common

Karsten1987 · 2018-09-27T13:44:38Z

new CI:

Martin-Idel-SI · 2018-09-27T14:13:19Z

new CI (my fault, test_msgs were still being find_package'd):

Linux
Linux-aarch64
macOS
Windows

- Apparently ament_export_dependencies does not work in rosbag2_test_common

- Clock topic is no longer present on all nodes - Remove assumptions on foreign ros topics

Martin-Idel-SI · 2018-09-27T15:48:25Z

Let's try a new CI then:

Linux
Linux-aarch64
macOS
Windows

* First implementation of writer * Extract storage interface and sqlite3 implementation * Add test for sqlite storage * Split main() and rosbag2::record() * Add close() method to Storage * Add getMessage() method and refactor test * Refactor SqliteStorage constructor and open() * Add linters * Fix uncrustify * Fix cpplint * Specify test working directory * Better error handling * Use gmock matchers for assertions * Add test fixture for SqliteStorage tests * Extract message retrieval in tests into separate method * Add integration test for rosbag2::record() * Add ignore files for empty packages * Introduce create() method and refactor open() * Use shared pointer of Storage instead of SqliteStorage * Remove getDatabaseHandle() method * Fix uncrustify * Improve storage interface and add storage factory * Remove need of sleep() from integration test by usage of std::future * Move deletion of test database from fixture constructor to destructor * Use sqlite3 directly in intergration test instead of own sqlite wrapper * Move rosbag2::record() into Rosbag2 class * Use the test name as database file name * Add build instructions to README * ros2GH-37 Rename camelCase methods to snake_case * Use common test fixture * Add RAII wrapper for sqlite API * Mock away sqlite from sqlite_storage test * Use more reasonable assert * Add test * Add virtual destructor to WritableStorage * Use file_name instead of database_name in StorageFactory * Implement saving of test files in a tmp directory for linux/Mac * Try to implement saving of test files in a tmp directory for Windows * Write and use proper gmock SqliteWrappe mock * Refactor integration test and get rid of promise/future where possible * Throw exception in resource aquisition constructors * Make SqliteWrapper destructor virtual * Refactor test fixture and update SqliteWrapper mock * Fix warning when moving a temporary object * ros2GH-38 Refactor integration test * ros2GH-38 Get rid of superfluous string constructor in emplace_back() * ros2GH-38 Assert also execute_query() argument in sqlite_storage_test * ros2GH-38 add StorageFactory test * ros2GH-38 Refactor rosbag2 Test Fixture * ros2GH-40 Add first implementation of a rosbag reader and publisher * ros2GH-40 Add StorageFactory test when reading non-existing file * ros2GH-40 Fix uncrustify * ros2GH-40 Minor cleanup of CMakeLists * ros2GH-40 Wrap sqlite statements * ros2GH-40 Remove superfluous import * ros2GH-40 Use better include * ros2GH-40 Add play integration test * ros2GH-40 Fix Warning when moving a temporary object in reading * ros2GH-40 Initialize database pointer to nullpointer * ros2GH-40 Fix reader integration test * ros2GH-40 Polish storage wrapper * Revert "ros2GH-40: Wrap sqlite statements" * ros2GH-38 Fix Test Fixture after rebase * ros2GH-38 Refactor read_integration_test and refix Windows conversion warning * ros2GH-38 Add StorageFactory test * Simplify storage factory test * ros2GH-38 Try to fix flaky test * ros2GH-38 Move rclcpp::shutdown() at the end * ros2GH-41 Fix windows warning due to virtual explicit operator bool * ros2GH-41 Use sqlite3 vendor package in rosbag2 * ros2GH-41 Stop linking tests to sqlite * ros2GH-41 Fix test fixture on Windows * ros2GH-41 Cleanup test fixture includes * ros2GH-41 Print test database name * ros2GH-41 Correctly determine temp dir on Windows * ros2GH-41 Show error message on sqlite_open failure * ros2GH-41 Actually create temp dir on Windows * ros2GH-41 Fix bool conversion warning in VS2015 build * Fix CMakeLists.txt after rebase * ros2GH-40 Implement workouround to fix flaky test * Update package.xml * Add gtest test dependencies to package.xml * ros2GH-40 Move to sqlite3_storage_plugin folder - The separation into the intended structure and plugin apis is not there yet. However, most code will stay in the storage plugin for sqlite3 file. - Proper separation of this code into storage plugin and rosbag layer will be done in bosch-io/aos-rosbag2#5. * ros2GH-40 Add TODO comments and small cleanup

Karsten1987 and others added 30 commits September 20, 2018 08:46

rosbag2_transport package with python interface

3ae3a46

use cpp for python extension

3151e52

use rosbag2_transport cpp API

7a3b3df

use rosbag2_transport API in cli

f3f0e73

linters

25c3da6

ros2GH-25 Rename target librosbag2 to rosbag2

e0a844b

CMake already prepends libraries with `lib`, so the old name resulted in `liblibrosbag2`

ros2GH-21 Initial call of rosbag2.record() from rosbag2_transport

55c23e2

ros2GH-21 Add missing copyright header

200a537

ros2GH-21 Cleanup clang tidy issues

ec4352c

ros2GH-21 Remove rclcpp dependency from rosbag2

ca21e0a

ros2GH-21 Wire rosbag play into CLI

b419300

ros2GH-21 Add missing test_depend in rosbag2_transport package.xml

9937f97

ros2GH-21 Unify name of python import

1d94085

ros2GH-21 Enable -a in CLI, show help on wrong args

146d042

ros2GH-85 Introduce topic and type struct for readability

5dbea9c

ros2GH-85 Do not export sqlite3 as dependency from default plugins

6485950

- not referenced in header, therefore unnecessary

ros2GH-85 Move rosbag2 except typesupport to rosbag2_transport

4369d03

ros2GH-85 Add rosbag2 wrapper

a67739f

ros2GH-85 Change signature of create_topic to take TopicWithType

327515f

ros2GH-85 Use rosbag2 in rosbag2_transport

30e018a

- Don't link against rosbag2_storage anymore

ros2GH-84 Cleanup package.xmls and CMakeLists everywhere

2ba271a

ros2GH-85 Fix Windows build

e31f417

ros2GH-21 Add missing init() and shutdown() in record

f831694

ros2GH-85 Add visibility control to rosbag2

67c9d4e

ros2GH-85 Cleanup and documentation

08eb248

ros2GH-87 Add test package rosbag2_tests

fd1810c

ros2GH-87 [WIP] Add first working prototype of an end-to-end test

c495245

ros2GH-87 Use test_msgs instead of std_msgs/String in end-to-end test

011b48c

ros2GH-87 Use SIGTERM instead of SIGKILL and refactor test

ad14c7e

ros2GH-87 Make end-to-end test work on Windows

181eee5

ros2GH-25 rename rosbag2_test_commons -> rosbag2_test_common

35c0abb

Karsten1987 reviewed Sep 27, 2018

View reviewed changes

anhosi and others added 4 commits September 27, 2018 10:56

ros2GH-25 Overwrite already existing test.bag when recording

33c4f2e

This is a temporary solution and will be handled properly once a file path can be passed via the cli.

ros2GH-25 Cleanups

c2d414d

- Log every subscription - move all dependencies onside BUILD_TESTING for rosbag2_test_common

fix cmake typo for test_common

3111fd8

Remove superfluous loop in rosbag2 transport

534dba0

Delete superfluous test_msgs dependency

de966f0

Martin-Idel-SI added 2 commits September 27, 2018 17:03

Add rclcpp to test dependencies

53c9cae

- Apparently ament_export_dependencies does not work in rosbag2_test_common

Fix rosbag2 node test

490697f

- Clock topic is no longer present on all nodes - Remove assumptions on foreign ros topics

Martin-Idel-SI force-pushed the rosbag2_transport branch from 5bad5b8 to 490697f Compare September 27, 2018 15:04

Fix dependencies by exporting them explicitly

4939325

Karsten1987 approved these changes Sep 28, 2018

View reviewed changes

Karsten1987 merged commit 1370c7f into ros2:master Sep 28, 2018

Karsten1987 removed the in review label Sep 28, 2018

Karsten1987 mentioned this pull request Sep 28, 2018

rosbag_write_integration_test is flaky #25

Closed

greimela-si deleted the rosbag2_transport branch September 28, 2018 13:38

Martin-Idel-SI pushed a commit to bosch-io/rosbag2 that referenced this pull request Nov 30, 2018

ros2GH-38 Add storage_id option to CLI for play

1298439

Martin-Idel-SI pushed a commit to bosch-io/rosbag2 that referenced this pull request Dec 5, 2018

ros2GH-38 Add storage_id option to CLI for play

dfa33e4

Martin-Idel-SI pushed a commit to bosch-io/rosbag2 that referenced this pull request Dec 6, 2018

ros2GH-38 Add storage_id option to CLI for play

a916b19

Martin-Idel-SI pushed a commit to bosch-io/rosbag2 that referenced this pull request Dec 6, 2018

ros2GH-38 Add storage_id option to CLI for play

c9bd5c2

Martin-Idel-SI pushed a commit to bosch-io/rosbag2 that referenced this pull request Dec 11, 2018

ros2GH-38 Add storage_id option to CLI for play

56312de

Martin-Idel-SI pushed a commit to bosch-io/rosbag2 that referenced this pull request Dec 13, 2018

ros2GH-38 Add storage_id option to CLI for play

6c71b45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce rosbag2_transport layer and CLI #38

Introduce rosbag2_transport layer and CLI #38

Martin-Idel-SI commented Sep 20, 2018

Martin-Idel-SI commented Sep 26, 2018

Karsten1987 commented Sep 27, 2018

greimela-si commented Sep 27, 2018

Karsten1987 commented Sep 27, 2018

Karsten1987 left a comment

Karsten1987 Sep 27, 2018

Karsten1987 Sep 27, 2018

Karsten1987 Sep 27, 2018

anhosi Sep 27, 2018

Karsten1987 Sep 28, 2018

Karsten1987 commented Sep 27, 2018

Martin-Idel-SI commented Sep 27, 2018

Martin-Idel-SI commented Sep 27, 2018

Introduce rosbag2_transport layer and CLI #38

Introduce rosbag2_transport layer and CLI #38

Conversation

Martin-Idel-SI commented Sep 20, 2018

Martin-Idel-SI commented Sep 26, 2018

Karsten1987 commented Sep 27, 2018

greimela-si commented Sep 27, 2018

Karsten1987 commented Sep 27, 2018

Karsten1987 left a comment

Choose a reason for hiding this comment

Karsten1987 Sep 27, 2018

Choose a reason for hiding this comment

Karsten1987 Sep 27, 2018

Choose a reason for hiding this comment

Karsten1987 Sep 27, 2018

Choose a reason for hiding this comment

anhosi Sep 27, 2018

Choose a reason for hiding this comment

Karsten1987 Sep 28, 2018

Choose a reason for hiding this comment

Karsten1987 commented Sep 27, 2018

Martin-Idel-SI commented Sep 27, 2018

Martin-Idel-SI commented Sep 27, 2018