Add beeline-style propagation of the cmd context to called commands #74

DavidS · 2020-11-19T17:15:35Z

When using something like https://github.com/puppetlabs/rspec_honeycomb_formatter , it would be nice to connect the individual spans to the bigger picture of the buildevent steps and commands. This change uses the beeline's PropagationContext marshaling to pass on the current event to the called command.

In the style of the rest of the HONEYCOMB_* environment variables and the similarily named X-Honeycomb-Trace HTTP header, this establishes HONEYCOMB_TRACE to pass the propagation context to other processes.

I'm also adopting this naming for the two other projects my team is maintaining where trace contexts are passed around:

DavidS · 2020-11-19T17:30:49Z

This is my first significant piece of go, so caveat emptor 😅 please be gentle and thorough.

I'm not convinced that my choice of environment variable is great and would be open to better suggestions.

MikeGoldsmith

Functionally, the PR looks good. I've one super minor suggestion about reusing a variable.

I don't think HTTP_X_HONEYCOMB_TRACE is the right choice for an env var to hold trace state. I'll see if I can find a more suitable alternative. If this is something we are expecting to support, we'd need have consistency across the beelines to try to read the env var.

MikeGoldsmith · 2020-12-08T15:24:17Z

cmd_cmd.go

-			err := runCommand(subcmd)
+
+			// copy out the current set of fields to avoid later modification
+			localFields := map[string]interface{}{}


Minor: create a variable for the spanID so we don't need to fmt.Sprintf twice 😄

good call. fixed that.

MikeGoldsmith · 2020-12-08T15:26:17Z

cmd_cmd.go

 	fmt.Println("running /bin/bash -c", subcmd)
 	cmd := exec.Command("/bin/bash", "-c", subcmd)

+	cmd.Env = append(os.Environ(),
+		"HTTP_X_HONEYCOMB_TRACE=" + propagation.MarshalHoneycombTraceContext(prop),


I'm not sure this is right thing to use either, but then also I don't have an alternative suggestion.

This env var is only used in beeline-ruby: https://github.com/search?q=org%3Ahoneycombio+HTTP_X_HONEYCOMB_TRACE&type=code.

I've changed this to only HONEYCOMB_TRACE. This matches the other HONEYCOMB_* environment variables as well as the X-Honeycomb-Trace HTTP header.

I should also add a bit of docs around that, I guess.

It might be worth noting that the HTTP_X_HONEYCOMB_TRACE is not an environment variable in the ruby beeline, it's only pulled from the rack environment

DavidS · 2020-12-08T16:11:00Z

Also added a bit of documentation into the README on how to use the variable.

MikeGoldsmith

Yeah, I think this is good. I don't believe other beelines try to read an env var for trace context and HONEYCOMB_TRACE is a good choice. However, given a string value most beelines can construct a context object. It's a little manual for now and we can think about how we'd like to handle that more elegantly later.

I'd like a second review from @robbkidd for the ruby example, as it's not my primary language.

MikeGoldsmith · 2020-12-08T16:22:59Z

README.md

+# establish a command-level span, linking to the buildevent
+process_span = Honeycomb.start_span(name: File.basename($PROGRAM_NAME), serialized_trace: ENV['HONEYCOMB_TRACE'])
+Honeycomb.add_field_to_trace('process.full_name', $PROGRAM_NAME)
+Honeycomb.add_field_to_trace('process.args', ARGV)


This is an example, but I think we should be careful advocating adding cmd args as trace fields. Many apps provide sensitive data via command args which would be be leaked.

I can remove that line, but fwiw, buildevents itself captures ARGV.

DavidS · 2020-12-08T17:12:22Z

I'd like a second review from @robbkidd for the ruby example, as it's not my primary language.

The example is straight from https://github.com/puppetlabs/rspec_honeycomb_formatter/blob/e32d0a6d5115a3175cbee7506191b8a8578dfb59/lib/rspec_honeycomb_formatter.rb#L14-L29 and quite simliar to the other place I've used this: https://github.com/puppetlabs/puppet_litmus/blob/ed88fa024bffe50008ddf3d0756d7a9603fbdcb7/lib/puppet_litmus/rake_helper.rb#L12

robbkidd · 2020-12-08T21:46:27Z

README.md

+Honeycomb.add_field_to_trace('process.args', ARGV)
+
+# override the HONEYCOMB_TRACE for sub-processes
+ENV['HONEYCOMB_TRACE'] = process_span.to_trace_header unless ENV['HONEYCOMB_TRACE']


My understanding of this line is that ENV['HONEYCOMB_TRACE'] will only be set to the new trace header if the environment variable is not already set, the equivalent of ...

ENV['HONEYCOMB_TRACE'] ||= process_span.to_trace_header

The mention of "override" in the comment has me thinking the expected behavior will be to set the starting trace context for child processes to the trace information modified above in the parent process. But I don't think that is what is happening here.

oof, you're right! And I was worried about my golang skills 😅

When using something like https://github.com/puppetlabs/rspec_honeycomb_formatter , it would be nice to connect the individual spans to the bigger picture of the buildevent steps and commands. This change uses the beeline's `PropagationContext` marshaling to pass on the current event to the called command. In the style of the rest of the HONEYCOMB_* environment variables and the similarily named `X-Honeycomb-Trace` HTTP header, this establishes `HONEYCOMB_TRACE` to pass the propagation context to other processes.

DavidS · 2020-12-09T09:44:25Z

Updated the ruby example to fix the ENV override as noticed by @robbkidd and commented/disabled the ARGV capturing as flagged by @MikeGoldsmith

@robbkidd

Thanks to @robbkidd for pointing out my folly in honeycombio/buildevents#74 (comment)

This is to align this with the conversation and changes in honeycombio/buildevents#74

Hand-build binary from honeycombio/buildevents#74

DavidS · 2020-12-10T20:22:21Z

I've tested a locally built buildevents binary in puppetlabs/puppetlabs-testing#324 and got a complete trace from GHA through rake, through the process rake starts, through the webservice call to our provisioning backend to the call to terraform applying and including our rspec tests running in that github action and calling into the newly provisioned machine:

pretty chuffed!

MikeGoldsmith

Hey @DavidS - thanks for the extra work on this. I think looks really good and I'm happy with the generic env var name that is used to pass trace context onto sub-processes to parse.

robbkidd

That build trace looks mighty purdy.

…ite_env * 'main' of github.com:honeycombio/buildevents: (40 commits) add apply labels workflow (#93) GitLab CI example (#90) Build an arm64 release (#91) * publish a draft release to GitHub (#87) Prepare for 0.5.0 release, add changelog (#86) Add support for overriding default fields (#76) fix providerAzurePipelines in main.go (#84) add bitbucket support (#85) Ensure PRs from forks can run CI checks (#82) added quiet option to cmd (#80) Use public CircleCI context for build secrets (#77) add codeowners with integration-team (#78) Encode spaces in dataset names for the URL (#73) Add beeline-style propagation of the cmd context to called commands (#74) we're on circle for this, no need to build on travis any longer (#69) make watch job more efficient (#70) drop vendor/ and upgrade go to 1.15.2 (#68) add GitHub actions as a supported CI provider, with data from env vars Correct a download link Add support for Azure Piplines and some initial env vars ... # Conflicts: # README.md # cmd_root.go # common.go # main.go

DavidS force-pushed the subcmd-propagation branch 3 times, most recently from e89a213 to f3a35b0 Compare November 19, 2020 17:27

DavidS marked this pull request as ready for review November 19, 2020 17:29

kvrhdn mentioned this pull request Nov 21, 2020

Provide a Honeycomb context to applications run inside the workflow honeycombio/gha-buildevents#6

Closed

MikeGoldsmith reviewed Dec 8, 2020

View reviewed changes

DavidS force-pushed the subcmd-propagation branch 3 times, most recently from 5467f4a to 50b0a2c Compare December 8, 2020 16:10

MikeGoldsmith approved these changes Dec 8, 2020

View reviewed changes

robbkidd reviewed Dec 8, 2020

View reviewed changes

DavidS force-pushed the subcmd-propagation branch from 50b0a2c to 7c0838d Compare December 9, 2020 09:43

DavidS added a commit to DavidS/rspec_honeycomb_formatter that referenced this pull request Dec 9, 2020

Always enforce passing through the trace context

a324ebf

Thanks to @robbkidd for pointing out my folly in honeycombio/buildevents#74 (comment)

DavidS added a commit to DavidS/rspec_honeycomb_formatter that referenced this pull request Dec 9, 2020

Update honeycomb trace ENV var to new name

3833425

This is to align this with the conversation and changes in honeycombio/buildevents#74

DavidS added a commit to DavidS/puppet_litmus that referenced this pull request Dec 9, 2020

Update honeycomb trace ENV var to new name

ad849b7

This is to align this with the conversation and changes in honeycombio/buildevents#74

This was referenced Dec 9, 2020

Update honeycomb trace ENV var to new name puppetlabs/rspec_honeycomb_formatter#12

Merged

Update honeycomb trace ENV var to new name puppetlabs/puppet_litmus#355

Merged

DavidS added a commit to DavidS/puppetlabs-testing that referenced this pull request Dec 10, 2020

Test buildevents with HONEYCOMB_TRACE integration

e301786

Hand-build binary from honeycombio/buildevents#74

DavidS mentioned this pull request Dec 10, 2020

DO NOT MERGE - Test buildevents with HONEYCOMB_TRACE integration puppetlabs/puppetlabs-testing#324

Closed

DavidS added a commit to DavidS/puppetlabs-testing that referenced this pull request Dec 10, 2020

Test buildevents with HONEYCOMB_TRACE integration

eb714a6

Hand-build binary from honeycombio/buildevents#74

DavidS added a commit to DavidS/puppetlabs-testing that referenced this pull request Dec 10, 2020

Test buildevents with HONEYCOMB_TRACE integration

8980c7d

Hand-build binary from honeycombio/buildevents#74

DavidS added a commit to DavidS/puppetlabs-testing that referenced this pull request Dec 10, 2020

Test buildevents with HONEYCOMB_TRACE integration

0ebe3a6

Hand-build binary from honeycombio/buildevents#74

DavidS added a commit to DavidS/puppetlabs-testing that referenced this pull request Dec 10, 2020

Test buildevents with HONEYCOMB_TRACE integration

90b98c0

Hand-build binary from honeycombio/buildevents#74

DavidS added a commit to DavidS/puppetlabs-testing that referenced this pull request Dec 10, 2020

Test buildevents with HONEYCOMB_TRACE integration

71484e0

Hand-build binary from honeycombio/buildevents#74

MikeGoldsmith approved these changes Dec 15, 2020

View reviewed changes

robbkidd approved these changes Dec 15, 2020

View reviewed changes

MikeGoldsmith merged commit 23401fd into honeycombio:main Dec 17, 2020

DavidS deleted the subcmd-propagation branch December 17, 2020 14:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add beeline-style propagation of the cmd context to called commands #74

Add beeline-style propagation of the cmd context to called commands #74

DavidS commented Nov 19, 2020 •

edited

Loading

DavidS commented Nov 19, 2020

MikeGoldsmith left a comment

MikeGoldsmith Dec 8, 2020

DavidS Dec 8, 2020

MikeGoldsmith Dec 8, 2020

DavidS Dec 8, 2020

martin308 Dec 8, 2020

DavidS commented Dec 8, 2020

MikeGoldsmith left a comment

MikeGoldsmith Dec 8, 2020 •

edited

Loading

DavidS Dec 8, 2020 •

edited

Loading

DavidS commented Dec 8, 2020

robbkidd Dec 8, 2020

DavidS Dec 9, 2020

DavidS commented Dec 9, 2020

DavidS commented Dec 10, 2020

MikeGoldsmith left a comment

robbkidd left a comment

Add beeline-style propagation of the cmd context to called commands #74

Add beeline-style propagation of the cmd context to called commands #74

Conversation

DavidS commented Nov 19, 2020 • edited Loading

DavidS commented Nov 19, 2020

MikeGoldsmith left a comment

Choose a reason for hiding this comment

MikeGoldsmith Dec 8, 2020

Choose a reason for hiding this comment

DavidS Dec 8, 2020

Choose a reason for hiding this comment

MikeGoldsmith Dec 8, 2020

Choose a reason for hiding this comment

DavidS Dec 8, 2020

Choose a reason for hiding this comment

martin308 Dec 8, 2020

Choose a reason for hiding this comment

DavidS commented Dec 8, 2020

MikeGoldsmith left a comment

Choose a reason for hiding this comment

MikeGoldsmith Dec 8, 2020 • edited Loading

Choose a reason for hiding this comment

DavidS Dec 8, 2020 • edited Loading

Choose a reason for hiding this comment

DavidS commented Dec 8, 2020

robbkidd Dec 8, 2020

Choose a reason for hiding this comment

DavidS Dec 9, 2020

Choose a reason for hiding this comment

DavidS commented Dec 9, 2020

DavidS commented Dec 10, 2020

MikeGoldsmith left a comment

Choose a reason for hiding this comment

robbkidd left a comment

Choose a reason for hiding this comment

DavidS commented Nov 19, 2020 •

edited

Loading

MikeGoldsmith Dec 8, 2020 •

edited

Loading

DavidS Dec 8, 2020 •

edited

Loading