Investigate performance of anonymous usage statistics requests

[sairam]

Describe the bug

Telemetry data is being sent to fishtownanalytics.sinter-collect.com on dbt run. The speed of dbt run improved to 17 sec from 111sec (~2min) for about when the domain was blackholed (pointing to 127.0.0.1)

Expected behavior

When disabled telemetry:

Finished running 100 view models, 2 table models in 16.80s.

When enabled telemetry:

Finished running 100 view models, 2 table models in 111.62s.

Each request to the telemetry server is taking about 1second. For 100 view models, its adding additional 100 seconds or 1 second per run.

The output of dbt --version:

installed version: 0.15.0
   latest version: 0.15.0

Up to date!

Is there a way to disable telemetry via commandline or via env variable?

[drewbanin]

hey @sairam - check out the docs for disabling anonymous usage statistics here: https://docs.getdbt.com/docs/configure-your-profile#section-usage-statistics

I'm surprised to see this is ballooning your dbt runtime - we coded this such that a single failed event request would prevent future requests from being made. Let me take a deeper look here - the behavior you're seeing is not right.

[sairam]

Thanks for the link, I initially searched for telemetry and analytics, but could not find the right config.

config:
  send_anonymous_usage_stats: False

@drewbanin , feel free to close this issue.

[sairam]

I wanted to share that this was run with 1 thread.

Looks like a request to a remote server is happening after each view/action is done which explains close to 100second delay. Changing the location to edge endpoint may help instead of us-east-1.

[drewbanin]

Ok - that's not right - 1 second is far, far too long for these individual requests. I'll leave this issue open to investigate adding edge endpoints, batching the requests, or making the requests async. They actually used to be async but we had issues with the thread not being cleaned up and it causing dbt to hang. Thanks again for bringing this one to our attention.

[sairam]

would it possible to have a dedicated thread just for sending out the analytics?

[drewbanin]

Yep, that's totally possible. Alternatively, we could try batching these events and sending more of them at once. The payloads are pretty tiny (timestamps, the command that was run, etc) - the really slow thing here is just the round-trip to us-east-1 I think.

[beckjake]

Fixed in #2089