Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ScalableSingleBinary operational run mode #1004

Merged
merged 15 commits into from
Oct 12, 2021

Conversation

zalegrala
Copy link
Contributor

@zalegrala zalegrala commented Oct 1, 2021

What this PR does:
Add a new module ScalableSingleBinary that will run all services and also make use of memberlist for cluster awareness.

Which issue(s) this PR fixes:
Fixes #86

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

cmd/tempo/app/modules.go Outdated Show resolved Hide resolved
cmd/tempo/app/modules.go Outdated Show resolved Hide resolved
cmd/tempo/app/modules.go Outdated Show resolved Hide resolved
cmd/tempo/main.go Outdated Show resolved Hide resolved
@zalegrala zalegrala force-pushed the haOperationalRunMode branch 4 times, most recently from 9fdb7f0 to 7a57a98 Compare October 4, 2021 17:21
@zalegrala zalegrala marked this pull request as ready for review October 4, 2021 19:16
@zalegrala zalegrala changed the title Ha operational run mode Add ScalableSingleBinary operational run mode Oct 4, 2021
@zalegrala
Copy link
Contributor Author

What other documentation do we want to include here? We don't currently say much about the target=all, so I'm wondering how much needs to be said to differentiate.

@zalegrala
Copy link
Contributor Author

I've included some brief docs about our run modes.

Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far nice work. Have a few cleanup requests on the doc and then let's add a docker-compose example that uses this new mode so people can see a simple way to set it up.

docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
integration/e2e/e2e_test.go Show resolved Hide resolved
integration/microservices/README.md Show resolved Hide resolved
@zalegrala
Copy link
Contributor Author

Oh whoops, with all the rebasing I failed to add the docker-compose example. I'll get that updated and clean up the docs per your suggestions. Thanks for the review.

@mdisibio
Copy link
Contributor

mdisibio commented Oct 6, 2021

Late to the conversation, but I would like to propose another module naming change. With the addition of a scaleable single binary mode, I'm finding the naming of the "all" module somewhat misleading. It runs all components but differently. The new scaleable mode is really closer to "all", as in it runs all components without changes.

Could we do something like the following? Note it doesn't change the commandline for --target=all

// The various modules that make up tempo.
const (
	SingleBinary         string = "all"
	ScalableSingleBinary string = "scalable-single-binary"
)

@zalegrala
Copy link
Contributor Author

I've included the change you mention @mdisibio.

@zalegrala
Copy link
Contributor Author

Hold for @KMiller-Grafana review.

docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
@zalegrala
Copy link
Contributor Author

The docs have been updated from @KMiller-Grafana, and I also dropped the configuration section for this mode. Hows that look now?

docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Show resolved Hide resolved
example/docker-compose/scalable-single-binary/readme.md Outdated Show resolved Hide resolved
example/docker-compose/scalable-single-binary/readme.md Outdated Show resolved Hide resolved
example/docker-compose/scalable-single-binary/readme.md Outdated Show resolved Hide resolved
example/docker-compose/scalable-single-binary/readme.md Outdated Show resolved Hide resolved
example/docker-compose/scalable-single-binary/readme.md Outdated Show resolved Hide resolved
integration/microservices/README.md Outdated Show resolved Hide resolved
integration/microservices/README.md Outdated Show resolved Hide resolved
integration/microservices/README.md Outdated Show resolved Hide resolved
@zalegrala
Copy link
Contributor Author

Thanks for the review and updates @KMiller-Grafana.

docs/tempo/website/operations/deployment.md Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Outdated Show resolved Hide resolved
docs/tempo/website/operations/deployment.md Show resolved Hide resolved
@zalegrala zalegrala force-pushed the haOperationalRunMode branch 2 times, most recently from c372808 to 6916a70 Compare October 12, 2021 13:59
@joe-elliott joe-elliott merged commit 458b8e8 into grafana:main Oct 12, 2021
A single binary mode deployment runs all top-level components in a single
process, forming an instance of Tempo. The single binary mode is the simplest
to deploy but cannot but can not horizontally scale. Refer to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
to deploy but cannot but can not horizontally scale. Refer to
to deploy, but cannot but can not horizontally scale out by increasing the quantity of components. Refer to

Comment on lines +68 to +70
that all components are deployed in one binary but it is capable of
horizontally scaling. This mode offers some flexibility of scaling without the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rewrite misses telling readers what scalable single binary mode is. Please consider this suggestion.

Suggested change
A scalable single binary deployment is similar to the single binary mode in
that all components are deployed in one binary but it is capable of
horizontally scaling. This mode offers some flexibility of scaling without the
A scalable single binary deployment is similar to the single binary mode in
that all components are deployed within one binary. Horizontally scale out is achieved by instantiating more than one single binary. This mode offers some flexibility of scaling without the configuration


In microservices mode, components are deployed in distinct processes. Scaling
is per component which allows for greater flexibility in scaling and more
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
is per component which allows for greater flexibility in scaling and more
is per component, which allows for greater flexibility in scaling and more

Comment on lines +3 to +6
This example aims to make it easier to measure and analyze tempo performance in
micro-services mode. There are already many examples for running tempo under
load, but they use the single-binary approach and are not representative of
what is occurring in larger installations. Here tempo is run with separate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This example aims to make it easier to measure and analyze tempo performance in
micro-services mode. There are already many examples for running tempo under
load, but they use the single-binary approach and are not representative of
what is occurring in larger installations. Here tempo is run with separate
This example aims to make it easier to measure and analyze Tempo performance in
micro-services mode. There are already many examples for running Tempo under
load, but they use the single-binary approach and are not representative of
what is occurring in larger installations. Here Tempo is run with separate

Comment on lines +26 to +28
This example is expected to be used in conjunction with tempo development in a
rapid feedback loop. It is assumed you have a working go installation and a
copy of tempo already cloned somewhere.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This example is expected to be used in conjunction with tempo development in a
rapid feedback loop. It is assumed you have a working go installation and a
copy of tempo already cloned somewhere.
This example is expected to be used in conjunction with Tempo development in a
rapid feedback loop. It is assumed you have a working Go installation and a
copy of the Tempo repository already cloned somewhere.

@@ -38,4 +47,5 @@ docker-compose up -d --scale synthetic-load-generator=4
```

# Key Metrics
As tempo is designed to be very horizontally scaleable, the key metrics are _per volume unit_, i.e. spans / s / cpu core.

As tempo is designed to be very horizontally scalable, the key metrics are _per volume unit_, i.e. spans / s / cpu core.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
As tempo is designed to be very horizontally scalable, the key metrics are _per volume unit_, i.e. spans / s / cpu core.
As Tempo is designed to be horizontally scalable, the key metrics are _per volume unit_, i.e. spans / s / cpu core.

zalegrala added a commit to zalegrala/tempo that referenced this pull request Oct 12, 2021
zalegrala added a commit to zalegrala/tempo that referenced this pull request Oct 13, 2021
zalegrala added a commit to zalegrala/tempo that referenced this pull request Oct 13, 2021
zalegrala added a commit to zalegrala/tempo that referenced this pull request Oct 13, 2021
annanay25 pushed a commit that referenced this pull request Oct 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add HA Scaleable Mode
4 participants