Skip to content

Commit

Permalink
Merge Sumo Logic updates to OTC
Browse files Browse the repository at this point in the history
Merge Sumo tail-sampling extensions
Update source processor to new API
Fix imports order
Update deps for OTC v0.14.0
Expose tailsamplingprocessor metrics
Config update (#289)
CircleCI configuration update and linter fixes
Tail sampling processor - update to v0.16
Source processor - update to v0.16
Tailsampling processor order fix
Introduce cascading filter processor (#359)
Introduce cascading filter processor
Switch to AWS ECR (#409)
Switch to AWS ECR
Investigate issue with image tagging (#410)
Move envs into $BASH_ENV
AWS CLI orb
Update README.md
Clarify which span attributes are updated/added and `probablistic_filtering_ratio` behavior
Provide invert (like grep -v) match capability (#509)
Bump OTC-Sumo version
Add Sumo examples (#681)
Update README.md
Update base dependency to v0.19.0-sumo
Remove uneccesary return values per linter
Switch master -> main
Address linter errors in sourceprocessor
Use cloud.namespace on AWS resourcedetection
Update ec2.go
TRACING-1684 | Change cloud.namespace tag from "ec2" to "aws/ec2".
TRACING-1684 | Test fix
Switch examples to use OTLP/HTTP (#864)
Bump SumoLogic OTC base version
Fix cascadingfilterprocessor unit test
Introduce telegrafreceiver (#935)
Increase testbed memory limits
AWS OTel Collector templates (#788)
Add AWS OTel Collector config templates
Update core version to v0.22.0
AWS Distro for Opentelemetry collector configuration file (#983)
Add AWS Distro Collector config
Change endpoint, add insecure flag
Update readme
Update Telegraf for changes in core v0.22.0
Rebase on v0.24.0 upstream changes
Add opentelemetry-collector-builder with Makefile and Github Action to build on PR (#1300)
Add telegrafreceiver to opentelemetry-collector-builder config
Add tracing-tests into pipeline (#1302)
Install fluentbit (#1312)
Add publish-check for all tags and branches
Fix CI to actually produce artifacts for tags
Update config.yml (#1412)
Update aws-otel-config-file.yaml (#1537)
Add resourcedetection
Add additional receivers
Sumo Logic Syslog Processor (#1313)
add sumologicsyslogprocessor
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Co-authored-by: Patryk Małek <69143962+pmalek-sumo@users.noreply.github.com>
Update k8sprocessor with recent changes from sumologic-otel-collector
  • Loading branch information
pmm-sumo committed Oct 1, 2021
1 parent ba83ef0 commit 2c6c0c2
Show file tree
Hide file tree
Showing 107 changed files with 10,533 additions and 1,418 deletions.
389 changes: 254 additions & 135 deletions .circleci/config.yml

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions .github/workflows/opentelemetry-collector-builder.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: OpenTelemetry Collector Builder

on:
pull_request:
branches:
- main

jobs:
# otelcolbuilder_post_go1_16:
# runs-on: ubuntu-20.04
# strategy:
# matrix:
# go: [ '1.16' ]
# steps:
# - uses: actions/checkout@v2
# - name: Setup go
# uses: actions/setup-go@v2
# with:
# go-version: ${{ matrix.go }}
# - name: Print go version
# run: go version
# - name: Build OpenTelemetry distro
# working-directory: ./otelcolbuilder/
# run: |
# go install github.com/open-telemetry/opentelemetry-collector-builder@v0.24.0
# make build

# Just build on 1.15 for now because of a weird issue:
# https://github.com/actions/setup-go/issues/107
otelcolbuilder_pre_go1_16:
runs-on: ubuntu-20.04
strategy:
matrix:
go: [ '1.15' ]
steps:
- uses: actions/checkout@v2
- name: Setup go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Print go version
run: go version
- name: Print go env
run: go env
- name: Build OpenTelemetry distro
working-directory: ./otelcolbuilder/
run: |
make install-prego1.16
make build
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ integration-coverage.txt
coverage.html
integration-coverage.html

# vagrant
.vagrant
ubuntu-bionic-18.04-cloudimg-console.log

# Wix
*.wixobj
*.wixpdb
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ run:
docker-component: check-component
GOOS=linux GOARCH=amd64 $(MAKE) $(COMPONENT)
cp ./bin/$(COMPONENT)_linux_amd64 ./cmd/$(COMPONENT)/$(COMPONENT)
docker build -t $(COMPONENT) ./cmd/$(COMPONENT)/
docker build -t $(COMPONENT) -f ./cmd/$(COMPONENT)/Dockerfile .
rm ./cmd/$(COMPONENT)/$(COMPONENT)

.PHONY: check-component
Expand Down
124 changes: 21 additions & 103 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,110 +1,28 @@
---
# SumoLogic / OpenTelemetry Collector Contrib
This is a repository for OpenTelemetry Collector Contrib with additional Sumo Logic extensions. It is based
on the [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) and
[SumoLogic flavor](https://github.com/SumoLogic/opentelemetry-collector) of
[core distribution of the Collector](https://github.com/open-telemetry/opentelemetry-collector).

<p align="center">
<strong>
<a href="https://opentelemetry.io/docs/collector/getting-started/">Getting Started</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/CONTRIBUTING.md">Getting Involved</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://gitter.im/open-telemetry/opentelemetry-service">Getting In Touch</a>
</strong>
</p>

<p align="center">
<a href="https://goreportcard.com/report/github.com/open-telemetry/opentelemetry-collector-contrib">
<img alt="Go Report Card" src="https://goreportcard.com/badge/github.com/open-telemetry/opentelemetry-collector-contrib?style=for-the-badge">
</a>
<a href="https://circleci.com/gh/open-telemetry/opentelemetry-collector-contrib">
<img alt="Build Status" src="https://img.shields.io/circleci/build/github/open-telemetry/opentelemetry-collector-contrib?style=for-the-badge">
</a>
<a href="https://codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/branch/main/">
<img alt="Codecov Status" src="https://img.shields.io/codecov/c/github/open-telemetry/opentelemetry-collector-contrib?style=for-the-badge">
</a>
<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/releases">
<img alt="GitHub release (latest by date including pre-releases)" src="https://img.shields.io/github/v/release/open-telemetry/opentelemetry-collector-contrib?include_prereleases&style=for-the-badge">
</a>
<img alt="Beta" src="https://img.shields.io/badge/status-beta-informational?style=for-the-badge&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAAAXNSR0IArs4c6QAAAIRlWElmTU0AKgAAAAgABQESAAMAAAABAAEAAAEaAAUAAAABAAAASgEbAAUAAAABAAAAUgEoAAMAAAABAAIAAIdpAAQAAAABAAAAWgAAAAAAAACQAAAAAQAAAJAAAAABAAOgAQADAAAAAQABAACgAgAEAAAAAQAAABigAwAEAAAAAQAAABgAAAAA8A2UOAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAAVlpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KTMInWQAABK5JREFUSA2dVm1sFEUYfmd2b/f2Pkqghn5eEQWKrRgjpkYgpoRCLC0oxV5apAiGUDEpJvwxEQ2raWPU+Kf8INU/RtEedwTCR9tYPloxGNJYTTQUwYqJ1aNpaLH3sXu3t7vjvFevpSqt7eSyM+/czvM8877PzB3APBoLgoDLsNePF56LBwqa07EKlDGg84CcWsI4CEbhNnDpAd951lXE2NkiNknCCTLv4HtzZuvPm1C/IKv4oDNXqNDHragety2XVzjECZsJARuBMyRzJrh1O0gQwLXuxofxsPSj4hG8fMLQo7bl9JJD8XZfC1E5yWFOMtd07dvX5kDwg6+2++Chq8txHGtfPoAp0gOFmhYoNFkHjn2TNUmrwRdna7W1QSkU8hvbGk4uThLrapaiLA2E6QY4u/lS9ItHfvJkxYsTMVtnAJLipYIWtVrcdX+8+b8IVnPl/R81prbuPZ1jpYw+0aEUGSkdFsgyBIaFTXCm6nyaxMtJ4n+TeDhJzGqZtQZcuYDgqDwDbqb0JF9oRpIG1Oea3bC1Y6N3x/WV8Zh83emhCs++hlaghDw+8w5UlYKq2lU7Pl8IkvS9KDqXmKmEwdMppVPKwGSEilmyAwJhRwWcq7wYC6z4wZ1rrEoMWxecdOjZWXeAQClBcYDN3NwVwD9pGwqUSyQgclcmxpNJqCuwLmDh3WtvPqXdlt+6Oz70HPGDNSNBee/EOen+rGbEFqDENBPDbtdCp0ukPANmzO0QQJYUpyS5IJJI3Hqt4maS+EB3199ozm8EDU/6fVNU2dQpdx3ZnKzeFXyaUTiasEV/gZMzJMjr3Z+WvAdQ+hs/zw9savimxUntDSaBdZ2f+Idbm1rlNY8esFffBit9HtK5/MejsrJVxikOXlb1Ukir2X+Rbdkd1KG2Ixfn2Ql4JRmELnYK9mEM8G36fAA3xEQ89fxXihC8q+sAKi9jhHxNqagY2hiaYgRCm0f0QP7H4Fp11LSXiuBY2aYFlh0DeDIVVFUJQn5rCnpiNI2gvLxHnASn9DIVHJJlm5rXvQAGEo4zvKq2w5G1NxENN7jrft1oxMdekETjxdH2Z3x+VTVYsPb+O0C/9/auN6v2hNZw5b2UOmSbG5/rkC3LBA+1PdxFxORjxpQ81GcxKc+ybVjEBvUJvaGJ7p7n5A5KSwe4AzkasA+crmzFtowoIVTiLjANm8GDsrWW35ScI3JY8Urv83tnkF8JR0yLvEt2hO/0qNyy3Jb3YKeHeHeLeOuVLRpNF+pkf85OW7/zJxWdXsbsKBUk2TC0BCPwMq5Q/CPvaJFkNS/1l1qUPe+uH3oD59erYGI/Y4sce6KaXYElAIOLt+0O3t2+/xJDF1XvOlWGC1W1B8VMszbGfOvT5qaRRAIFK3BCO164nZ0uYLH2YjNN8thXS2v2BK9gTfD7jHVxzHr4roOlEvYYz9QIz+Vl/sLDXInsctFsXjqIRnO2ZO387lxmIboLDZCJ59KLFliNIgh9ipt6tLg9SihpRPDO1ia5byw7de1aCQmF5geOQtK509rzfdwxaKOIq+73AvwCC5/5fcV4vo3+3LpMdtWHh0ywsJC/ZGoCb8/9D8F/ifgLLl8S8QWfU8cAAAAASUVORK5CYII=">
</p>
## Docker Images
Docker images for all releases are published at https://hub.docker.com/r/sumologic/opentelemetry-collector

<p align="center">
<strong>
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/vision.md">Vision</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/design.md">Design</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/monitoring.md">Monitoring</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/performance.md">Performance</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security.md">Security</a>
&nbsp;&nbsp;&bull;&nbsp;&nbsp;
<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/roadmap.md">Roadmap</a>
</strong>
</p>
### Building docker locally

---
```
docker build -f cmd/otelcontribcol/Dockerfile -t otelcontribcol .
```

# OpenTelemetry Collector Contrib
## Differences from the core release

This is a repository for OpenTelemetry Collector contributions that are not
part of the [core
distribution](https://github.com/open-telemetry/opentelemetry-collector) of the
Collector. Typically, these contributions are vendor specific
receivers/exporters and/or components that are only useful to a relatively
small number of users.
SumoLogic version of OpenTelemetry Collector introduces a number of additions over the plain version:

> Please note that this repository and its releases are a superset of the core repository.
## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

Triagers ([@open-telemetry/collector-contrib-triagers](https://github.com/orgs/open-telemetry/teams/collector-contrib-triagers))
- [Alolita Sharma](https://github.com/alolita), AWS
- [Punya Biswal](https://github.com/punya), Google
- [Steve Flanders](https://github.com/flands), Splunk

Approvers ([@open-telemetry/collector-contrib-approvers](https://github.com/orgs/open-telemetry/teams/collector-contrib-approvers)):

- [Alex Boten](https://github.com/codeboten), Lightstep
- [Anthony Mirabella](https://github.com/Aneurysm9), AWS
- [Anuraag Agrawal](https://github.com/anuraaga), AWS
- [Daniel Jaglowski](https://github.com/djaglowski), observIQ
- [Dmitrii Anoshin](https://github.com/dmitryax), Splunk
- [Juraci Paixão Kröhling](https://github.com/jpkrohling), Red Hat
- [Kevin Brockhoff](https://github.com/kbrockhoff), Daugherty Business Solutions
- [Pablo Baeyens](https://github.com/mx-psi), DataDog
- [Owais Lone](https://github.com/owais), Splunk

Maintainers ([@open-telemetry/collector-contrib-maintainer](https://github.com/orgs/open-telemetry/teams/collector-contrib-maintainer)):

- [Bogdan Drutu](https://github.com/BogdanDrutu), Splunk
- [Tigran Najaryan](https://github.com/tigrannajaryan), Splunk

Learn more about roles in the [community repository](https://github.com/open-telemetry/community/blob/main/community-membership.md).

## PRs and Reviews

When creating a PR please following the process [described
here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CONTRIBUTING.md#how-to-structure-prs-to-get-expedient-reviews).

News PRs will be automatically associated with the reviewers based on
[CODEOWNERS](.github/CODEOWNERS). PRs will be also automatically assigned to one of the
maintainers or approvers for facilitation.

The facilitator is responsible for helping the PR author and reviewers to make progress
or if progress cannot be made for closing the PR.

If the reviewers do not have approval rights the facilitator is also responsible
for the official approval that is required for the PR to be merged and if the facilitator
is a maintainer they are responsible for merging the PR as well.

The facilitator is not required to perform a thorough review, but they are encouraged to
enforce Collector best practices and consistency across the codebase and component
behavior. The facilitators will typically rely on codeowner's detailed review of the code
when making the final approval decision.

We recommend maintainers and approvers to keep an eye on the
[project board](https://github.com/orgs/open-telemetry/projects/3). All newly created
PRs are automatically added to this board. (If you don't see the PR on the board you
may need to add it manually by setting the Project field in the PR view).
* Extensions to [k8sprocessor](https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/master/processor/k8sprocessor)
which include more tags being collected and field extraction enhancements
* A [sourceprocessor](https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/master/processor/sourceprocessor) that
adds additional tags (mostly relevant to K8s environment) and provides some data filtering rules
* [Cascading filter processor](https://github.com/pmm-sumo/opentelemetry-collector-contrib/tree/remote-conf-poc/processor/cascadingfilterprocessor)
extensions, which include *cascading* policy with two-pass rules for determining span budget for each of the defined rules
* Support for spans in [filtering processor](https://github.com/SumoLogic/opentelemetry-collector/tree/master/processor/filterprocessor)
* Additional release schedule, which allows to quickly introduce bugfixes and extensions
17 changes: 17 additions & 0 deletions Vagrantfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure('2') do |config|
config.vm.box = 'ubuntu/bionic64'
config.vm.box_check_update = false
config.vm.host_name = 'opentelemetry-collector-contrib'
config.vm.network :private_network, ip: "192.168.33.33"

config.vm.provider 'virtualbox' do |vb|
vb.gui = false
vb.memory = 4096
vb.name = 'opentelemetry-collector-contrib'
end
config.vm.provision 'file', source: 'vagrant', destination: 'vagrant'
config.vm.provision 'shell', path: 'vagrant/provision.sh'
end
71 changes: 71 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Examples
## Kubernetes configuration

### Helm chart values template
[kubernetes/custom-values.yaml](./kubernetes/custom-values.yaml) contains
an example template for Sumologic Kubernetes Collection Helm chart, which
installs OpenTelemetry Collector in Agent and Gateway configuration, as described
in the [documentation](https://help.sumologic.com/Traces/Getting_Started_with_Transaction_Tracing/Set_up_traces_collection_for_Kubernetes_environments).

After filling the template values, you can install it following
[Sumologic Kubernetes Collection installation instructions](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/release-v2.0/deploy/docs/Installation_with_Helm.md)
For example, by running following commands:
```shell
helm repo add sumologic https://sumologic.github.io/sumologic-kubernetes-collection
kubectl create namespace sumologic
helm upgrade --install my-release -n sumologic sumologic/sumologic -f custom-values.yaml
```

### Helm chart values template with cascading filter enabled

Additionally, [kubernetes/custom-values-cascading-filter.yaml](./kubernetes/custom-values-cascading-filter.yaml)
includes an alternative example template that enables cascading filter,
as described in [trace filtering documentation](https://help.sumologic.com/Traces/Getting_Started_with_Transaction_Tracing/What_if_I_don't_want_to_send_all_the_tracing_data_to_Sumo_Logic%3F).
Note that cascading filter is currently supported only for single-instance
OpenTelemetry Collector deployments.

## Non-kubernetes configuration

### Agent configuration (should be run on each host/node)
[non-kubernetes/agent-configuration-template.yaml](non-kubernetes/agent-configuration-template.yaml) contains
an OpenTelemetry Collector YAML file which includes configuration
for OpenTelemetry Collector running in Agent mode. It should be
deployed on each host/node within the system.

### Gateway configuration (should be run per each cluster/data-center/etc.)
[non-kubernetes/gateway-configuration-template.yaml](non-kubernetes/gateway-configuration-template.yaml) contains
an OpenTelemetry Collector YAML file which includes configuration
for OpenTelemetry Collector running in Gateway mode.

Additionally, for [non-kubernetes/gateway-configuration-template-with-cascading-filter.yaml](non-kubernetes/gateway-configuration-template-with-cascading-filter.yaml)
the configuration also includes cascading filter config,
which is described in more detail in [trace filtering documentation](https://help.sumologic.com/Traces/Getting_Started_with_Transaction_Tracing/What_if_I_don't_want_to_send_all_the_tracing_data_to_Sumo_Logic%3F).

Please refer to [relevant documentation](https://help.sumologic.com/Traces/Getting_Started_with_Transaction_Tracing/Set_up_traces_collection_for_other_environments)
for more details.

### AWS OTel Collector configuration file
[non-kubernetes/aws-otel-config.yaml](non-kubernetes/aws-otel-config.yaml) contains
an AWS OpenTelemetry Collector distrubtion YAML file which includes configuration
for OpenTelemetry Collector. Should be deployed on the AWS environments.

### AWS OTel Collector for ECS in EC2 mode template
[non-kubernetes/aws-otel-ecs-ec2-deployment.yaml](non-kubernetes/aws-otel-ecs-ec2-deployment.yaml) contains
an AWS OpenTelemetry Collector distribution YAML file which includes
CloudFormation template. It should be deployed on the AWS ECS EC2
environment.

### AWS OTel Collector for ECS in Fargate mode template
[non-kubernetes/aws-otel-ecs-fargate-deployment.yaml](non-kubernetes/aws-otel-ecs-fargate-deployment.yaml) contains
an AWS OpenTelemetry Collector distribution YAML file which includes
CloudFormation template. It should be deployed on the AWS ECS Fargate
environment.

### AWS OTel Collector for EC2 deployment template
[non-kubernetes/aws-otel-ec2-deployment.yaml](non-kubernetes/aws-otel-ec2-deployment.yaml) contains
an AWS OpenTelemetry Collector distribution YAML file which includes
CloudFormation template. It should be deployed on the AWS EC2.

### AWS Distro for OpenTelemetry configuration
[aws_lambda/aws-distro-collector-lambda-layer-config.yaml](aws_lambda/aws-distro-collector-lambda-layer-config.yaml) contains
an [AWS Distro for Opentelemetry Collector](https://github.com/aws-observability/aws-otel-lambda/tree/main/extensions/aoc-extension) YAML file which includes configuration for collector installed in a Lambda Layer. Collector requires *SUMOLOGIC_HTTP_TRACES_ENDPOINT_URL* environment variable to be set.
16 changes: 16 additions & 0 deletions examples/aws_lambda/aws-distro-collector-lambda-layer-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
receivers:
otlp:
protocols:
grpc:
endpoint: "localhost:55680"

exporters:
otlphttp:
traces_endpoint: $SUMOLOGIC_HTTP_TRACES_ENDPOINT_URL
insecure: true

service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp]
Loading

0 comments on commit 2c6c0c2

Please sign in to comment.