Flakiness in CWL log filtering integration tests

[SaxyPandaBear]

Describe the bug
CloudWatch logs agent-side filtering integration tests are failing consistently.

https://github.com/aws/amazon-cloudwatch-agent/runs/6816733448?check_suite_focus=true

[SaxyPandaBear]

Looking at it further, it doesn't fail consistently. Rerunning it does show it can pass.

Failed attempt:

aws_instance.integration-test (remote-exec):=== RUN   TestWriteLogsToCloudWatch/Client-side_log_filtering
aws_instance.integration-test (remote-exec):2022/06/09 23:22:46 Copy File resources/config_log_filter.json to /opt/aws/amazon-cloudwatch-agent/bin/config.json
aws_instance.integration-test (remote-exec):2022/06/09 23:22:46 File resources/config_log_filter.json abs path /home/ec2-user/amazon-cloudwatch-agent/integration/test/cloudwatchlogs/resources/config_log_filter.json
aws_instance.integration-test (remote-exec):2022/06/09 23:22:46 File : resources/config_log_filter.json copied to : /opt/aws/amazon-cloudwatch-agent/bin/config.json
aws_instance.integration-test (remote-exec):2022/06/09 23:22:47 Agent has started
aws_instance.integration-test: Still creating... [4m0s elapsed]
aws_instance.integration-test: Still creating... [4m10s elapsed]
aws_instance.integration-test (remote-exec):2022/06/09 23:23:07 Writing 200 lines to /tmp/test.log
aws_instance.integration-test: Still creating... [4m20s elapsed]
aws_instance.integration-test: Still creating... [4m30s elapsed]
aws_instance.integration-test (remote-exec):2022/06/09 23:23:28 Agent is stopped
aws_instance.integration-test (remote-exec):2022/06/09 23:23:28 Checking i-0c5a4dc7a184febda/i-0c5a4dc7a184febda since 2022-06-09T23:22:46Z for 100 expected logs
aws_instance.integration-test (remote-exec):2022/06/09 23:23:28 Done paginating log events for i-0c5a4dc7a184febda/i-0c5a4dc7a184febda and found 0 logs
aws_instance.integration-test (remote-exec):    cwl_util.go:76:
aws_instance.integration-test (remote-exec):        	Error Trace:	cwl_util.go:76
aws_instance.integration-test (remote-exec):        	            				publish_logs_test.go:79
aws_instance.integration-test (remote-exec):        	Error:      	Not equal:
aws_instance.integration-test (remote-exec):        	            	expected: 100
aws_instance.integration-test (remote-exec):        	            	actual  : 0
aws_instance.integration-test (remote-exec):        	Test:       	TestWriteLogsToCloudWatch/Client-side_log_filtering
aws_instance.integration-test (remote-exec):--- FAIL: TestWriteLogsToCloudWatch (83.02s)

Successful attempt:

aws_instance.integration-test (remote-exec):=== RUN   TestWriteLogsToCloudWatch/Client-side_log_filtering
aws_instance.integration-test (remote-exec):2022/06/09 18:52:32 Copy File resources/config_log_filter.json to /opt/aws/amazon-cloudwatch-agent/bin/config.json
aws_instance.integration-test (remote-exec):2022/06/09 18:52:32 File resources/config_log_filter.json abs path /home/ec2-user/amazon-cloudwatch-agent/integration/test/cloudwatchlogs/resources/config_log_filter.json
aws_instance.integration-test (remote-exec):2022/06/09 18:52:32 File : resources/config_log_filter.json copied to : /opt/aws/amazon-cloudwatch-agent/bin/config.json
aws_instance.integration-test (remote-exec):2022/06/09 18:52:32 Agent has started
aws_instance.integration-test: Still creating... [4m40s elapsed]
aws_instance.integration-test: Still creating... [4m50s elapsed]
aws_instance.integration-test (remote-exec):2022/06/09 18:52:52 Writing 200 lines to /tmp/test.log
aws_instance.integration-test: Still creating... [5m0s elapsed]
aws_instance.integration-test: Still creating... [5m10s elapsed]
aws_instance.integration-test (remote-exec):2022/06/09 18:53:13 Agent is stopped
aws_instance.integration-test (remote-exec):2022/06/09 18:53:13 Checking i-0ae6c93b5ed9a126f/i-0ae6c93b5ed9a126f since 2022-06-09T18:52:32Z for 100 expected logs
aws_instance.integration-test (remote-exec):2022/06/09 18:53:14 Done paginating log events for i-0ae6c93b5ed9a126f/i-0ae6c93b5ed9a126f and found 100 logs

My best guess is there needs to be some tweaking with the timing for publishing logs, stopping the agent, and checking the backend. I can play around with that after I fix my fork