Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(tools): fabric AIO image log access in CI #643

Closed
petermetz opened this issue Mar 10, 2021 · 0 comments · Fixed by #644
Closed

fix(tools): fabric AIO image log access in CI #643

petermetz opened this issue Mar 10, 2021 · 0 comments · Fixed by #644
Assignees
@petermetz
Labels
bug Something isn't working dependencies Pull requests that update a dependency file Developer_Experience Fabric

Comments

@petermetz
Copy link
Member

Describe the bug

If something goes wrong in the Fabric AIO container while my tests are running in the CI environment where I don't have direct shell access, then I'm out of luck when it comes to diagnosing the issue. My only recourse is to reproduce the issue on my local machine and then extract the container logs manually there. All this assuming of course that the issue reproduces locally to begin with!

To Reproduce

  1. Make a mistake in your tests, something that needs you to have access to the AIO container's logs to realize what the issue is/was.
  2. Push the code to the CI instead of waiting an hour to have the CI script run on your slow laptop.
  3. Observe that the CI has failed
  4. Observe that you don't know why and have no way of figuring out either (without reproducing the issue again elsewhere).

Expected behavior

Should be trivial to determine why a test failed even when the relevant information is logged inside a container.

Logs/Stack traces

N/A which is the point.

Screenshots

N/A

Cloud provider or hardware configuration:

GH Actions CI Environment

Operating system name, version, build:

Ubuntu 20.04 LTS

Hyperledger Cactus release version or commit (git rev-parse --short HEAD):

aa070ad

Hyperledger Cactus Plugins/Connectors Used

N/A

Additional context

Realized this while working on the Fabric chain code deployment endpoint.

cc: @takeutak @sfuji822 @hartm @jonathan-m-hamilton @AzaharaC @jordigiam @kikoncuo
@petermetz petermetz added bug Something isn't working dependencies Pull requests that update a dependency file Developer_Experience Fabric labels Mar 10, 2021
@petermetz petermetz self-assigned this Mar 10, 2021
petermetz added a commit to petermetz/cacti that referenced this issue Mar 10, 2021
@petermetz
fix(tools): fabric AIO image log access in CI hyperledger#643
3c94c0f 
This commit is meant to make it much easier to debug
issues with the Fabric 2.x AIO container image in the
future when new bugs (inevitably) arise.

Meaning that the logs of the child processess such as
the fabric network creator script or the docker daemon will
all be piped through to the logs of the container itself
instead of just being dumped into log files under
/var/log/... which has the upside of not having to
shell into the container when something goes wrong
and one wishes to debug what happened. Instead the
logs of the container will show all the information
right away.

In addition to the above change a flag was added to
the FabricTestLedger class' constructor options so
that one  can tell the class to dump the continer's
log stream straight onto it's own logger meaning
that now if you are debugging a test case you will
be able to see all the logs that are being output
straight in the logs of the test case itself.
This can be very spammy at times, hence the flag
that can be used to turn it on or off depending on
what is needed more.

One more small change that we made to help debugging:
The supervisord config file now specifies the loglevel
of supervisord as debug instead of info.

Fixes hyperledger#643

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
@petermetz petermetz mentioned this issue Mar 10, 2021
Merged
petermetz added a commit that referenced this issue Mar 11, 2021
@petermetz
fix(tools): fabric AIO image log access in CI #643
9d9f805 
This commit is meant to make it much easier to debug
issues with the Fabric 2.x AIO container image in the
future when new bugs (inevitably) arise.

Meaning that the logs of the child processess such as
the fabric network creator script or the docker daemon will
all be piped through to the logs of the container itself
instead of just being dumped into log files under
/var/log/... which has the upside of not having to
shell into the container when something goes wrong
and one wishes to debug what happened. Instead the
logs of the container will show all the information
right away.

In addition to the above change a flag was added to
the FabricTestLedger class' constructor options so
that one  can tell the class to dump the continer's
log stream straight onto it's own logger meaning
that now if you are debugging a test case you will
be able to see all the logs that are being output
straight in the logs of the test case itself.
This can be very spammy at times, hence the flag
that can be used to turn it on or off depending on
what is needed more.

One more small change that we made to help debugging:
The supervisord config file now specifies the loglevel
of supervisord as debug instead of info.

Fixes #643

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
AzaharaC pushed a commit to kikoncuo/cactus that referenced this issue Mar 12, 2021
@petermetz @AzaharaC
fix(tools): fabric AIO image log access in CI hyperledger#643
e8dffa3 
This commit is meant to make it much easier to debug
issues with the Fabric 2.x AIO container image in the
future when new bugs (inevitably) arise.

Meaning that the logs of the child processess such as
the fabric network creator script or the docker daemon will
all be piped through to the logs of the container itself
instead of just being dumped into log files under
/var/log/... which has the upside of not having to
shell into the container when something goes wrong
and one wishes to debug what happened. Instead the
logs of the container will show all the information
right away.

In addition to the above change a flag was added to
the FabricTestLedger class' constructor options so
that one  can tell the class to dump the continer's
log stream straight onto it's own logger meaning
that now if you are debugging a test case you will
be able to see all the logs that are being output
straight in the logs of the test case itself.
This can be very spammy at times, hence the flag
that can be used to turn it on or off depending on
what is needed more.

One more small change that we made to help debugging:
The supervisord config file now specifies the loglevel
of supervisord as debug instead of info.

Fixes hyperledger#643

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
jordigiam pushed a commit to kikoncuo/cactus that referenced this issue Apr 8, 2021
@petermetz
fix(tools): fabric AIO image log access in CI hyperledger#643
3a9f859 
This commit is meant to make it much easier to debug
issues with the Fabric 2.x AIO container image in the
future when new bugs (inevitably) arise.

Meaning that the logs of the child processess such as
the fabric network creator script or the docker daemon will
all be piped through to the logs of the container itself
instead of just being dumped into log files under
/var/log/... which has the upside of not having to
shell into the container when something goes wrong
and one wishes to debug what happened. Instead the
logs of the container will show all the information
right away.

In addition to the above change a flag was added to
the FabricTestLedger class' constructor options so
that one  can tell the class to dump the continer's
log stream straight onto it's own logger meaning
that now if you are debugging a test case you will
be able to see all the logs that are being output
straight in the logs of the test case itself.
This can be very spammy at times, hence the flag
that can be used to turn it on or off depending on
what is needed more.

One more small change that we made to help debugging:
The supervisord config file now specifies the loglevel
of supervisord as debug instead of info.

Fixes hyperledger#643

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@petermetz petermetz

Labels
bug Something isn't working dependencies Pull requests that update a dependency file Developer_Experience Fabric
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(tools): fabric AIO image log access in CI #643 petermetz/cacti
1 participant
@petermetz