-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GC overhead limit exceeded #1032
Comments
The error looks to be occuring in the minion launched to gather block coverage, so gc parameters need to be passed to the child process. This can be done using pitest's Unless this module is excessively large, it seems likely that there is some sort of memory leak in the tests when run via pitest. I can see powermock is in use, which is a likely culprit. What else is in use? Also, |
Thanks! As you can see I already use jvmArgs. Is there anything am I missing?
Junit tests. Is there anyway to narrow down which test(s) is(are) the culprit? This is exactly the output of the error on the second command:
|
Sorry, I completely misread the command line. That looks correct to me for passing params to the minions, however, now we have the full stack trace, the GC overhead limit exceeded error looks to be being thrown in the main process. So, working around this would require tweaking memory settings in both the minion and the main process. Which still leaves the question of what is going on. Either
Turning on verbose logging might help identify a problematic test (I usually pipe the whole output to a file and wade through). Disabling any test that use powermock / anything the manipulates bytecode might also help narrow down the issue to a list of culprets. A thread dump of the main process would also shed some light. |
How do I do that? Setting MAVEN_OPTS maybe? If I try to set up:
I get an error complaining about the heap size. I remove the -Xmx1024M flag I get an error regarding the reservedcochecachesize
How do I disable any particular test? with excludedTestClasses?
Which command do you recommend me for that? |
This is because of the commas. Try:
|
I still get
|
Then I wonder whether there are any non-visible/non-space characters in there. For example, with both JDK 11 and 17 I get:
Observe how my error message says |
Many thanks, apparently on Windows you can´t use quotes with MAVEN_OPTS, so you have to use:
without quotes After excluding a powermock class it took more than 4 hours but no luck:
|
I have to say I am using JDK 8 and Maven 3.8.4 |
@mgonzcast Can you give an indication of how large this module is? How long do the tests take to run wihtout pitest? Roughly how many thousand lines of code is it? |
Thanks for being so responsive. The module has 17k lines of code and 1.6K of junit tests. I would say the tests take around 20-30 minutes to run. |
Do you have any pitest configuration in the pom? |
No, nothing in pom about pitest. I have to say this is part of three module project but no pom has any reference about pitest. The other two modules are quite smaller (2.7k lines of code and 1.2k junit tests) and run pitest without any issue. |
I have checked and tests take around 7 minutes to run, sorry. I am going to start excluding classes from pitest so I can narrow down where the issue is:
Do you think is the best way? |
You'll need to use |
7 minutes to 4 hours is quite a leap. Generally the coverage system adds about 10% overhead. The slow down is almost certainly due to the excessive memory consumption. |
I think this issue has been previously reported here Which is fortunate as I am able to replicate it using the assertj project as an input. The issue is not as severe, but memory consumption is much higher than I'd expect. I've identified something in the coverage code which looks (on the face of it) to be incredibly stupid, and should significantly reduce memory consumption if fixed. I'm not sure if this is the actual root cause problem here, or just an exasperating factor. |
So, should I wait until you get the chance to look at it? |
@mgonzcast Anything you find would be useful, but unless repeatedly running a test suite is something that brings you great pleasure, you might want to wait until the fix that assertj has highlighted is confirmed + available. |
:) No great pleasure I see the tests after your changes have worked. Let me know how I update pitest and if maybe the pitest sonarqube plugin has to be updated as well. |
It will likely be a little while before this is merged in. Changes to the coverage system have subtly degraded the performance of the overall mutation analysis on the past, so I'll need to do a bit of playing around to sanity check it. Since it's a big change that affects the file pitest produces, it will go into a 1.9.0 release. I'll probably release the other queued changes as a 1.8.x release before then. To play with it before then, you'll need to check out and build it. If you do a local install, the version number will be 1.0.0-SNAPSHOT. |
When do you expect to release those changes? I can wait a couple of weeks.
Not sure which parts do i have to rebuild and where to place them in my machine. Do I have to download code as it is now? Any instructions? |
@mgonzcast Should be released some time next week. |
@mgonzcast This is now released. 1.9.0 contains a fix for high memory consumption in the main process (it has no effect on the minions), and a change that greatly speedups up JUnit 5 tests during the coverage stage. Since you look to have high memory consumption in the minions as well, I don't know how big the impact will be, but hopefully with things improved in the main process it will be easier to find any remaining issues. |
@hcoles Thanks. Today I tried the new release 1.9.0 with the following settings:
And I get:
|
Thanks @mgonzcast, I assume this is for the large module? Can you enable verbose logging and see if it provides any additional info? |
Sure @hcoles. I enabled -X debug option and I get this in the logs:
|
@hcoles any ideas of what can I try now to narrow down the issue? |
@mgonzcast If you set pitest's own logging to verbose (-Dverbose=true), then it might show something, although I'm not too hopeful. After that it would be question of whittling away at the module to turn it into the smallest thing that reproduces the issue, and that you are happy to share (which can be done privately if that helps). |
@hcoles I did and I get this strange error (I replaced real strings for anonimity):
I don´t quite understand why It´s checking that module because I added to the list of Excluded classes, maybe I am excluding it wrongly? Also I still see the socket error...maybe the antivirus is coming to play?
|
@mgonzcast How are you excluding the powermock tests? Although I'm not sure what's happening, they are a likely root cause. It is certainly possible that antivirus is interfering with the communication between the main process and the minion, there have been reports of this in the past. I don't see how it would result in the OutOfMemoryError you reported earlier however. |
I was excluding classes with -DexcludedClasses switch. I have searched for the string powermock and I found all powermock instances (all in src\test). Your question made me think that I was doing the exclusion wrongly and after googling around I found another switch -DexcludedTestClasses. If I exclude the tests classes with powermock using that switch, pitest doesn´t fail but I get this poor results:
I will try to narrow more the list |
Just wondering...why some tests with powermock don´t work with pitest and others do? Do I have to look for anything in particular? |
@mgonzcast I'm not sure what the issue is. If you could post an example of a test that doesn't work, that would be helpful. |
As you said it was mockito tests. It makes sense:
Excluding those test classes I was able to run the pitest. But it takes around 6-7 hours for 1300 lines of code (originally before exclusion 18k lines of code) |
@mgonzcast Can you post a full (simplified) example of a test that causes the issue? Pitest already runs tests against MockitoJunitRunner as part of its build, so more info will be required to reproduce this. |
PowerMocks are used too (sorry I didn´t say earlier). PowerMock example:
@hcoles Does this help? |
@hcoles do you need more examples? |
@mgonzcast I've not been able to reproduce the problem. If you can put together an minimal project that reproduces the issue, that would help. |
Hi,
I have several modules in a project and only (the biggest one) fails with GC overhead limit exceeded.
I have searched around and I can´t find how to fix this.
Originally I run this command (which works in the rest of modules):
mvn clean install org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML -DexportLineCoverage=true -DtimestampedReports=false
and I get:
[INFO] Defaulting target classes to match packages in build directory [INFO] Defaulting target tests to match packages in test build directory 11:38:57 PIT >> INFO : Verbose logging is disabled. If you encounter a problem, please enable it before reporting an issue. 11:39:04 PIT >> INFO : Incremental analysis reduced number of mutations by 0 11:39:04 PIT >> INFO : Created 668 mutation test units in pre scan 11:39:05 PIT >> INFO : Sending 861 test classes to minion 11:39:05 PIT >> INFO : Sent tests to minion \11:43:12 PIT >> INFO : MINION : log4j:ERROR A "org.apache.log4j.xml.DOMConfigurator" object is not assignable to a "org.apache.log4j.spi.Configurator" variable. 11:43:12 PIT >> INFO : MINION : log4j:ERROR The class "org.apache.log4j.spi.Configurator" was loaded by 11:43:12 PIT >> INFO : MINION : log4j:ERROR [org.powermock.core.classloader.javassist.JavassistMockClassLoader@1e313166] whereas object of type 11:43:12 PIT >> INFO : MINION : log4j:ERROR "org.apache.log4j.xml.DOMConfigurator" was loaded by [sun.misc.Launcher$AppClassLoader@18b4aac2]. 11:43:12 PIT >> INFO : MINION : log4j:ERROR Could not instantiate configurator [org.apache.log4j.xml.DOMConfigurator]. /11:59:21 PIT >> INFO : MINION : Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. 11:59:21 PIT >> INFO : MINION : Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= -
I have tried:
mvn clean install org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML -DexportLineCoverage=true -DtimestampedReports=false -DjvmArgs="-Xmx4096M,-XX:ReservedCodeCacheSize=2048M,-XX:-UseGCOverheadLimit" -Dthreads=4 " -Dfeatures="+CLASSLIMIT" -DwithHistory
But although the number of tests unit decreased, the number of test classes sent to minion are still 865 and I get an outofmemroy and GC overhead limit exceeded.
Any hint?
The text was updated successfully, but these errors were encountered: