bin/rake and rake have different results

[sunnyrjuneja]

Hi! I noticed that rake test and bin/rake test have different coverage results. Its a rails project with grape mounted. The files being skipped are the app/api_engine folder (grape app). Any ideas on how I can get them to equal?

ubuntu@box103:~/web$ bin/rake test
Run options: --seed 8237

# Running:

.......................................................................

Finished in 1.146666s, 61.9186 runs/s, 73.2559 assertions/s.

71 runs, 84 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to /tmp/circle-artifacts.TrN8o5k/coverage. 324 / 326 LOC (99.39%) covered.

ubuntu@box103:~/web$ rake test
Run options: --seed 33576

# Running:

.......................................................................

Finished in 1.144196s, 62.0523 runs/s, 73.4140 assertions/s.

71 runs, 84 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to /tmp/circle-artifacts.TrN8o5k/coverage. 406 / 408 LOC (99.51%) covered.

[bf4]

Do you have spring running? Try bin/spring status bin/spring stop bin/rake test bundle exec rake test

Closing as is not a simplecov issue. See #340

[sunnyrjuneja]

@bf4 Thanks for getting back so quickly.

I am using spring and it was running. However, restarting spring doesn't resolve the problem. I also tried adding a .simplecov file with the following configuration:

require 'simplecov'
SimpleCov.start do
  add_filter 'vendor'
  add_filter 'config'
  add_filter 'bundle'

  add_group 'API', 'app/api_engine'

  minimum_coverage 95
end

if ENV['CIRCLE_ARTIFACTS']
  dir = File.join('..', '..', '..', ENV['CIRCLE_ARTIFACTS'], 'coverage')
  SimpleCov.coverage_dir(dir)
end

Just to confirm, this has nothing to do with simplecov and I should file an issue with Spring instead?

[bf4]

Did you try bundle exec rake instead of bin/rake like I suggested? Please read #340 simplecov only knows what files are loaded from when it starts until when the program stops. That's under your control, and outside of spring or simplecov.

[sunnyrjuneja]

Hey @bf4, I've read #340 a few times. Both now and when you first posted it. I don't want to waste your time and I'm sorry if it seems that I don't get it. I'm very grateful for both your help and contributions to simplecov.

I know the bundle exec rake works perfectly but bundle exec rake takes significantly longer to run bin/rake. The reason I posted my .simplecov is because that I thought that was partly what #340 was about.

After rereading this issue, I think it might be possible that I wasn't explicitly clear enough.

bundle exec rake returns the correct coverage and works as expected.
bin/rake returns the incorrect coverage is excluding the app/api_engine directory. api_engine is both on my paths and autoload paths in config/application.rb. I've also added the group in .simplecov like above.

Is there anything else I can do to load the app/api_engine directory?

λ →  time bundle exec rake
# Omitted
bundle exec rake  3.83s user 0.47s system 97% cpu 4.402 total

λ →  time bin/rake
# Omitted
bin/rake  0.11s user 0.00s system 5% cpu 2.183 total

[colszowka]

@whatasunnyday Is it possble that your bundled rake version is different from the latest one installed on your machine/used via bin/rake?

My idea is that your engine tests run in a separate process for the bin/rake method and hence get overwritten. Maybe try to add a SimpleCov.command_name 'Engine' somewhere in a test file where your engine code is tested and see if that resolves it.

[sunnyrjuneja]

@colszowka It appears they're all same.

λ →  rake --version
rake, version 10.4.2
λ →  bundle exec rake --version
rake, version 10.4.2
λ →  bin/rake --version
rake, version 10.4.2

However, your suggestion of adding SimpleCov.command_name 'Engine to a file where my engine code does increase the coverage by a small amount. Is there anything I can do other than decorate every file with SimpleCov.command_name 'Engine'?

[bf4]

Please paste contents of bin/rake

Also try env DISABLE_SPRING=1 bin/rake

What I'm trying to say is that your bin/rake is either loading or
preloading app code before simplecov activates the Coverage module,
which only tracks the first time code is executed

Spring causes so many surprising failures

See https://github.com/rails/spring/blob/master/README.md

[sunnyrjuneja]

Here's bin/rake:

#!/usr/bin/env ruby
begin
  load File.expand_path("../spring", __FILE__)
rescue LoadError
end
require_relative '../config/boot'
require 'rake'
Rake.application.run

Using env DISABLE_SPRING=1 bin/rake returns the correct amount of coverage.

[bf4]

So, we're good?

[sunnyrjuneja]

@bf4 I might be misunderstanding but no, right?

I believe env DISABLE_SPRING=1 bin/rake is the same as running bundle exec rake so it isn't surprising that the coverage is correct and that it takes as long as bundle exec rake.

λ →  time be rake
# omitted
bundle exec rake  3.65s user 0.43s system 97% cpu 4.169 total

λ →  time env DISABLE_SPRING=1 bin/rake
# omitted
env DISABLE_SPRING=1 bin/rake  3.50s user 0.43s system 97% cpu 4.034 total

I can understand if Spring is out of your domain expertise and try to follow up with some maintainers there.

Is that the next logical step?

[bf4]

@whatasunnyday I no longer understand what the issue is. The reason you are getting different coverage is because spring affects when your code has loaded relative to when SimpleCov started running. We've established this, no? We've proven it by setting the env var, by looking at the file, by running bundle exec. At this point it sounds to me like you want spring to both load code and not load code at the same time, which I am sure is not what you mean.

The app/api_engine dir sounds like it's within the SimpleCov.root, so it's not a matter of filtering.

[sunnyrjuneja]

@bf4 I want app/api_engine to be included in the coverage when I run bin/rake through Spring. At this point, I'm not sure its possible. Sorry to have wasted your time.

[bf4]

Thanks for your report and following up. Not everyone does that, so it's appreciated. And I think we both understand the library and its quirks a little better now, no?

As a general rule, if I'm using a code reloader, my goal is more about re-running tests a lot for feedback. When I'm done, then I'll run my tests and care about the coverage. The way I do that is:

# .simplecov
@minimum_coverage = ENV.fetch("COVERAGE_MINIMUM") { 100.0 }.to_f.round(2)
SimpleCov.start "some_profile" # the profile I defined
if ENV["COVERAGE"] =~ /\Atrue\z/i
  puts "[COVERAGE] Running with SimpleCov HTML Formatter"
  SimpleCov.formatters = [SimpleCov::Formatter::HTMLFormatter]
else
  SimpleCov.formatters = []
end
SimpleCov.at_exit do
  SimpleCov.result.format!
  percent = Float(SimpleCov.result.covered_percent)
  if percent < @minimum_coverage
    abort "Spec coverage was not high enough: #{percent.round(2)} is > #{@minimum_coverage}%"
  else
    puts "Nice job! Spec coverage (#{percent.round(2)})is still above #{@minimum_coverage}%"
  end
end

So, I get feedback while the tests are running that may or may not be useful, but it only generates a report when I set COVERAGE=true

[sunnyrjuneja]

And I think we both understand the library and its quirks a little better now, no

Very much so. And thank you for your persistence in helping. I really do appreciate that.

Your tip is helpful. Thanks for that.

[colszowka]

@whatasunnyday So if this is an issue with spring, maybe you also could consider trying out spin. It's a very lightweight preloader with which I have had very little trouble in the past playing nice with simplecov!

[sunnyrjuneja]

@colszowka Thanks for the tip. I'll definitely investigate. I'm going to follow up with the people at Spring too.

[sunnyrjuneja]

Not sure if anyone is still interested but I required simplecov inside of config/spring.rb and that's added a lot of the missing files into simplecov. However, the coverage % being reported now is much lower. It seems that bundle exec rake is really the only way to get correct coverage results as of now.