Bug #10296
rspec unit tests are not reporting failure when they should be
| Status: | Closed | Start date: | 10/26/2011 | |
|---|---|---|---|---|
| Priority: | High | Due date: | ||
| Assignee: | - | % Done: | 0% |
|
| Category: | testing | |||
| Target version: | 2.7.12 | |||
| Affected Puppet version: | development | Branch: | https://github.com/puppetlabs/puppet/pull/385 | |
| Keywords: | rspec schedule testing | |||
| Votes: | 0 |
Description
I was working on a patch to the schedule type and as part of this I was writing some additional unit tests into the spec/unit/type/schedule_spec.rb file
I had written a number of tests which I fully expected to fail and then ran them. Much to my surprise they all showed as passing.
After some investigation, every call like “@schedule.should be_match” always reports passing, even if it should have failed. After dropping some debugging statements in schedule’s match? method it seems that it is never even being invoked.
All tests of the form “@schedule.should_not be_match” do invoke match? in schedule and pass/fail appropriately.
I don’t know enough about how rspec works to troubleshoot it further, but it seems like a considerable issue to me. In fact, I believe that if this was working properly you would see at least one test failure relating to arrays of ranges.
A simple change that can be made to the unit test to show this behavior is:
diff --git a/spec/unit/type/schedule_spec.rb b/spec/unit/type/schedule_spec.rb
index b302d95..7a04abc 100755
--- a/spec/unit/type/schedule_spec.rb
+++ b/spec/unit/type/schedule_spec.rb
@@ -81,12 +81,12 @@ describe Puppet::Type.type(:schedule) do
end
it "should match when the start time is before the current time and the end time is after the current time" do
- @schedule[:range] = "10:59:50 - 11:00:10"
+ @schedule[:range] = "10:59:50 - 10:59:55"
@schedule.should be_match
end
it "should not match when the start time is after the current time" do
- @schedule[:range] = "11:00:05 - 11:00:10"
+ @schedule[:range] = "10:59:05 - 11:00:10"
@schedule.should_not be_match
end
Before this change, both show success (as you would expect). After this change, This first one should report failure, and doesn’t. The second one should report failure and does. Dropping a debugging print in schedule’s match? method will show that “should"s never invoke match?. This is the single failure run after the above changes:
Failures:
1) Puppet::Type::Schedule Puppet::Type::Schedule when matching ranges should not match when the start time is after the current time
Failure/Error: @schedule.should_not be_match
expected match? to return false, got true
# ./schedule_spec.rb:90
Finished in 0.52018 seconds
41 examples, 1 failure, 1 pending
Thank you.
Related issues
History
Updated by Sean Millichamp 7 months ago
- Priority changed from Normal to High
After giving this a few seconds more thought, I am increasing this to High priority. I know that the testing framework is depended on pretty heavily to report accurately on the success/failure of tests and I don’t know if this issue is limited only to the schedule tests or widespread throughout all tests.
Updated by James Turnbull 7 months ago
- Category set to testing
- Status changed from Unreviewed to Needs More Information
- Assignee set to Sean Millichamp
Good catch. Sean – what Puppet version?
Updated by Sean Millichamp 7 months ago
This was against a recent update from the master branch.
I am running on Fedora 15 using the Ruby 1.8.7 that ships with it.
Updated by Sean Millichamp 7 months ago
- Affected Puppet version set to development
- Branch set to master
Updated by James Turnbull 7 months ago
- Status changed from Needs More Information to Accepted
- Assignee deleted (
Sean Millichamp)
Updated by Justin Stoller 7 months ago
(had a little free time during lunch and this sounded interesting, FWIW)
Changing these tests from
@schedule.should be_match
to the uglier, but more explicit
@schedule.match?.should be_true
causes two (different than mentioned) tests to fail with wrong number of argument errors. While in irb initializing an instance of using the same steps in the tests I get all of the correct behavior (without args):
(note: it was about noon when I did this [and didn’t stub out Time in irb])
>> require 'puppet' >> @sched = Puppet::Type.type(:schedule).new(:name => 'testing') >> @sched[:range] = "10:59:50 - 10:59:55" >> @sched.match? => false >> @sched[:range] = "10:59:50 - 16:00:00" >> @sched.match? => true >> @sched[:range] = "13:59:50 - 16:00:00" >> @sched.match? => false
Updated by Sean Millichamp 7 months ago
I can confirm Justin’s findings.
If I take the original spec code from master (before the diffs I posted) and replace all instances of:
@schedule.should be_match
with
@schedule.match?.should be_true
or, if they have a parameter passed, replacing all instances of:
@schedule.should be_match(previous)
with:
@schedule.match?(previous).should be_true
Then I get one failure:
Failures:
1) Puppet::Type::Schedule Puppet::Type::Schedule when matching ranges should match the upper array of ranges
Failure/Error: @schedule.match?.should be_true
expected false to be true
# ./schedule_spec.rb:116
Finished in 0.5196 seconds
41 examples, 1 failure, 1 pending
I believe that this is correctly failing now. I could not get arrays of ranges to work in my prior interactive testing of schedules and this failure supports my experience that they do not function.
What should the next course of action be? I could submit a patch that would adjust all of the tests to the “@schedule.match?.should” style but I feel like leaving the “@schedule.should be_match” style unfixed would be asking for this to happen again and likely should be addressed by someone who knows rspec.
Updated by Patrick Carlisle 5 months ago
Puppet::Type has a method named should, from before we were using rspec. Unfortunately, if you write a test and it calls should on a type object the test will silently pass. We have an alias in place called must. You can replace should with must and get the correct behavior. It looks like all the schedule tests have this problem. I’ve fixed some of them in #10321, and I’m taking a look at the rest now.
Updated by Sean Millichamp 4 months ago
- Status changed from Accepted to In Topic Branch Pending Review
Patrick,
I have gone through all of the “should” form test and converted them to the “must” variant. There was one test that was a “should”, but both the description and logic of it seemed to indicate that it ought to have been a “should_not” and, due to “should” always passing, was never caught as an error. I converted that from should to must_not as part of the clean up.
I just submitted a Github pull request: https://github.com/puppetlabs/puppet/pull/385
Updated by James Turnbull 4 months ago
- Branch changed from master to https://github.com/puppetlabs/puppet/pull/385
Updated by Stefan Schulte 4 months ago
- Status changed from In Topic Branch Pending Review to Merged - Pending Release
- Target version set to 2.7.11
Updated by Matthaus Litteken 3 months ago
- Status changed from Merged - Pending Release to Closed
- Target version changed from 2.7.11 to 2.7.12
Released in 2.7.12rc1