Feature #12528
Skip test should break execution for a test case, an additional method should be added to confine tests to certain hosts
| Status: | Accepted | Start date: | 02/08/2012 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: |  Justin Stoller |
% Done: | 0% |
|
| Category: | Harness DSL | |||
| Target version: | - | |||
| Patch: | None | Branch: | ||
| Keywords: | ||||
| Votes: | 0 |
Description
We’ve been talking about improving the “skip_test” method.
Currently most of the usage of “skip_test” is to log the state as “skipped” and then NOT execute the rest of the test case. Its become apparent though that another usage of “skip_test” is also occurring, using “skip_test” to log the state as “skipped” and then continue on testing with a filtered subset of hosts.
This is a request for the method “skip_test” to log the state as “skipped” and then halt execution of the rest of the test case. In addition to this we should allow an easy way to filter tests.
I propose adding a “confine” like method that excludes any host that is inappropriate for the current test from being included in any of the other host methods (ie. would not be included in the results returned by “agents” method)
exclude_hosts :role => 'dashboard', :platform => 'windows'
This method would then call “skip_test” method IF there were no valid hosts after doing the exclusion.
Related issues
History
Updated by Chris Price 3 months ago
In what scenarios would the skip_test continue to be used under this proposal? For tests that were known to be globally broken, to effectively disable them until someone had a chance to fix them?
Updated by Justin Stoller 3 months ago
Good point. Currently in the PE steps we’re using skip_test to skip tests that are configuration specific. Example refactor that I’m working on for pe setup step:
skip_test “Only need to resign console certs when upgrading when we have a dashboard," + " the dashboard and master are on separate nodes and the version we are upgrading from is 1.2” unless dashboard && dashboard != master && options[:upgrade] =~ /^1\.2/
This is configuration specific skip, its not truly broken (or pending, I believe we should add a pending method but that’d be a different ticket, and I don’t believe it really falls in the domain of “filtering” hosts.
Updated by Josh Cooper 3 months ago
I would like to see include and exclude based confines, e.g. for testing scheduled tasks and cron, respectively, on Windows agents.
Updated by Justin Stoller 3 months ago
- Status changed from Investigating to Accepted
I think we’re in general agreement that the skip_test should be made to halt execution of a test.
We can’t implement that until we agree on a way to properly exclude hosts though, because a significant minority of the skip_test usage in the tests uses it more as a filter than a full skip.
I’ve created ticket with my ideas #12672, and a plan to more forward for the time being and long term goal of implementing a confine like behavior, I’d like you guys to take a look at it.