We have a check-in build that performs some unit testing but finally calls to octo command line to deploy to the CI environment. The final step of the deploy is a test automation project using Specflow and NUnit and Selenium which is executed in Deploy.ps1:
& $nunit “FunctionalTests.BDD.Features.dll” “/noshadow” “/framework:net-4.0” “/xml:Tests.nunit.xml” “/include:ci”
All is working fine
However, not all of our tests are being executed because the CI would then take way too long for every checkin.
What I would like to do, is have a separate build that runs every night or twice a day and runs ALL TESTS which means that I would like to remove the “include:ci”. Is this a good way of thinking? How can Octopus help me with this. I’m consistently looking at Octo Command Line tool but it doesn’t seem to have a variables parameter that I can adjust per build which can tell what tests should run.
I will be facing the (almost) exact same challenge in the upcoming months as well :).
So just like you, we have a CI build with unit tests and some acceptance tests that utilize SpecFlow (thumbs up for using it! :)).
However, we want to be able to execute a secondary, slower batch of tests, just like you mentioned.
Just one remark: “What I would like to do, is have a separate build that runs every night or twice a day and runs ALL TESTS” We are actually applying the same policy in our attempt at CI, but having Nightly builds is an outdated pre-CI practice which is not encouraged. I can tell you from experience that juggling both a CI and a Nightly build just so that you can run different sets of tests on the same deliverable, is a waste of your time in the long run due to the duplicated efforts that you need to do maintenance on the 2 build definitions. My remark against Nightly builds stems from my own experience and from the conclusions I’ve made through reading the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
I will presume that you have just one build which needs to have different deployments. So with that in mind, here’s the plan that I’m thinking of implementing:
-1. The CI build goes through and then you deploy it to an environment and trigger the fast-running test (unit tests and Cucumber/SpecFlow scenarios)
-2. On a predefined trigger (once every night or twice a day), a new deployment of the latest CI build will be deployed through Octo.exe to a given environment where post-deployment you will trigger the slow-running batches of tests.
After the tests complete you can do either of the following again in the post-deploy PS script:
—a. Send an email with the test results to all developers/stakeholders.
—b. Return a non-zero exit code, thus failing the deployment step (“Deploy and run ALL TESTS”)
—c. Although your CI build is say, several hours old, build tools like MSBuild allow you to modify the outcome status (Succeed/Failed) and mention a specific reason (“Acceptance tests failed” for example), thus your deployment can help you mark that CI build as a Stable Nightly or a Release Candidate, for example.
There’s currently no way to pass variables when deploying, you are quite right. I can’t really think of a good way to do this in 1.X apart from setting up two seperate projects, or having two steps (one for CI and one without) and using the ‘skip steps’ feature to choose which one to run. However there’s no way to specify skip steps from octo.exe currently.
However in 2.0 we’re adding some features which will make this nicer:
You’ll be able to pass variables during a deployment
Your test results can be returned as ‘artifacts’, and can be viewed on your Octopus server
Unfortunately, we are using Jenkins at the moment which does an ok Job calling msbuild and octo. I might switch to Team City eventually, but that’s different issue.
The 2nd point you mention above is what I’m just struggling with at the moment because I’m not sure how I would hook Octo.exe to tell which tests to run. Then, I wasn’t sure whether I should be building the source code all over again and deploying everything again just to run all the tests.
I’m getting swayed to just create a job that performs "Run All Tests " by calling NUnit directly juts like Deploy.ps1 is during CI and this would happen on the last successful deploy.
I have my own command line tool, so it would be something like:
exe -TaskName “RunBDD” -Categories “*”
The only problem I have with this strategy is that I now have Jenkins “knowing” where my CI server is instead of being managed completely in Octopus Environments… more to follow.
So… the solution i mention above is working perfectly.
There was no point initiating a new build and deploy. Instead, a smoke test runs every night now for the last successful release on the CI server by calling NUnit-console.exe.
Sweeeeeeet…
I’m definitely interested to see what the solution will be for the up-coming release so that it’s just built-in to Octopus.
