In my last post, I covered the basics for creating a protractor test suite to test an AngularJS application. The next step for all self-respecting developers is having the suite running as part of a CI/CD pipeline. I promised to outline how you might do this with TeamCity. Before I do that, I want to cover some of the stages that you might miss.

Writing a protractor test suite boils down to writing JavaScript. As with any development effort this might result in good code or it might result in a ball of mud. Whilst the success of the test suite is dependent on the quality of the tests, good automated suites are also dependent on the quality of the code that makes them up. The automated test suite will run regularly, so it needs to be reliable and efficient. The software will evolve, it will change, so the tests will need to change too. Your test code needs to be simple and easy to maintain. The software principles that apply to production code such as SOLID, should also apply to your test code.

Testing Patterns

A typical pattern used when writing automated end 2 end tests is Page Objects. The idea is that the complexity of driving a particular page is removed from the test itself and encapsulated in a page object. The page object has intimate knowledge of a page in the application and exposes common features that the tests will need. The pattern reduces the amount of repeated code and provides an obvious point of reuse. The page object evolves at the same rate as the underlying application functionality. Done correctly the tests themselves become decoupled from changes in the application.

As the page object and the underlying page end up in step it becomes natural for the developer and the tester to work closely. The developer helps the tester understand the capability of the page. The tester then builds a page object to expose this functionality to tests. Done correctly this collaboration drives out an application that is easy to test. As new functionality is created the tester can influence the developer to write the page in a way that is easy for them. One common area is the use of selectors to find elements within the page’s DOM.

Tests written after a page is finished often result in complex and/or unreliable selectors. Complex XPath expressions or custom logic to search the DOM tree for the correct element is common. Tests written like that become hard to maintain. On the other hand, if the tester and developer work together and importantly, the developer understands how their work will be tested, then pages that are easy to test are the result.

Testing the Tests

Writing test code starts to look and feel like writing production code. Similar tooling should be in place. Don’t let your testers struggle with basic editors when your developers are using fully featured IDEs with syntax highlight and intelli-sense. Testers working with protractor should use a decent editor such as VSCode, Sublime or ATOM with a compliment of JavaScript plugins. In-particular use linting tools to ensure that the JavaScript written for tests matches similar style guidelines as the main application AngularJS code.

You should also ensure that it is easy to test your test. This link demonstrates how to set up VSCode to run a test specification by hitting F5. This article itself advertises that you are able to debug protractor tests. Whilst I have managed to hit breakpoints in my tests the debugging experience has fallen well short of my expectation. I have not had the time to investigate further.

Team City Integration

The final step of this is to have your test suite running as part of your CI/CD pipeline. Protractor is a e2e testing framework. End 2 end tests may take some time to execute. Large test suites may take several hours. CI builds should take minutes not hours so it is advisable to run these types of test suite overnight. Daily feedback is better than no feedback at all. If you have a deployment build that deploys to a specific environment each night you could tag execution of the test suite to the end of that. If you deploy to an environment less regularly you can still run the suite each night.

TeamCity happens to offer comprehensive statistical analysis of repeat test runs. It provides trends that show if particular tests are getting slower and it will also highlight tests that are flaky – i.e. ones that sometimes work and sometimes fail even if nothing has changed.

The simplest way to integrate your suite into TeamCity is to use the TeamCity Jasmine reporter. You can add the reporter to your conf.js like this. Don’t forget to add it your package.json file.

exports.config = {
    framework: "jasmine2",
    capabilities: {
        browserName: "chrome"
    },
    onPrepare() {
        let jasmineReporters = require("jasmine-reporters");
        jasmine.getEnv().addReporter(new jasmineReporters.TeamCityReporter());
    },
};

In the last post, I created a gulp task to execute the suite. The final step is to add a build step to execute it.

protractor1

I have a npm install step that ensure all the dependencies required by the tests are downloaded. I’m then executing gulp e2e to run the test.

You can see this is working on the build summary. You can see a count of passing and failing tests.protractor2

As your test suite is outputting results just the way TeamCity likes them you can drill into your tests and get all sort of useful stats, like this.

protractor3

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s