Digging into Cypress.IO in a sprint to replace my almost-completely-useless stack of Selenium IDE tests for Store Locator Plus, I’ve run into some issues that seem to go beyond my lack of knowledge with the tool.

Yes, it is a great step in the right direction and it can be man-handled to do what I need.  For now.   But tools should not be forcing you to do things a specific way. That are supposed to make you more efficient, no re-train you or add extra steps.

Cypress.IO has shortcomings when it comes to doing real-world end-to-end testing of web apps.   Order of tests is one of them.

Assumption of Order pattern

A very common test pattern in the real-world is what I’ll call the “Assumption of Order” pattern.   Maybe there is a real “QA professional” pattern that accounts for this mode, but I’ve not seen it referenced anywhere else so I’m going with it as my made-up term.

In my “Assumption of Order” test pattern it is important that at least some of the test in a series of <insert common test type name here> tests.    In my case I test Store Locator Plus “the way a user interacts with the system” as best I can.   It is a series of Integration tests, Smoke tests, and Acceptance tests.     Plus a little Unit and System tests mixed in.

Having a written series of tests beats the “let’s just use the new version and see if it breaks for a few weeks” test pattern or “my shit doesn’t stink / I’m a programming God” don’t test a damn thing pattern that most WordPress plugin developers seems to go by these days.    However to speed up testing and not eat up all my time just writing tests I need to be as efficient as possible.

Enter Assumption of Order testing.   I can get through testing a LOT faster if I can ASSUME that my “Load 5,000 Locations” test has already run (and passed) when I run my “Look Up Locations 5 Miles Away” and “Look Up Locations 10 Miles Away”.    Having a future test rely on “data leftovers” from a prior test is imperfect.   But in years of doing software for everything from my pet projects to Fortune 500 companies to military applications PERFECT software is never-released software.   Same goes for testing.

A testing anti-pattern

As it turns out, it would make my life a LOT easier if my new testing tool – Cypress.IO , happened to allow users to set the order in which the test scripts run.

It doesn’t.     At least not as far as I can tell.     I soon found this post on the Cypress-IO Github issues board posted back in January 2017.  Apparently this is “anti-pattern” so the people behind Cypress.IO, or at least Brian Mann (assuming by his Github handle) does not believe this is a big deal.

Maybe not but it sure as hell makes writing tests a lot more painful.

If I want to tow a 40-foot boat with my Ferrari so be it.   Let me be an ass if I so choose.

My anti anti-pattern argument

As posted on the Github thread:

 

As @digitaldavenyc points out full blown web apps often test complex system interactions.

 

Should every test be able to run independently and in a vacuum? Sure. In theory. Real world and theoretical science do not always get along. It may be anti-pattern but coloring 100% within “the pattern” lines means my test scripts take 27 hours to complete instead of 27 minutes.

Use Case–

 

My app allows users to add locations to a data set on the back end. On the front end it renders a map of locations.

 

Front-End test: Make sure 10 locations are returned from Charleston SC

This is heavily dependent on the data set that includes two primary elements: location data + user-set options such as “default radius to be searched”.

 

I could write the test to first pull the options data (default: 5 mile radius), THEN pull all location data (5k entries), THEN let the script loop through all 5,000 to calculate the distance to create the baseline of valid results, THEN run the front-end query and make sure it returns the same list my test calculated.

 

I’ve now created another point of failure as my test script code is more complex and prone to errors. Not too mention it takes a LOT longer to run by not being able to make data assumptions.

 

OR

 

I could write a test that ASSUMES my “load_5k_locations_spec.js” has executed and passed. The test is now pull the option data, load a “valid_5_miles_from_charleston.json” fixture, run the front-end query and compare it against the displayed location divs.

An order of magnitude faster AND far less complex test script code.

NOW…

take the above and run the test for five different radius options.

 

I’d rather pull a valid_5_mile.json, valid_10_mile.json, etc. and compare against an assumed set of data that can ONLY be a valid assumption if I am certain my “load_5k_locations_spec.js” ran BEFORE all my “5miles_from_charleston.js” and “10miles_from_charleston.js” scripts ran.

Bonus Points: Have a Cy.passed( “load_5k_locations_spec” ) command the returns true|false if the specified test passed on the prior run — makes it easy to completely skip a test if a prior test run failed

 

No – not perfect “by the pattern” rulebook implementations, but in the real world people have deadlines. Making tools that are malleable enough to meet user’s needs versus doing things “strictly by the book” is what makes them powerful.

 

I’m fairly certain chain saws are not designed to be used to make ice sculptures – but people do it. If you try to do the same and cut your arm off because you don’t have the skills to use that tool in a manner it was not intended, consider it a “learning curve”.

Thoughts?

So there you have it – my argument for “Assumption of Order” as a valid real-world test pattern.

Have a better way to do it?   Am I wrong?   Did I miss something in Cypress.IO?   Is there another tool out there to test reactive web apps that is not as complicated as Selenium WebDriver, as broken as Selenium IDE, or as “pattern or die” as Cypress.IO?   Let me know in the comments.