One basic trait of all scripts needs to be a clear determination of result. ‘Sorta’ is not a state we want in automation. Its okay-ish in exploratory testing (as it leads to other avenues of exploration) but not here.
Here is a script I stumbled upon this week.
@pytest.marks('deep', 'videosearch')
def test_next_navigation_not_visible_for_less_than10_search_result(self):
search_results_page = self.video_dashboard_page.perform_search(SearchTerms.random_video())
if search_results_page.number_of_search_pages() < MINIMUM_PAGE_COUNT_FOR_NEXT_TO_APPEAR:
self.assertFalse(search_results_page.is_next_navigation_visible())
Notice how if the data criteria is not met then the determination of result is going to always be a pass. In essence,
@pytest.marks('deep', 'videosearch')
def test_next_navigation_not_visible_for_less_than10_search_result(self):
pass
This is the sort of thing that causes failures to be reported as passes (since the check was never encountered) and the gradual erosion of trust in the automation. The fix was to fail the script if the data necessary wasn't returned and only if we have what was necessary to do the assert. (As discussed in Pagination for data sets you don't control.)
Just another thing to keep in mind when scanning through your automation code.
Comments 2
I can’t understand your description of the fix.
If that were my script, I’d have set up a loop that iterated until the number of search results returned *was* less than the constant, and then done the check.
Posted 20 Jan 2012 at 1:59 am ¶And that’s what was there originally. But. What if that looping took 6 hours to meet that condition? That would have blocked the gathering of other information (and that is the only reason to do automation in the first place). The solution to the unbounded wait time problem is to put in a max number of loops to try, but then still you have the problem of the script passing even if never got to the check it was based around if the loop expired.
What would have been more useful of course is if I had put in the [a] solution in the post.
@pytest.marks('deep', 'videosearch') def test_next_navigation_not_visible_for_less_than10_search_result(self): search_results_page = self.video_dashboard_page.perform_search("some search term we know rather than a random one") if search_results_page.number_of_search_pages() < MINIMUM_PAGE_COUNT_FOR_NEXT_TO_APPEAR: self.assertFalse(search_results_page.is_next_navigation_visible()) else: pytest.fail("Could not get the data necessary for the script")Posted 20 Jan 2012 at 8:17 am ¶Post a Comment