Allow functional test automation on Firefox 47+

[LeeDr]

Kibana version: any

OS version: any

Original install method (e.g. download page, yum, from source, etc.): any

Description of the problem including expected versus actual behavior: With Firefox 47 released June 7, 2016, the selenium standalone server no longer works. To run the automation against Firefox 47+ we'll need to run a Marionette server.
See https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver

Steps to reproduce:

1. Let your Firefox update to 47 or later
2. try to run functional tests (npm run test:ui)
3. the browser opens but then closes before any tests run

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

This PR is switching the functional test automation to use Chrome browser; #7442
So we shouldn't be blocked from running the tests, but ideally we could run the automation on multiple browsers.

[cuff-links]

This issue isn't a "problem" with any browser, nor Kibana. In FF 47, Mozilla updated Firefox to the updated WebDriver Spec for certain endpoints. If you run the tests, as is on FF, you will notice that you get a "Command Not Found" error when trying to run /execute on an element. The webdriver spec has updated this endpoint to/execute/sync but the library we use, leadfoot has not been updated to compensate. I filed an issue and am working on a patch that will soon be merged.

There is another issue when typing into elements. The parameter name has changed from value to text and accepts a string as an argument rather than an array of characters. This causes breakage there, as well. I am currently working on a fix for that issue as well.

These issues being solved should get the tests up and running but I am now bumping into Firefox Specific issues that will need to be addressed for the tests to work on FF.

[cuff-links]

Issue 1: Using the roughly tilde when selecting data-test-subj does not work in FF. At all.

[data-test-subj~='blah'] fails whereas [data-test-subj='blah'] works. But the problem with that is that some of the tests now break on chrome when you remove the tilde. We need to assess why the tilde was chosen for all data-test-subj and see how we can make it the opposite.

[cuff-links]

Issue 2: Using Leadfoots findDisplayedBy* calls does not work in Firefox. Rather, it's findBy* that works. According to LeadFoot docs, the findDisplayedBy* calls should only be used as a last resort as they are inherently slower. I haven't found the root cause as to why they don't work in FF, but FF never finds the element when this is used.

[cuff-links]

These changes cannot be made with a simple string replace as they break existing tests on Chrome (I tried). Rather, we need to dig into test cases to find out why these mechanisms were chosen. Likely, each was chosen for a specific case. We will then need to test if that case is also true on FF and then move forward with a solution. This has gotten much larger than just making some updates to Leadfoot.

/cc @spalger @LeeDr @epixa

[LeeDr]

One thought, go ahead and replace all the findDisplayedBy calls and remove the ~ in the data-test-subj methods and put that in a PR. Then we can distribute the effort of fixing tests to more people to get it done.

Another thought is if we could do something like @epixa proposed for the new platform and segregate (maybe in config.js, or two different config files) the tests (test suites) that would pass on both browsers from the tests that only pass on Chrome. Then as we fix tests to work on both we move them to that config/group. This method could actually be merged as long as Jenkins kept running only the full set of Chrome tests until all tests were supported on Firefox. But the problem I think is not really the tests but the page objects with selectors. I can't think of any easy way to fork those into an old (Chrome only) and new (Chrome and Firefox) set and use the correct ones in tests. Maybe this just takes us back to a PR branch.

[cuff-links]

Hi Lee,

I agree with the statement that it would be difficult (if not impossible) to fork the page objects in an efficient way. I will go ahead with the PR. I will use this issue as meta to report errors in the different apps as they show up.

[cuff-links]

Gonna be adding test failures here. We can check them off as they are fixed.

Chrome

• Context
  • Tests don't run
• Getting Started
• Status
• Home
• Management
• Time Lion
• Dashboards
  • Adding Visualizations
    • Failure: Could not find findByCssSelector [data-test-subj="paginateNext"]
• Discover
  • Should not show "No Results"
    • existsByCssSelector [data-test-subj="discoverNoResults"] Expected true to equal false
• Visualize
  • creating and using Painless numeric scripted fields
    • Failure: findByCssSelector [data-test-subj="showFilterActions"] Unable to locate element

Firefox

• Context
  • Tests don't run
• Status
• Home
• Time Lion
• Management
  • should sort ascending
    • Failure findByCssSelector [data-test-subj="removeAllFilters"] could not be scrolled into view
• Dashboards
  • Is Disabled By Default
    • Failure findByCssSelector [data-test-subj="confirmModalConfirmButton"] Unable to locate element
• Discover
  • save query should show toast message and display query name
    • Failure: findByCssSelector [data-test-subj="discoverSaveButton"] Element not visible on the page
• Getting Started
  • should configure Terms aggregation on play_name
    • findByCssSelector [group-name="buckets"] vis-editor-agg-params:not(.ng-hide) .field-select : Unable to locate element
• Visualize
  • should visualize scripted field in vertical bar chart
    • Failure findByCssSelector [data-test-subj="showFilterActions"] Unable to locate element

[cuff-links]

Of course, since our tests build upon each other, I can't check subsequent tests once a failure is found.

[cuff-links]

@LeeDr Assessment done. This is where we're at right now with FF and Chrome. Many of FF tests don't even get to start because of a failure in the before all hook.

[cuff-links]

/cc @LeeDr Update. Fixed issue with clicking kibana indices. Our main issues now reside around Leadfoot's not updating the /active endpoint and the issue with typing index patterns and the asterisk not being appended to the end.

[cuff-links]

Commited patch for the Leadfoot issue

[cuff-links]

Updated which tests are failing. We have over 50% of our tests passing now. I believe the rest are just tweaking selectors to work on both browsers.

[LeeDr]

Closing in favor of #19010
The consensus is that there's too much work (and on-going maintenance work) to stay with Leadfoot.