-
Notifications
You must be signed in to change notification settings - Fork 10.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests: Add visual regression tests #27619
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
0af751f
tests: Add visual regression tests
ascorbic f137bee
Add config
ascorbic d60385a
Wait for images to load
ascorbic 6660374
Do multiple window sizes
ascorbic a601c97
Adjust settings for comparisons
ascorbic be908cb
Fix wrapper
ascorbic e1f9913
Add 1024 window
ascorbic 147efe4
Save reports
ascorbic 80b83f4
Merge branch 'feat/image-resolvers' into tests/add-visual-tests
ascorbic 530d8d7
Update fixed image too large snapshot
ascorbic 11f124b
Disable video
ascorbic a10050e
Update readme
ascorbic f3f6680
Update element id
ascorbic c9803fb
Force 1x pixel density
ascorbic d52bbd6
Update readme
ascorbic File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# Project dependencies | ||
.cache | ||
node_modules | ||
yarn-error.log | ||
|
||
# Build directory | ||
/public | ||
.DS_Store | ||
|
||
# Cypress output | ||
cypress/videos/ | ||
cypress/screenshots/ | ||
cypress/results/ | ||
__diff_output__ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
The MIT License (MIT) | ||
|
||
Copyright (c) 2015 gatsbyjs | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# Visual regression tests | ||
|
||
This test suite uses [cypress-image-snapshot](https://github.com/jaredpalmer/cypress-image-snapshot) | ||
to compare screenshots of pages or elements with a saved snapshot. | ||
|
||
To add a test, add a page to `src/pages`, then add a test to `cypress/integration`, or add to an existing spec. | ||
|
||
If tests fail, a comparison image will be written to `__diff_output__`. When running in CircleCI, this is uploaded to artifacts. | ||
|
||
## Considerations | ||
|
||
Remember that the test will run on Linux in CI, so avoid tests that might change between platforms. | ||
Using default fonts is an example. In general, if you're not testing the text itself then exclude it from your tests. Rather than comparing the full page, a good idea is to compare a wrapper element. There is a component provided for this purpose for images. | ||
|
||
Specifying large screen sizes can also cause problems when running locally on a small screen. The image tests use a maximum of 1024x768. The device pixel density is forced to 1, so running tests will look strange on Retina screens. This is to ensure screenshots match, whichever monitor or headless CI the tests rae running on. | ||
|
||
## Updating snapshots | ||
|
||
Run `yarn cy:update-snapshots` if you need to update them. Please note that unlike Jest, this doesn't delete outdated snapshots, so if you remove a test make sure to remove its snapshots too. | ||
|
||
## Credits | ||
|
||
Test images of Cornwall by [Benjamin Elliott](https://unsplash.com/photos/lH0_kBu5iyo) and [Red Zeppelin](https://unsplash.com/photos/uJMxXtH-Qso) via Unsplash. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
{ | ||
"baseUrl": "http://localhost:9000", | ||
"video": false, | ||
"reporter": "junit", | ||
"reporterOptions": { | ||
"mochaFile": "cypress/results/junit-[hash].xml", | ||
"overwrite": false, | ||
"html": false, | ||
"json": true | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
{ | ||
"name": "Using fixtures to represent data", | ||
"email": "hello@cypress.io", | ||
"body": "Fixtures are a great way to mock data for responses to routes" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
const testCases = [ | ||
["fixed image", "/images/fixed"], | ||
["fixed image smaller than requested size", "/images/fixed-too-big"], | ||
["fluid image", "/images/fluid"], | ||
["constrained image", "/images/constrained"], | ||
] | ||
|
||
const sizes = [["iphone-6"], ["ipad-2"], [1027, 768]] | ||
|
||
describe(`GatsbyImage`, () => { | ||
sizes.forEach(size => { | ||
testCases.forEach(([title, path]) => { | ||
describe(`${title}`, () => { | ||
it(`renders correctly on ${size.join("x")}`, () => { | ||
cy.viewport(...size) | ||
cy.visit(path) | ||
// Wait for main image to load | ||
cy.get("[data-main-image]").should("have.css", "opacity", "1") | ||
// Wait for blur-up | ||
cy.wait(1000) | ||
cy.get("#test-wrapper").matchImageSnapshot() | ||
}) | ||
}) | ||
}) | ||
}) | ||
}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
// *********************************************************** | ||
// This example plugins/index.js can be used to load plugins | ||
// | ||
// You can change the location of this file or turn off loading | ||
// the plugins file with the 'pluginsFile' configuration option. | ||
// | ||
// You can read more here: | ||
// https://on.cypress.io/plugins-guide | ||
// *********************************************************** | ||
|
||
// This function is called when a project is opened or re-opened (e.g. due to | ||
// the project's config changing) | ||
const { addMatchImageSnapshotPlugin } = require("cypress-image-snapshot/plugin") | ||
|
||
module.exports = (on, config) => { | ||
addMatchImageSnapshotPlugin(on, config) | ||
on("before:browser:launch", (browser = {}, launchOptions) => { | ||
if (browser.family === "chromium" || browser.family === "chrome") { | ||
// Make retina screens run at 1x density so they match the versions in CI | ||
launchOptions.push("--force-device-scale-factor=1") | ||
} | ||
return launchOptions | ||
}) | ||
} |
Binary file added
BIN
+577 KB
...e.js/GatsbyImage -- constrained image -- renders correctly on 1027x768.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+292 KB
...age.js/GatsbyImage -- constrained image -- renders correctly on ipad-2.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+79.2 KB
...e.js/GatsbyImage -- constrained image -- renders correctly on iphone-6.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+35.5 KB
...s/image.js/GatsbyImage -- fixed image -- renders correctly on 1027x768.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+35.5 KB
...ots/image.js/GatsbyImage -- fixed image -- renders correctly on ipad-2.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+35.5 KB
...s/image.js/GatsbyImage -- fixed image -- renders correctly on iphone-6.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+431 KB
...xed image smaller than requested size -- renders correctly on 1027x768.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+431 KB
...fixed image smaller than requested size -- renders correctly on ipad-2.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+297 KB
...xed image smaller than requested size -- renders correctly on iphone-6.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+577 KB
...s/image.js/GatsbyImage -- fluid image -- renders correctly on 1027x768.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+292 KB
...ots/image.js/GatsbyImage -- fluid image -- renders correctly on ipad-2.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+68.2 KB
...s/image.js/GatsbyImage -- fluid image -- renders correctly on iphone-6.snap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
// *********************************************************** | ||
// This example support/index.js is processed and | ||
// loaded automatically before your test files. | ||
// | ||
// This is a great place to put global configuration and | ||
// behavior that modifies Cypress. | ||
// | ||
// You can change the location of this file or turn off | ||
// automatically serving support files with the | ||
// 'supportFile' configuration option. | ||
// | ||
// You can read more here: | ||
// https://on.cypress.io/configuration | ||
// *********************************************************** | ||
|
||
import "gatsby-cypress" | ||
import { addMatchImageSnapshotCommand } from "cypress-image-snapshot/command" | ||
|
||
addMatchImageSnapshotCommand({ | ||
customDiffDir: `/__diff_output__`, | ||
customDiffConfig: { | ||
threshold: 0.1 | ||
}, | ||
failureThreshold: 0.03, | ||
failureThresholdType: `percent` | ||
}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
const path = require(`path`) | ||
|
||
module.exports = { | ||
siteMetadata: { | ||
title: `Gatsby Visual tests`, | ||
}, | ||
plugins: [ | ||
{ | ||
resolve: `gatsby-source-filesystem`, | ||
options: { | ||
name: `img`, | ||
path: `${__dirname}/src/images/`, | ||
}, | ||
}, | ||
`gatsby-plugin-image`, | ||
`gatsby-plugin-sharp`, | ||
`gatsby-transformer-sharp`, | ||
], | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
{ | ||
"name": "gatsby-starter-default", | ||
"description": "Gatsby default starter", | ||
"version": "1.0.0", | ||
"author": "Kyle Mathews <mathews.kyle@gmail.com>", | ||
"dependencies": { | ||
"cypress": "^3.1.0", | ||
"cypress-image-snapshot": "^3.1.1", | ||
"gatsby": "^2.0.118", | ||
"gatsby-plugin-image": "^0.0.2", | ||
"gatsby-plugin-sharp": "^2.0.20", | ||
"gatsby-source-filesystem": "^2.3.35", | ||
"gatsby-transformer-sharp": "^2.5.19", | ||
"react": "^16.8.0", | ||
"react-dom": "^16.8.0" | ||
}, | ||
"keywords": [ | ||
"gatsby" | ||
], | ||
"license": "MIT", | ||
"scripts": { | ||
"build": "gatsby build", | ||
"develop": "gatsby develop", | ||
"format": "prettier --write '**/*.js'", | ||
"test": "cross-env CYPRESS_SUPPORT=y npm run build && npm run start-server-and-test", | ||
"start-server-and-test": "start-server-and-test serve http://localhost:9000 cy:run", | ||
"serve": "gatsby serve", | ||
"cy:open": "cypress open", | ||
"cy:run": "cypress run --browser chrome", | ||
"cy:update-snapshots": "cypress run --browser chrome --env updateSnapshots=true", | ||
"cy:clean-snapshots": "rimraf cypress/snapshots/*" | ||
}, | ||
"devDependencies": { | ||
"cross-env": "^5.2.0", | ||
"cypress-junit-reporter": "^1.3.1", | ||
"gatsby-cypress": "0.4.11", | ||
"is-ci": "^2.0.0", | ||
"prettier": "2.0.4", | ||
"start-server-and-test": "^1.7.1" | ||
}, | ||
"repository": { | ||
"type": "git", | ||
"url": "https://github.com/gatsbyjs/gatsby-starter-default" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
import * as React from "react" | ||
import { Link } from "gatsby" | ||
import PropTypes from "prop-types" | ||
|
||
const Header = ({ siteTitle }) => ( | ||
<div | ||
style={{ | ||
background: `rebeccapurple`, | ||
marginBottom: `1.45rem`, | ||
}} | ||
> | ||
<div | ||
style={{ | ||
margin: `0 auto`, | ||
maxWidth: 960, | ||
padding: `1.45rem 1.0875rem`, | ||
}} | ||
> | ||
<h1 style={{ margin: 0 }}> | ||
<Link | ||
to="/" | ||
style={{ | ||
color: `white`, | ||
textDecoration: `none`, | ||
}} | ||
> | ||
{siteTitle} | ||
</Link> | ||
</h1> | ||
</div> | ||
</div> | ||
) | ||
|
||
Header.propTypes = { | ||
siteTitle: PropTypes.string.isRequired, | ||
} | ||
|
||
export default Header |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a quick note about running the suite using
yarn test
. It's the same as other test suites, but we have those instructions in most READMEs.