Skip to content

Latest commit

 

History

History
932 lines (648 loc) · 33 KB

test_instructions_per_dependency_update.md

File metadata and controls

932 lines (648 loc) · 33 KB
Raw

Test Instructions per Dependency Update

Purpose

The purpose of this document is to establish a set of guidelines for testing the app during dependency updates. Specifically, it outlines a series of test instructions that should be adhered to when updating a dependency.

FAQ

Could someone overlook testing a dependency update on a new screen?

This document focuses on a set of screens that are essential to the app's functionality. The concept is that, to ensure an update's safety, there's no need to test every screen using that dependency; testing the core ones should suffice.

How much effort will it take to maintain this document?

Engineers should only update this document if new core screens are added, potentially introducing regressions if left untested, or if an existing core screen is removed, rendering the dependency update redundant. If an engineer observes that a core screen has been updated, they should revise the testing instructions as part of the dependency update pull request. This makes keeping this document up-to-date a low-effort and minimally maintenance-intensive task.

Should this document be blindly trusted?

No engineer should blindly trust and strictly adhere to the testing instructions provided in this document. These instructions serve as a starting point and a reminder for engineers working on a dependency update to ensure they test the core screens. Engineers should exercise their best judgment and conduct additional testing if they deem it necessary.

Do we need testing instructions for every dependency update?

Some dependency updates are quite broad and apply to the entire app, making it challenging to provide specific testing instructions (e.g., fragment, appcompat, compose, etc.). In such cases, engineers should use their best judgment to perform a smoke test on the entire app to ensure its correctness. Engineers should rely on their intuition to identify which dependencies warrant testing instructions. Therefore, this document should be regarded as a set of helpful guidelines rather than strict requirements.

Content

  1. Plugin
    1. Navigation
  2. AndroidX Jetpack
    1. WorkManager
  3. AndroidX/Google
    1. Preference
    2. ExoPlayer
    3. Webkit
  4. Firebase/Google
    1. Firebase
    2. FirebaseIid
    3. MLKitBarcodeScanning
    4. PlayServicesAuth
    5. PlayServicesCoreScanner
    6. PlayReview
  5. Tool
    1. Zendesk
    2. JSoup
  6. Other Core
    1. AutoService
    2. KotlinPoet
  7. Other UI
    1. Lottie
    2. UCrop
  8. Smoke Test
  9. Special

ℹ️ Every test instruction should be prefixed with one of the following:

  • [JP/WP] This test applies to both, the Jetpack and WordPress apps.
  • [JP] This test applies to the Jetpack app only.
  • [WP] This test applies to the WordPress app only.

Navigation [navigationVersion]

1. [JP/WP] Image Editing Flow [libs:image-editor]
  • Add a new blog post.
  • Add a new image block.
  • Choose an image and wait for it to be uploaded within the image block.
  • Click on the media options of this image (top right) and then click edit.
  • Verify that the Edit Image screen is shown and functioning as expected.
  • Crop the image and click the done menu option (top right).
  • Make sure the image is updated accordingly.

WorkManager [androidxWorkManagerVersion]

1. [JP/WP] UploadWorker.kt
  • Go to Post screen.
  • Create a new post and publish it.
  • Turn device offline.
  • Go to this post and update it.
  • Notice the warning message: We'll publish the post when your device is back online.
  • Turn device online.
  • Notice this post being automatically uploaded.
  • Open this post on a web browser and verify the post is indeed updated.
2. [JP] ReminderWorker.kt
  • Go to Site Settings screen.
  • Find the Blogging section, click on Reminders, toggle-on every day and click on Update.
  • Notice the All set! bottom sheet appearing, click Done.
  • Close the app, preferably swipe the app off.
  • Go to the device's Settings app, find the Date & Time section, turn Automatic date & time off.
  • Set the device's date to a day after today.
  • Open the app.
  • Verify the Blogging Reminders notification appearing. For example, the notification title could be Daily Prompt, while the notification description something like Is there anything you feel too old to do anymore?.
3. [JP] LocalNotificationWorker.kt
  • TODO
  • TODO
  • TODO
4. [JP] WeeklyRoundupWorker.kt
  • TODO
  • TODO
  • TODO

Preference (AndroidX) [androidxPreferenceVersion]

1. [JP] Notifications Settings [NotificationsSettingsFragment.java]
  • Go to Notifications tab.
  • Click on the Gear setting button (top-right).
  • Verify that the Notification Settings screen is displayed.
  • Click on each of the settings within the Notification Settings settings screen and verify that every setting works as expected.

Preference (Android) [N/A]

ℹ️ Note that this is not an actual dependency, it being part of the Android SDK, that it is deprecated and will be migrated to its AndroidX equivalent in the future. [Issue]

1. [JP/WP] Account Settings [AccountSettingsFragment.kt]
  • Go to Me tab.
  • Click on the Account Settings button.
  • Verify that the Account Settings screen is displayed.
  • Click on each of the settings within the Account Settings screen and verify that every setting works as expected.
2. [JP/WP] App Settings [AppSettingsFragment.java]
  • Go to Me tab.
  • Click on the App Settings button.
  • Verify that the App Settings screen is displayed.
  • Click on each of the settings within the App Settings screen and verify that every setting works as expected, including the inner settings like the Privacy Settings.
  • Do the same for the Debug Settings screens.
3. [JP/WP] Site Settings [SiteSettingsFragment.java]
  • Go to Site Settings screen.
  • Verify that the Site Settings screen is displayed.
  • Click on each of the settings within the Site Settings screen and verify that every setting works as expected.
4. [JP/WP] Jetpack Settings - Security [JetpackSecuritySettingsFragment.java]

ℹ️ Prerequisite: To have this setting displayed you must have a Jetpack connected site.

  • Go to Site Settings screen.
  • Find the Jetpack Settings section and click on Security.
  • Verify that the Security setting screen is displayed.
  • Click on each of the settings within the Security settings screen and verify that every setting works as expected.
5. [JP] Notifications Settings [NotificationsSettingsFragment.java]
  • Go to Notifications tab.
  • Click on the Gear setting button (top-right).
  • Verify that the Notification Settings screen is displayed.
  • Click on each of the settings within the Notification Settings settings screen and verify that every setting works as expected.
6. [JP/WP] Edit Post Screen [EditPostActivity.java]

ℹ️ Editing a new post uses PreferenceManager to setDefaultValues(...) for Account Settings.

  • Go to Post screen.
  • Edit a new post, add a few of the main blocks and verify that everything is workings as expected.

ExoPlayer [googleExoPlayerVersion]

1. [JP/WP] Image Preview
  • Go to Post screen.
  • Edit a new post, add an Image block and then an image into it.
  • Having added an image to the image block, click on the image block and then again on the image within.
  • This will launch Media Preview screen and the image should be previewed.
  • Verify that the Image Preview is working as expected, that rotating the device works and finally that clicking back navigates you back to the post edit screen.
2. [JP/WP] Video Preview
  • Go to Post screen.
  • Edit a new post, add an Video block but don't add a video into it just yet.
  • Find the Video block, click on ADD VIDEO and then Choose from device to choose a video.
  • From the list of available videos to choose from screen, click on the Play button on the center on any video. FYI: If you don't click on the center, this video will get selected for use, but it won't play.
  • This will launch Media Preview screen and the video should start playing.
  • Verify that the Video Preview is working as expected, that rotating the device works and finally that clicking back navigates you back to the list of available videos to choose from screen.

Webkit [androidxWebkitVersion]

1. [TODO] TODO
  • TODO
  • TODO
  • TODO

Firebase [firebaseBomVersion]

1. [JP/WP] Push Notification on Post Being Published [GCMMessageService.java]
  • Add a new blog post.
  • Add any number of blocks.
  • Publish the post.
  • Verify that, on post being published, you get a push notification with the post's title, which, when you click on it, navigates you to the 'Notifications Detail' screen for this post.

FirebaseIid [firebaseIidVersion]

1. [TODO] TODO
  • TODO
  • TODO
  • TODO

MLKitBarcodeScanning [googleMLKitBarcodeScanningVersion]

1. [TODO] TODO
  • TODO
  • TODO
  • TODO

PlayServicesAuth [googlePlayServicesAuthVersion]

ℹ️ These set of testing instructions are for internal contributors only, which can have access to upload an .aab to Google Play Store. As such, external contributors can't possibly test this update.

1. [JP] Google Sign-in on Jetpack

ℹ️ Prerequisite: If you don't have access to upload an .aab to Google Play Store for Jetpack - Website Builder, post a request to +systemsrequests.

  • Check-out this branch and edit the version.properties file to pick a dummy versionName and versionCode, preferable using a higher versionCode to the existing one.
    • I recommend using a versionCode of 1000000001 and above.
    • I recommend using an explicit versionName like deps-update-google-play-services-20-4-1 (similar to the name of this branch itself)
  • Build the release .aab locally using this command: ./gradlew bundleJetpackVanillaRelease
  • Upload the locally generated .aab, found within the WordPress/build/outputs/bundle/jetpackVanillaRelease/ build folder, and into Google Play Console (Jetpack - Website Builder) via: App Bundle Explorer > Upload new version (top-right)
    • Wait for the upload to complete, then via this newly uploaded app version, navigation to the Downloads, on the Assets section, find the Signed, universal APK and click the Download icon (middle-right) to download the signed .apk locally.
    • After having this signed .apk downloaded locally, go ahead and click on the Delete app bundle to delete the this app bundle from the list of available app bundles.
  • Install this download signed .apk using this command: adb install -r Downloads/jpandroid-deps-update-google-play-services-20-4-1-Signed.apk (note that you can use ./tools/rename_apk_aab.sh to rename the download signed .apk)
  • If you were already logged-in to Jetpack, log-out and log-in again using the Continue with Google button (note that you need to use a non A8C account to be able to complete this flow).
  • Verify that the Google sign-in works, that you have successfully logged-in and are able to use the app as expected.
2. [WP] Google Sign-in on WordPress

ℹ️ Prerequisite: If you don't have access to upload an .aab to Google Play Store for WordPress - Website Builder, post a request to +systemsrequests.

  • Check-out this branch and edit the version.properties file to pick a dummy versionName and versionCode, preferable using a higher versionCode to the existing one.
    • I recommend using a versionCode of 1000000001 and above.
    • I recommend using an explicit versionName like deps-update-google-play-services-20-4-1 (similar to the name of this branch itself)
  • Build the release .aab locally using this command: ./gradlew bundleWordPressVanillaRelease
  • Upload the locally generated .aab, found within the WordPress/build/outputs/bundle/wordpressVanillaRelease/ build folder, and into Google Play Console (WordPress - Website Builder) via: App Bundle Explorer > Upload new version (top-right)
    • Wait for the upload to complete, then via this newly uploaded app version, navigation to the Downloads, on the Assets section, find the Signed, universal APK and click the Download icon (middle-right) to download the signed .apk locally.
    • After having this signed .apk downloaded locally, go ahead and click on the Delete app bundle to delete the this app bundle from the list of available app bundles.
  • Install this download signed .apk using this command: adb install -r Downloads/wpandroid-deps-update-google-play-services-20-4-1-Signed.apk (note that you can use ./tools/rename_apk_aab.sh to rename the download signed .apk)
  • If you were already logged-in to WordPress, log-out and log-in again using the Continue with Google button (note that you need to use a non A8C account to be able to complete this flow).
  • Verify that the Google sign-in works, that you have successfully logged-in and are able to use the app as expected.

PlayServicesCodeScanner [googlePlayServicesCodeScannerVersion]

1. [JP] Scan Login Code

Step.1:

  • Build and install the Jetpack app (note that you don't need a release build, a debug build will suffice).
  • Log in to the Jetpack app with a WP.com account (note that you need to use a non A8C account and a non 2FA enabled account).
  • Navigate to the Me screen (click on avatar at top-right).
  • (STOP)

Step.2:

  • Head over to your desktop and open a web browser (note that using an incognito tab works best).
  • Browse to wordpress.com (note that if you are logged-in, log-out first).
  • Tap the Log In link (top-right).
  • Tap the Login via the mobile app link in the list of options below the main Continue button (bottom-middle).
  • Verify you are on the Login via the mobile app view and Use QR Code to login is shown, along with a QR code for you to scan.
  • (STOP)

Step.3:

  • Head back to your mobile.
  • Tap the Scan Login Code item on the Me screen you are currently at.
  • Scan the QR code on the web browser.
  • Follow the remaining prompts on your mobile to log in to WordPress on your web browser (desktop), verify that you have successfully logged-in and are able to use WordPress as expected.

PlayReview [googlePlayReviewVersion]

1. In app reviews
  • Perform a clean install.
  • Publish three (AppReviewManager.TARGET_COUNT_POST_PUBLISHED + 1) new posts or stories.
  • Verify that there are no crashes.

Zendesk [zendeskVersion]

1. [JP] Zendesk Update [Tickets & Contact Support]

ℹ️ This test only works when testing with a normal, non-a8c user account.

  • Go to Me tab -> Help -> Tickets.
  • Verify that the My tickets Zendesk related screen opens and that it lists all of your tickets (or non if you don't have any yet).
  • Go back to the Help settings screen and then click on Contact Support.
  • Verify that the Help Zendesk related screen opens and that you are prompt to Send a message....
  • Type Testing Zendesk X.Y.Z update, please ignore and thank you! and then send your message.
  • After sending the first message, tap the Contact support 📢 button to create the support ticket.
  • Go back to the Help settings screen and then click on Tickets again.
  • Verify that your previously sent message is listed in there and that you can click on it. PS: You could also check your emails and verify that you got a Thank you for contacting the WordPress.com support team! (#1234567) email.
  • Verify that clicking on it navigates you to the inner screen where your message is shown as Delivered along with a predefined automated mobile support response message.

JSoup [jsoupVersion]

1. [JP/WP] Classic Editor [Aztec]

ℹ️ Disable the Block Editor before testing this. You can do so by going to Site Settings -> Editor section -> Use Block Editor option and switch that off. 📝 Note that you might need to be on a business plan to have this Editor section available as not all sites have this option.

  • Go to Posts screen and create a new post.
  • Add an Image, Video and any other entry (like Paragraph, Ordered List, Quote, etc).
  • Publish this newly created post.
  • Verify that this newly created Classic Editor related post of yours is being displayed as expected, both when previewing it from within the Posts and Reader screens. FYI: On Reader, you will find you post within the FOLLOWING tab.
2. [JP/WP] Block Editor [Gutenberg]

ℹ️ Enable the Block Editor before testing this. You can do so by going to Site Settings -> Editor section -> Use Block Editor option and switch that on. 📝 Note that you might need to be on a business plan to have this Editor section available as not all sites have this option.

  • Go to Posts screen and create a new post.
  • Add an Image, Gallery, Video, Audio, File, Media Text and Cover blocks to the post.
  • Publish this newly created post.
  • Verify that this newly created Block Editor related post of yours is being displayed as expected, both when previewing it from within the Posts and Reader screens. FYI: On Reader, you will find you post within the FOLLOWING tab.
3. [JP] Reader Post Details [ReaderPostRenderer.java]
  • Go to Reader screen and click on various posts.
  • Verify that each and every post, along with all their details is being displayed as expected.
4. [JP] Stats Insights - Latest Post Summary [LatestPostSummaryMapper.kt]

ℹ️ If the Latest Post Summary card is not being displayed, navigate to the bottom of the Stats screen and click on the Add new stats card. Then, enabled the Latest Post Summary from within the Posts and Pages group.

  • Go to Stats screen and its INSIGHTS tab.
  • Scroll to the Latest Post Summary card and verify that it is being displayed as expected.

AutoService [googleAutoServiceVersion]

1. [JP/WP] Remote Features [@Feature]

️️ℹ️ Just test the most recent remote feature flag.

2. [JP/WP] Features in Development [@FeatureInDevelopment]

ℹ️ Just test the most recent feature flag in development.

3. [JP/WP] Remote Field Configs [@RemoteFieldDefaultGenerater]

ℹ️ Just test the most recent remote remote field configs.

4. [JP/WP] Experimental Features [@Experiment]

ℹ️ Just test the most recent experimental feature.

KotlinPoet [squareupKotlinPoetVersion]

1. [JP/WP] Remote Features [@Feature]

️️ℹ️ Just test the most recent remote feature flag.

2. [JP/WP] Features in Development [@FeatureInDevelopment]

ℹ️ Just test the most recent feature flag in development.

3. [JP/WP] Remote Field Configs [@RemoteFieldDefaultGenerater]

ℹ️ Just test the most recent remote remote field configs.

4. [JP/WP] Experimental Features [@Experiment]

ℹ️ Just test the most recent experimental feature.

Lottie [lottieVersion]

1. [JP/WP] JP Install Full Plugin Animation [JPInstallFullPluginAnimation.kt]
  • TODO
  • TODO
  • TODO
1. [WP] Jetpack Static Poster [JetpackStaticPoster.kt]
  • TODO
  • TODO
  • TODO
1. [JP] Notifications Screen [NotificationsDetailListFragment.kt]
  • TODO
  • TODO
  • TODO

UCrop [uCropVersion]

1. [JP/WP] Image Edit Screen [PreviewImageFragment.kt + CropFragment.kt]
  • Add a new blog post.
  • Add a new image block.
  • Choose an image and wait for it to be uploaded within the image block.
  • Click on the media options of this image (top right) and then click edit.
  • Crop the image and click the done menu option (top right).
  • Verify the image is updated accordingly.

Smoke Test

  1. Plugin
    1. GoogleServices
    2. Dagger
  2. Kotlin
    1. Kotlin
    2. Coroutines
  3. EventBus
    1. EventBus
  4. AndroidX Core
    1. Annotation
    2. ArchCore
    3. Lifecycle
    4. Core
    5. Activity
    6. Fragment
    7. AppCompat
  5. AndroidX Compose
    1. Compose
    2. ComposeMaterial3
    3. ComposeConstraintLayout
    4. Coil
  6. AndroidX/Google
    1. RecyclerView
    2. ViewPager2
  7. Material/Google
    1. Material
    2. ConstraintLayout
  8. Network
    1. Retrofit
    2. Volley
    3. Glide
    4. Gson
  9. Tool
    1. InstallReferrer
  10. Other Core
    1. ApacheCommons
  11. Other
    1. Desugar

ℹ️ Some smoke test instructions might have an Extra section with addition instructions.

GoogleServices [googleServicesVersion]

Dagger [daggerVersion]

Kotlin [kotlinVersion] + [androidxComposeCompilerVersion]

Coroutines [kotlinxCoroutinesVersion]

EventBus [eventBusVersion]

Annotation [androidxAnnotationVersion]

ArchCore [androidxArchCoreVersion]

Lifecycle [androidxLifecycleVersion]

Core [androidxCoreVersion]

Activity [androidxActivityVersion]

Fragment [androidxFragmentVersion]

AppCompat [androidxAppcompatVersion]

Compose [androidxComposeBomVersion]

ComposeMaterial3 [androidxComposeMaterial3Version]

ComposeConstraintLayout [androidxConstraintlayoutComposeVersion]

Coil [coilComposeVersion]

RecyclerView [androidxRecyclerviewVersion]

ViewPager2 [androidxViewpager2Version]

Material [googleMaterialVersion]

ConstraintLayout [androidxConstraintlayoutVersion]

Retrofit [squareupRetrofitVersion]

Volley [androidVolleyVersion]

Glide [glideVersion] + [glideVolleyVersion]

Gson [googleGsonVersion]

InstallReferrer [androidInstallReferrerVersion]

ApacheCommons [apacheCommonsTextVersion]

Desugar [androidDesugarVersion]

Extra

Focus on testing push notifications and/or blogging reminders since this was why this library was introduced to this project in the first place (see here and here).

Special

  1. Gradle
    1. Gradle
  2. Plugin
    1. AGP
    2. Sentry
    3. Detekt
    4. ViolationComments
  3. React Native
    1. ReactNative

ℹ️ Every special test instructions have a Why & Extra section to further explain why that dependency update is so special and how to deal with it.

Gradle [gradleWrapper]

Why & How
  • TODO
  • TODO
  • TODO

AGP [agpVersion]

Why & How
  • TODO
  • TODO
  • TODO

Sentry [sentryVersion]

Why & How

sentryVersion in this project relates to Sentry Gradle Plugin only. Sentry SDK is bundled with Automattic-Tracks-Android.

Why?

We use Sentry Gradle Plugin to send ProGuard mapping files and source context files to Sentry. It makes stacktrace readable on Sentry dashboard. This should be the main focus when testing after bumping sentryVersion.

To Test

Please build the release variant (vanillaRelease) of both WordPress and Jetpack flavors and verify if issues are sent correctly. You can use the following snippet.

PATCH (warning: it'll probably have some conflicts in the future when `WPMainActivityViewModel` change. It's more for an idea: 
Subject: [PATCH] tests: add a test for features in development generation
---
Index: WordPress/src/main/java/org/wordpress/android/viewmodel/main/WPMainActivityViewModel.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/WordPress/src/main/java/org/wordpress/android/viewmodel/main/WPMainActivityViewModel.kt b/WordPress/src/main/java/org/wordpress/android/viewmodel/main/WPMainActivityViewModel.kt
--- a/WordPress/src/main/java/org/wordpress/android/viewmodel/main/WPMainActivityViewModel.kt	(revision 806913d9fb807250cecd5b24b36001d55ea4c255)
+++ b/WordPress/src/main/java/org/wordpress/android/viewmodel/main/WPMainActivityViewModel.kt	(date 1710772966823)
@@ -5,6 +5,7 @@
 import androidx.lifecycle.LiveData
 import androidx.lifecycle.MutableLiveData
 import androidx.lifecycle.distinctUntilChanged
+import com.automattic.android.tracks.crashlogging.CrashLogging
 import kotlinx.coroutines.CoroutineDispatcher
 import kotlinx.coroutines.flow.firstOrNull
 import org.wordpress.android.R
@@ -67,6 +68,7 @@
     private val bloggingPromptsStore: BloggingPromptsStore,
     @Named(UI_THREAD) private val mainDispatcher: CoroutineDispatcher,
     private val shouldAskPrivacyConsent: ShouldAskPrivacyConsent,
+    private val crashLogging: CrashLogging,
 ) : ScopedViewModel(mainDispatcher) {
     private var isStarted = false

@@ -161,6 +163,7 @@
         launch { loadMainActions(site) }

         updateFeatureAnnouncements()
+        crashLogging.sendReport(Throwable("Test crash"))
     }

     @Suppress("LongMethod")

Detekt [detektVersion]

Why & How
  • TODO
  • TODO
  • TODO

ViolationComments [violationCommentsVersion]

Why & How
  • TODO
  • TODO
  • TODO

ReactNative [facebookReactVersion]

Why & How
  • TODO
  • TODO
  • TODO