Add support for Android Auto

[haggaie]

What is it?

• Bugfix (user facing)
• Feature (user facing)
• Codebase improvement (dev facing)
• Meta improvement to the project (dev facing)

Description of the changes in your PR

• Expose media controls and playback status on Android Auto, as well as a basic browsing interface, showing playlists, history, and supporting basic searches.

Issues, bugs, and missing features

• Items (playlists and streams) are not updated in Android Auto when they are in the app.
• When closing the player from its mini view, playback continues. It should be stopped instead.
• When starting playback on Android Auto then opening the app, the mini player doesn't appear and isn't synchronized with the current state when "opened" (opening the details page of a content with autoplay disabled, pressing the play button plays the content on the details page)
• Remote playlists are limited to their first page.
• when starting the app then connecting to an Desktop Head Unit. The DHU was blocked on the loading step happening before showing playback controls normally, the app was playing the audio in the meantime then the app crashed.
• Non square thumbnails seems to forced to the square format in the miniplayer commands if you enable the setting of the Desktop Head Unit.
• Integration with voice search - voice search from within the app works, but search from the google assistant (e.g., "play on NewPipe") only supports applications from the Google Play Store.

Future enhancements

• Show progress bars for played items and for the one currently playing, if applicable (see https://developer.android.com/training/cars/media?hl=en#browse-progress-bar for the latter).
• Add more content to Android Auto.

Screenshots/Screen Record

Fixes the following issue(s)

• Fixes [Feature Request] Compatibility with Android Auto #1758

APK testing

The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR.

Due diligence

• I read the contribution guidelines.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[Stypox]

Interesting! Is it possible that using MediaBrowserServiceCompat along with setSessionToken might solve some issues related to the notification? Also, from the docs of MediaBrowserServiceCompat, it enables "applications to browse media content provided by an application". Do you intend to implement anything related to browsing content, or do you want to pretend there is no browasable content for simplicity and just implement media controls? Also, since the service is started whenever external applications need to browse media, isn't it possible that the player starts at random?

[haggaie]

Interesting! Is it possible that using MediaBrowserServiceCompat along with setSessionToken might solve some issues related to the notification?

I'm not familiar with these issues, but I can try testing them if they are reproduced with this patch. Can you point me in their direction?

Also, from the docs of MediaBrowserServiceCompat, it enables "applications to browse media content provided by an application". Do you intend to implement anything related to browsing content, or do you want to pretend there is no browasable content for simplicity and just implement media controls?

I wanted to start with media controls, because it seems useful in its own. I can set a play queue in advance and control it from the car. Later I would also like to add see browsing. I think that for Auto it should be limited - maybe things like saved playlists and search history? There's also an API for search queries I am hoping would allow integration with voice search.

Also, since the service is started whenever external applications need to browse media, isn't it possible that the player starts at random?

Yes, I think the documentation had something to say about it - that the service shouldn't start playing when it starts. It should only start playing when you get a callback from the MediaBrowserServiceCompat. But the service can tell how if it gets started by another UI or by a media browser, can't it?

I had a related question: at first I thought it would be better to create a separate service. I then saw that it is easier to use the existing player service, but do you think it is the right approach?

Another small problem I noticed with the pull request in its current state, is that the Android Auto's rendering of the media session notification shows the progress circle around the pause button in gray, rather than showing the current position. This is weird, because the notification on my phone does show a seekable progress bar.

[Stypox]

Can you point me in their direction?
the media session notification shows the progress circle around the pause button in gray, rather than showing the current position

You actually answered my question: the issue with the notification I was referring to is the one related to no seek bar showing up. And apparently no, setSessionToken does not fix the issue magically.

I wanted to start with media controls, because it seems useful in its own.

Yeah that's fine, you can take care of the rest later if you want. Integration with voice search would also be really nice.

But the service can tell how if it gets started by another UI or by a media browser, can't it?

Yes it can, but atm I think that part is broken, because there are user reports of the player popping up from nowhere or crashing at random when connecting or disconnecting bluetooth.

I had a related question: at first I thought it would be better to create a separate service. I then saw that it is easier to use the existing player service, but do you think it is the right approach?

Here are a few considerations:

• player code is already pretty messy, but maybe the PlayerService is not too messy
• when you do setSessionToken, you need an instance of the MediaBrowserServiceCompat, and it might be difficult to get hold of it if it's an external service (I guess it would require having two services running at the same time?)

So I think all in all it's best to keep it in PlayerService, but try to put as few code there as possible in order not to clutter the class (i.e. prefer creating utility classes).

[haggaie]

I'm trying to implement some simple browsing, but I got into lifecycle issues. The MediaBrowserServiceCompat is supposed to have a call to setSessionToken in onCreate (and expect it to be called only once), and I can solve this by creating the MediaSession in MediaSessionPlayerUi's constructor. However, if I close the notification, it causes the player service to clean up, but if I understand correctly, PlayerService remains alive because of the Android Auto media browser. Then if I try to play anything again, the service crashes because it has already destroyed its Player.

How can I handle this? Perhaps closing the notification shouldn't close the entire player, and instead, stop and close the notification UI?

[Stypox]

Perhaps closing the notification shouldn't close the entire player, and instead, stop and close the notification UI?

Closing the notification should definitely release the player. Afterwards any incoming intent to start the player should create a new player, while other intents should be treated separately (and NOT start the player). But it's a big mess. If you can't find out a way to do this, maybe go with the option of creating a separate service.

[haggaie]

Closing the notification should definitely release the player. Afterwards any incoming intent to start the player should create a new player, while other intents should be treated separately (and NOT start the player).

Makes sense, I tried doing something along these lines, though I don't differentiate between the intents yet. The code also still creates the player when the service is created, but I can change it as you wrote.

I also tried adding some browsing capabilities, but for some reason it only worked on the desktop HUD and not with my car. I'll have to check that out.

[goyalyashpal]

Android Auto's rendering of the media session notification shows the progress circle around the pause button in gray, rather than showing the current position.
- #9592 (comment)

• if i recall correct, then i don't think that android auto's rendering shows the seek bar in any player
  (and for good reasons - u don't want to even give a chance to driver to fiddle with seekbars)
• Have you tried using other players on android auto to verify? like youtube or via bluetooth?

---

OT: google also bought the "waze navigation" map app you are using, but they just don't show it anywhere as then, users will start abandoning even waze.

https://edition.cnn.com/2022/12/08/tech/google-waze-maps-merge/index.html
In February 2021, Waze’s former top executive Noam Bardin said it struggled to grow within Google and that Waze could have “probably grown faster and much more efficiently had we stayed independent.”

google and others are fiercely anti-competitive and purchase any company doing better than them. Remeber, Triple EEEs is their motto.

[haggaie]

• if i recall correct, then i don't think that android auto's rendering shows the seek bar in any player
  (and for good reasons - u don't want to even give a chance to driver to fiddle with seekbars)
• Have you tried using other players on android auto to verify? like youtube or via bluetooth?

Yes, that's what I've seen too. In Spotify for example the progress is shown as a circle that can't be manipulated, rather than a seek bar. Still, the circle shows playback progress with two colors. I'll try capturing a screenshot sometime.

OT: google also bought the "waze navigation" map app

I know, that's too bad because I liked Waze. Up until now, Google haven't merged it, even if they haven't grown it much. I think it was pretty good at predicting traffic, which is something that requires a popular service, so I might end up with Google Maps if they merge them.

[goyalyashpal]

the circle shows playback progress with two colors

oh yeah, that thin two coloring of the circle's border stroke, right? i can imagine that.

---

OT:

I know, that's too bad because I liked Waze.

same, i found this too while searching for alternates to google maps some years ago.
Anyways, I heard several other companies have come together to form a geolocation alliance or smth like that... oh yeah "Overture Maps Foundation" - article by TechCrunch, referred to by Nick (from TLE) in news video for '22 Dec W3

[haggaie]

Hi, I've updated the pull request and tried to keep the MediaSession and its connector in the new helper object so that they outlive the player.

I also tried making the media Ids more URI-like, inspired by VLC.

By the way, I also tried implementing the APIs for voice search (I think it should just be onPlayFromSearch/onPrepareFromSearch), but it didn't work. As far as I understand, Google Assistant only works with apps from the Google Play Store. I can't even invoke "open NewPipe", and this shouldn't require any code.

[aedwards00]

Apologies for my unfamiliarity with your development process, but is this likely to be included in the next release?

It's a feature I'm very much interested in and looks to the untrained eye like it's ready to be merged?

[AudricV]

Thank you very much for this feature!

Please apply the following changes and rebase your branch on top of the dev one, so we can test the changes on the latest release. I am trusting you for your usage and your implementation of the feature.

However, there is an issue with this PR: the single ExoPlayer library is being discontinued, as the library has now its place in the Media3 one.

This change removes the media session extension, as media session management is directly done by Media3, so this code would have to be rewritten soon if and when we migrate to Media3 the app.

Even if this migration may happen soon, I don't think we will migrate to Media3 in the next release and I think we can merge this pull request, especially because of the feature it gives to the app.

Finally, I see several changes in this PR related to the player service, especially on command interception. Wouldn't your changes fix the player crash issues we currently have when using a media command on a Bluetooth device and maybe the player ANRs?

[haggaie]

Thanks for your feedback. I'm trying to rebase, but I came across a build issue with the updated dev branch. I'm getting the following error:

> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
   > Could not find com.github.TeamNewPipe.NewPipeExtractor:extractor:v0.22.6.
     Searched in the following locations:
       - https://dl.google.com/dl/android/maven2/com/github/TeamNewPipe/NewPipeExtractor/extractor/v0.22.6/extractor-v0.22.6.pom
       - https://repo.maven.apache.org/maven2/com/github/TeamNewPipe/NewPipeExtractor/extractor/v0.22.6/extractor-v0.22.6.pom
       - https://jitpack.io/com/github/TeamNewPipe/NewPipeExtractor/extractor/v0.22.6/extractor-v0.22.6.pom
       - https://repo.clojars.org/com/github/TeamNewPipe/NewPipeExtractor/extractor/v0.22.6/extractor-v0.22.6.pom
     Required by:
         project :app > com.github.TeamNewPipe:NewPipeExtractor:v0.22.6
   > Could not find com.github.TeamNewPipe.NewPipeExtractor:NewPipeExtractor:v0.22.6.
     Searched in the following locations:
       - https://dl.google.com/dl/android/maven2/com/github/TeamNewPipe/NewPipeExtractor/NewPipeExtractor/v0.22.6/NewPipeExtractor-v0.22.6.pom
       - https://repo.maven.apache.org/maven2/com/github/TeamNewPipe/NewPipeExtractor/NewPipeExtractor/v0.22.6/NewPipeExtractor-v0.22.6.pom
       - https://jitpack.io/com/github/TeamNewPipe/NewPipeExtractor/NewPipeExtractor/v0.22.6/NewPipeExtractor-v0.22.6.pom
       - https://repo.clojars.org/com/github/TeamNewPipe/NewPipeExtractor/NewPipeExtractor/v0.22.6/NewPipeExtractor-v0.22.6.pom
     Required by:
         project :app > com.github.TeamNewPipe:NewPipeExtractor:v0.22.6
   > Could not find com.github.TeamNewPipe.NewPipeExtractor:timeago-parser:v0.22.6.
     Searched in the following locations:
       - https://dl.google.com/dl/android/maven2/com/github/TeamNewPipe/NewPipeExtractor/timeago-parser/v0.22.6/timeago-parser-v0.22.6.pom
       - https://repo.maven.apache.org/maven2/com/github/TeamNewPipe/NewPipeExtractor/timeago-parser/v0.22.6/timeago-parser-v0.22.6.pom
       - https://jitpack.io/com/github/TeamNewPipe/NewPipeExtractor/timeago-parser/v0.22.6/timeago-parser-v0.22.6.pom
       - https://repo.clojars.org/com/github/TeamNewPipe/NewPipeExtractor/timeago-parser/v0.22.6/timeago-parser-v0.22.6.pom
     Required by:
         project :app > com.github.TeamNewPipe:NewPipeExtractor:v0.22.6

Do you know what causes this?

Regarding moving to media3, I understand it can make most of this code obsolete. Once you are done with the move to media3, I could try to test its support for android auto and see when changes are needed.

[AudricV]

Do you know what causes this?

The recurrent Jitpack artifact Not found issue we have with NewPipe Extractor, update temporarily the extractor on your side to test the changes. We will try to switch soon from Jitpack to something else for NewPipe Extractor and our nanojson fork.

[haggaie]

Finally, I see several changes in this PR related to the player service, especially on command interception. Wouldn't your changes fix the player crash issues we currently have when using a media command on a Bluetooth device and maybe the player ANRs?

Is there an issue about this I can read? Maybe I could try and reproduce this problem and see if my patches help.

[AudricV]

#9095, #9390 for both issues and probably more which may be not listed as duplicates of them.

[haggaie]

#9095, #9390 for both issues and probably more which may be not listed as duplicates of them.

I tried reproducing #9095, and I believe it still happens with this PR. I tried using my car's console without Android Auto, only functioning as a Bluetooth headset. If I play a video on NewPipe, close it, and press "Play" on my car's console, the application crashes.

I got the following in the crash log:

java.lang.RuntimeException: Unable to start service org.schabi.newpipe.player.PlayerService@27e84bf with Intent { act=android.intent.action.MEDIA_BUTTON flg=0x10000010 cmp=org.schabi.newpipe.debug.androidauto/org.schabi.newpipe.player.PlayerService (has extras) }: java.lang.NullPointerException: Attempt to invoke virtual method 'org.schabi.newpipe.player.playqueue.PlayQueue org.schabi.newpipe.player.Player.getPlayQueue()' on a null object reference
    at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:4670)
    at android.app.ActivityThread.-$$Nest$mhandleServiceArgs(Unknown Source:0)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2176)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loopOnce(Looper.java:201)
    at android.os.Looper.loop(Looper.java:288)
    at android.app.ActivityThread.main(ActivityThread.java:7884)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)
  Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'org.schabi.newpipe.player.playqueue.PlayQueue org.schabi.newpipe.player.Player.getPlayQueue()' on a null object reference
    at org.schabi.newpipe.player.PlayerService.onStartCommand(PlayerService.java:90)
    at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:4652)
    ... 9 more

So I think closing the player must have left the service running but with the player object null. I haven't tested the dev branch for the same error, so this crash may be due to my changes and not the original one reported in #9095.

[snaik20]

@haggaie great to see this PR active again. I would love to review final changes. I have some updates in mind but will wait for your changes to finalize first.

[haggaie]

@haggaie great to see this PR active again. I would love to review final changes. I have some updates in mind but will wait for your changes to finalize first.

Thanks @snaik20, I think the code is in a good place to finalize, at least in terms of features. I fixed the tab color issue, added remote playlists to the playlist tab, and added a search function. I don't think there is other functionality missing for a first PR of this feature. Of course, there's always room for improvement, and I imagine the implementation could be made cleaner by integrating it deeper into the rest of NewPipe. For that, I'm sure I'll need feedback from the codebase's experts.

[snaik20]

I have some updates in mind but will wait for your changes to finalize first.
- @ snaik20 at #9592 (comment)

gentle ping 😃 @snaik20

Ahh, got busy elsewhere. Will get to it today, thanks.

[snaik20]

@haggaie great to see this PR active again. I would love to review final changes. I have some updates in mind but will wait for your changes to finalize first.

Thanks @snaik20, I think the code is in a good place to finalize, at least in terms of features. I fixed the tab color issue, added remote playlists to the playlist tab, and added a search function. I don't think there is other functionality missing for a first PR of this feature. Of course, there's always room for improvement, and I imagine the implementation could be made cleaner by integrating it deeper into the rest of NewPipe. For that, I'm sure I'll need feedback from the codebase's experts.

Sorry for the delay. I have gone through the changes. Looking great. I just updated the PR with some kotlin conversions for new files and fixed own comments.

I have not been able to test the PR though. Feel free to update my changes if found any issues in testing.

[haggaie]

Sorry for the delay. I have gone through the changes. Looking great. I just updated the PR with some kotlin conversions for new files and fixed own comments.

I have not been able to test the PR though. Feel free to update my changes if found any issues in testing.

No problem, thanks for the review. I've tested it locally (with the desktop head unit) and fixed a few issues. I'll try testing it in my car in the coming days.

[haggaie]

Sorry for the delay. I have gone through the changes. Looking great. I just updated the PR with some kotlin conversions for new files and fixed own comments.
I have not been able to test the PR though. Feel free to update my changes if found any issues in testing.

No problem, thanks for the review. I've tested it locally (with the desktop head unit) and fixed a few issues. I'll try testing it in my car in the coming days.

Hi @snaik20, I've tested the PR again both with the desktop head unit and in my car, and it worked fine.

[fleapower]

I've been using it in my car, as well, with no problems.

[Banner-Keith]

I've experienced a few bugs, but I think they are the same behavior as the current public release, and not directly related to the Android Auto integration. Nothing new. Works great.

[snaik20]

Thanks for the PR and testing. Approved the changes.

[fleapower]

Had a couple of errors over the last several days, but I can't repeat the errors. First one was an error on starting it with Android Auto, the second was when swiping to the suggested list. Haven't happened since. Everything is working great.

[AudricV]

Here is a summary of the current bugs I found:

Coming from #9592 (comment):

• Items (playlists and streams) are not updated in Android Auto when they are in the app.
• When closing the player from its mini view, playback continues. It should be stopped instead.
• When starting playback on Android Auto then opening the app, the mini player doesn't appear and isn't synchronized with the current state when "opened" (opening the details page of a content with autoplay disabled, pressing the play button plays the content on the details page). I am not sure if you should solve this, as it seems to be an issue not related to this PR and the feature it adds

Other issues:

• Remote playlists are limited to their first page. Like I said in Add support for Android Auto #9592 (comment), this is something related to pagination and should be handled in a separate PR.
• I also had issues once when starting the app then connecting to an Desktop Head Unit. The DHU was blocked on the loading step happening before showing playback controls normally, the app was playing the audio in the meantime then the app crashed. I don't have a stacktrace unfortunately, but it is either a foreground service start time limit reached or the YouTube extractor channels serialization failure currently present which I introduced with 0.27.2.
• Non square thumbnails seems to forced to the square format in the miniplayer commands if you enable the setting of the Desktop Head Unit. I don't know if that's a DHU issue or not.
  

Everything else I can test seem to work properly. I can't test search, since it seems to require Google Assistant and voice to do so, and I don't want to use it. If you have an other way to do a search with the DHU, please let me know.

For improvements, here is what should be done in others PRs:

• Show progress bars for played items and for the one currently playing, if applicable (see https://developer.android.com/training/cars/media?hl=en#browse-progress-bar for the latter).
• Add more content to Android Auto.

[AudricV]

Thanks a lot for maintaining this PR for so much time!

I have some comments on the current code changes, a part of them concern @snaik20's changes in this PR.

It would be really great if you can update your PR description to mention what is supported, what is not supported, what are the limitations and the known bugs, so we can open issues to track and fix/implement them in an easier way.

Maybe important code changes should be added too to the description.

The changes look good overall for me.

[AudricV]

Is there a reason why a simple Error is thrown in this case?

[haggaie]

I suppose NullPointerException would be more appropriate? Or returning null?

[AudricV]

Why is a content not supported error raised when a search error occurs?

[haggaie]

I wasn't sure how to translate the possible internal errors, so I just picked one. I suppose we could generate ERROR_CODE_ACTION_ABORTED or another value as a general error for search errors, or alternatively try to translate the individual potential errors. Do you know if the extractor has a list of potential exceptions?

[haggaie]

Here is a summary of the current bugs I found:

Thanks @AudricV, I'll update the PR text with a list of open issues.

I also had issues once when starting the app then connecting to an Desktop Head Unit. The DHU was blocked on the loading step happening before showing playback controls normally, the app was playing the audio in the meantime then the app crashed. I don't have a stacktrace unfortunately, but it is either a foreground service start time limit reached or the YouTube extractor channels serialization failure currently present which I introduced with 0.27.2.

I'm afraid I haven't seen this. When I connect the DHU the playback simply stops, but playback controls are immediately shown. Perhaps the error depends on some specific state of the app or the media session.

Non square thumbnails seems to forced to the square format in the miniplayer commands if you enable the setting of the Desktop Head Unit. I don't know if that's a DHU issue or not.

Could you explain the expected behavior, and where this square format setting is configured in the DHU?

Everything else I can test seem to work properly. I can't test search, since it seems to require Google Assistant and voice to do so, and I don't want to use it. If you have an other way to do a search with the DHU, please let me know.

The search I added to the PR doesn't work with Google Assistant, as Google Assistant seems to only work with Google Play Store's apps (I'll add that to the PR description). I added a search that appears from inside the app's browser view within Android Auto. Basically, you click the app icon and then the magnifying glass at the top.

I haven't tried it recently, but I believe you could test the same interfaces used by Android Auto and Google Assistant also with the Media Controller Test app (https://developer.android.com/media/optimize/mct), but note that it should have separate features prepare/playFromSearch for the Google Assistant interface and "test results" that I hope cover the android auto search feature.

[sonarcloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud