Enable AV1, HEVC via RTMP to YouTube

[nquah]

Implementation credited to yuriy-chumak

Description

• Adds support for HEVC and AV1 codecs to RTMP, according to the Enhanced RTMP spec

• Adds flag-guarded support for HDR metadata frame to RTMP, according to the Enhanced RTMP spec

Motivation and Context

Veovera Software Organization (aka VSO), a not-for-profit organization, was formed to help enhance software for the public good. VSO members currently include Adobe, Google and Veriskope. VSO published an Enhanced RTMP spec, which adds long-awaited support for HEVC and AV1 to the FLV container, enabling their use in RTMP.

Furthermore, with the advent of GPUs that support real-time AV1 encoding, we can now enable AV1 streaming.

This change updates the RTMP implementation in OBS to support HEVC and AV1. It additionally enables these codecs for RTMP ingestion to YouTube. This YouTube feature is currently in beta.

Note: YouTube transcodes live streams, and does not currently support AV1 for live delivery. Creators can still benefit from maintaining higher ingestion quality, with fewer compression artifacts, even at lower bitrates.

How Has This Been Tested?

• Tested using FFmpeg as the RTMP server, and FFplay as the client.

• Tested against YouTube’s RTMP server, up to 4K 60fps.

• Tested AV1 live encoding using NVENC (NVIDIA), QSV (Intel) and AMF (AMD) encoders.

Types of changes

New feature (non-breaking change which adds functionality)

Known Issues

• This feature is in beta at YouTube. Please report feedback.
• Not all encoder settings may yield equal results. Check existing YouTube recommendations for H.264/RTMP here for comparison, and please report feedback.
• Playback of rich content at high resolution may struggle for reasons unrelated to encoding performance, e.g. network connectivity or device performance of the client playing the stream.
• HDR via RTMP is flag-guarded and not yet supported by YouTube.
• 10-bit AV1 streaming to YouTube may not work properly yet.

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[RytoEX]

Some quick feedback.

Commit messages must follow the 50/72 rule, outlined in our Commit Guidelines.

Commit messages should have a prefix for the commit subject. In this case, as there are multiple components being modified, obs-outputs is probably most appropriate as the prefix. Then the commit message body should note that "this commit also modifies libobs, rtmp-services, and UI components". See other commits for examples.

Files must be formatted with clang-format 13.0.1. Currently, the file that is failing this check is plugins/obs-outputs/rtmp-av1.h.

I will leave code review to others.

[derrod]

I pushed my changes now. These mainly split up the PR into different commits to make things a little easier to review. It also splits out the change blocking HDR into a separate commit which can more easily be reverted later if this limitation no longer exists.

Additionally, there were a number of cleanup changes I made such as removing (now) unused code as well as removing the reliance on FFmpeg (which obs-outputs does not link to) for compilation.

[derrod]

Whoops somebody forgot to run this through Linux CI so a breaking change was undetected because MSVC does things differently, yay. Hopefully should be fixed now.

[tytan652]

Unicode character issues.

[flying1314]

could you provide the modified ffmpeg for us to test it

[Fenrirthviti]

could you provide the modified ffmpeg for us to test it

We are not using a modified ffmpeg. The changes for our support are to librtmp.

[winlinvip]

SRS media server support this extended RTMP specification, so you can push HEVC via RTMP to SRS very easy:

git clone https://github.com/ossrs/srs.git
cd srs/trunk && ./configure --h265=on && make
./objs/srs -c conf/http.ts.live.conf

Then, you can use OBS 29.1+ to push HEVC via RTMP.
Start OBS with the following settings in the Settings > Stream tab:

• Server: rtmp://localhost/live
• Stream Key: livestream
• Encoder: Please select the HEVC hardware encoder.

Finally, open the player http://localhost:8080/players/srs_player.html?stream=livestream.ts

Or use VLS or ffplay to play http://localhost:8080/live/livestream.ts