Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[extractor/twitch] Skip errors in GQL response #6318

Merged
merged 5 commits into from
Mar 4, 2023

Conversation

elyse0
Copy link
Contributor

@elyse0 elyse0 commented Feb 21, 2023

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

This PR fixes the problem where Twitch's GQL API sometimes returns an error before the actual data, breaking the extractor that assumed the video data would be at index 0.

Full json response: https://gist.github.com/elyse0/3b1679b6a0b8d7aa8112f75c1eb52403

Trimmed json

[
  {
    "errors": [
      {
        "message": "PersistedQueryNotFound"
      }
    ],
    "extensions": {
      "durationMilliseconds": 3,
      "operationName": "VideoMetadata",
      "requestID": "01GSTXCZDDA9GN7ZHBRSB9MP4Y"
    }
  },
  {
    "data": {
      "video": {
        "id": "1697007802",
        "moments": {
          ...
          "__typename": "VideoMomentConnection"
        },
        "__typename": "Video"
      }
    },
    "extensions": {
      "durationMilliseconds": 61,
      "operationName": "VideoPlayer_ChapterSelectButtonVideo",
      "requestID": "01GSTXCZDDA9GN7ZHBRSB9MP4Y"
    }
  },
  {
    "data": {
      "video": {
        "id": "1697007802",
        "seekPreviewsURL": "https://d1m7jfoe9zdc1j.cloudfront.net/66e41701aa009d198649_vickypalami_40717337225_1672796209/storyboards/1697007802-info.json",
        "__typename": "Video"
      }
    },
    "extensions": {
      "durationMilliseconds": 25,
      "operationName": "VideoPlayer_VODSeekbarPreviewVideo",
      "requestID": "01GSTXCZDDA9GN7ZHBRSB9MP4Y"
    }
  }
]

Fixes #6308

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
@garoto
Copy link

garoto commented Feb 24, 2023

With this patch applied, stream titles will randomly fail to be fetched. Which VOD fails to title fetch varies on every run of the for loop below:

> for /f %a in (vid) do @yt-dlp --get-title %a
Untitled Broadcast
The game in the year of our lord 2023 without FOV options 🤣🤣🤣
BIRTHDAY SPEEDRUNS
Returnal
Untitled Broadcast
SPOOKY SUNDAY - The Karaoke
Untitled Broadcast
Any Goblin Caves Enjoyers?

[ R:\ ]
> for /f %a in (vid2) do @yt-dlp --get-title %a
Untitled Broadcast
Untitled Broadcast
Untitled Broadcast
Elden Ring Any% Glitchless WR Attempts | Atomic Hearts/Rivals Later w/ !team
Untitled Broadcast
Untitled Broadcast
PURE DARKNESS Elden Ring in FIRST PERSON | Horror Mode | !Vods (I won the !KemonoQuest)
Beating Elden Ring in FIRST PERSON | All Remembrances | !Vods (I won the !KemonoQuest)

vid contents: https://0x0.st/Hs3g.txt
vid2 contents: https://0x0.st/Hs3E.txt

My yt-dlp environment:

[debug] Command-line config: ['-v']
[debug] User config "C:\Users\garoto\yt-dlp.conf": ['--extractor-args', 'youtube:skip=translated_subs', '--external-downloader-args', '-hide_banner -loglevel warning -stats', '--no-sponsorblock', '--no-mtime', '--no-warnings', '--ignore-errors', '--force-ipv4', '--restrict-filenames', '--merge-output-format', 'mkv', '--sub-lang', 'en.*,pt,pt-BR', '--write-sub', '--embed-subs', '--convert-subs', 'srt', '--format', '(bestvideo[width<=1920][fps<=60][vcodec^=vp9]+bestaudio[acodec=opus])/299+140/137+251/best/bestvideo+bestaudio', '--output', '%(title).70s-%(upload_date)s-%(uploader)s-%(extractor_key)s_%(id).19s.%(ext)s']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.8.2 (CPython AMD64 64bit) - Windows-8.1-6.3.9600-SP0 (OpenSSL 1.1.1d  10 Sep 2019)
[debug] exe versions: ffmpeg 4.0.2, ffprobe 4.0.2
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors

@elyse0
Copy link
Contributor Author

elyse0 commented Feb 24, 2023

Looking at the extractor and error more closely, the problem was that some _OPERATION_HASHES were not up to date

I've updated these hashes and updated some tests

This shoud fix the titles @garoto

@garoto
Copy link

garoto commented Feb 25, 2023

This shoud fix the titles @garoto

Tested a bunch of VODs and it seems to be fixed indeed. Thank you!

@pukkandan pukkandan merged commit 4a6272c into yt-dlp:master Mar 4, 2023
aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Occasional fatal error when downloading videos from Twitch
3 participants