Broken segments

[Molkree]

While working on Orcasound data I've encountered a few issues, all examples are taken from the rpi_bush_point/hls/1624818619:

1. some segments have silence in them
   live1049.ts

2. some segments are super short, less than one-tenth of a second
   The first one is live221.ts, 0.085333s long but there are more later

3. some segments seem to have broken headers?
   live808.ts seems to be broken, can't play it at all.
   And here's ffmpeg error I'm getting on it when trying to convert to .wav

ERROR:root:
ERROR:root:ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[mpeg @ 0x555c91ce6000] Format mpeg detected only with low score of 25, misdetection possible!
[mp2 @ 0x555c91d0c500] Header missing
    Last message repeated 1 times
[mpeg @ 0x555c91ce6000] decoding for stream 0 failed
[mpeg @ 0x555c91ce6000] Could not find codec parameters for stream 0 (Audio: mp2, 0 channels, s16p): unspecified frame size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, mpeg, from 'bush_point/1624818619/live808.ts':
  Duration: 00:00:03.57, start: 8081.641067, bitrate: 3 kb/s
    Stream #0:0[0x1c0]: Audio: mp2, 0 channels, s16p
Stream mapping:
  Stream #0:0 -> #0:0 (mp2 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[mp2 @ 0x555c91d0ca00] Header missing
Error while decoding stream #0:0: Invalid data found when processing input
[mp2 @ 0x555c91d0ca00] Header missing
Error while decoding stream #0:0: Invalid data found when processing input
Finishing stream 0:0 without any data written to it.
[abuffer @ 0x555c91cc4280] Value inf for parameter 'time_base' out of range [0 - 2.14748e+09]
    Last message repeated 3 times
[abuffer @ 0x555c91cc4280] Error setting option time_base to value 1/0.
[graph_0_in_0_0 @ 0x555c91cdeb40] Error applying options to the filter.
Error configuring filter graph
Conversion failed!

[scottveirs]

Last night it appears Sunset Bay restarted successfully, but streamed .ts segments with only zero-amplitude audio data. It feels like this happens once in a blue moon, but in this case I logged in with Dataplicity and confirmed with htop that both ffmpeg and upload_s3.py were running. Also, arecord -l suggested that the audio hardware device was pisound as expected.

From the S3 and player perspectives, the .m3u8 manifest and .ts segments seemed to be in place, but the silent .ts segments in the streaming timestamped bucket were anomalously small -- ~22 kB instead of the typical 107 kB file size for that location, e.g. in the next bucket timestamp that seemed to resolve the issue (via a manual stop/start of the container).

Here is an example clip that a listener tagged as silent -- https://live.orcasound.net/reports/cand_02v0gyoe1eO4nw9uOw6Sjt

Here is screenshot to the player before/after the container restart:

And here is a view of the zero-amplitude data in Audacity (after download of .ts segments and transcoding to mp3 via mp4...):

[scottveirs]

Possibly related issue -- https://www.reddit.com/r/ffmpeg/comments/manakd/no_audio_on_hls_stream/

[scottveirs]

A first this morning reviewing in Audacity the archived lossy .ts segments from this morning (6:30am local) concatenated into a continuous .mp3 file by my ts2mp3.sh script: I discerned a data gap 10-seconds into the recording. I noticed this because the first 10 seconds happened to have low ambient noise; then there is a jump to boat noise that clearly indicates a gap of many seconds or minutes.

This suggests a pretty major bug in the streaming code or upload performance from the Orcasound Lab hydrophone. Is this sort of gap (part of) the cause of the ~10-minutes of missing data per ~6-hour data streaming container run?

Logging in remotely with Dataplicity, I see these hints at a (pretty old) version of the orcanode code:

[scottveirs]

A final note: in further reviewing the last ~18 hours of streamed data from Orcasound Lab to understand when J pod first reached Haro Strait from points north, I discovered a full 6-hour data segment missing from the S3 bucket (5/4/24 18:30 - 5/5/24 00:30) --

-- likely related to (and now noted within) issue #18