Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port from librtlsdr to SoapySDR API (rtl_fm, rtl_sdr, and rtl_power) #1

Merged
merged 38 commits into from
Jul 17, 2016

Conversation

@rxseger
Copy link
Owner Author

rxseger commented Jul 14, 2016

Current status: able to list devices, open, set some settings, but cannot yet receive samples. librtlsdr has an rtlsdr_read_async API with a callback, data as pairs of unsigned 8-bit values, SoapySDR has streams, and can read in a variety of formats (SOAPY_SDR_U8 for compatibility), not yet working, some impedance mismatches, but not by much. Crashes in buffer handling:

rtl_tools $ lldb ./rtl_fm -- -M wbfm -f 107.1M 
(lldb) target create "./rtl_fm"
Current executable set to './rtl_fm' (x86_64).
(lldb) settings set -- target.run-args  "-M" "wbfm" "-f" "107.1M"
(lldb) r
Process 13136 launched: './rtl_fm' (x86_64)
Mac OS; Clang version 7.3.0 (clang-703.0.31); Boost_106000; UHD_003.009.004-0-unknown

Found Rafael Micro R820T tuner
Found 1 device(s):
  0: available=Yes, driver=rtlsdr, label=Generic RTL2832U OEM :: 3, manufacturer=Realtek, product=RTL2838UHIDIR, rtl=0, serial=3, tuner=Rafael Micro R820T, 

Found Rafael Micro R820T tuner
Bandwidth set to automatic resulted in 0 Hz.
Tuned to 107371000 Hz.
Oversampling input by: 6x.
Oversampling output by: 1x.
Buffer size: 8.03ms
Exact sample rate is: 1020000.026345 Hz
setupStream fail: error
Process 13136 stopped
* thread #2: tid = 0x4740c, 0x0000000101a707d8 librtlsdrSupport.so`std::__1::vector<signed char, std::__1::allocator<signed char> >::resize(unsigned long) + 4, stop reason = EXC_BAD_ACCESS (code=1, address=0x264c00788820)
    frame #0: 0x0000000101a707d8 librtlsdrSupport.so`std::__1::vector<signed char, std::__1::allocator<signed char> >::resize(unsigned long) + 4
librtlsdrSupport.so`std::__1::vector<signed char, std::__1::allocator<signed char> >::resize:
->  0x101a707d8 <+4>:  movq   (%rdi), %rax
    0x101a707db <+7>:  movq   0x8(%rdi), %rcx
    0x101a707df <+11>: movq   %rcx, %rdx
    0x101a707e2 <+14>: subq   %rax, %rdx
(lldb) bt
* thread #2: tid = 0x4740c, 0x0000000101a707d8 librtlsdrSupport.so`std::__1::vector<signed char, std::__1::allocator<signed char> >::resize(unsigned long) + 4, stop reason = EXC_BAD_ACCESS (code=1, address=0x264c00788820)
  * frame #0: 0x0000000101a707d8 librtlsdrSupport.so`std::__1::vector<signed char, std::__1::allocator<signed char> >::resize(unsigned long) + 4
    frame #1: 0x0000000101a6f9bc librtlsdrSupport.so`SoapyRTLSDR::rx_callback(unsigned char*, unsigned int) + 96
    frame #2: 0x0000000101a82c72 librtlsdr.0.dylib`_libusb_callback + 66
    frame #3: 0x000000010192eba2 libusb-1.0.0.dylib`usbi_handle_transfer_completion + 285
    frame #4: 0x000000010192f6ed libusb-1.0.0.dylib`handle_events + 879
    frame #5: 0x000000010192f148 libusb-1.0.0.dylib`libusb_handle_events_timeout_completed + 258
    frame #6: 0x0000000101a82ab6 librtlsdr.0.dylib`rtlsdr_read_async + 608
    frame #7: 0x0000000101a70f78 librtlsdrSupport.so`void* std::__1::__thread_proxy<std::__1::tuple<void (SoapyRTLSDR::*)(), SoapyRTLSDR*> >(void*) + 78
    frame #8: 0x00007fff91a7399d libsystem_pthread.dylib`_pthread_body + 131
    frame #9: 0x00007fff91a7391a libsystem_pthread.dylib`_pthread_start + 168
    frame #10: 0x00007fff91a71351 libsystem_pthread.dylib`thread_start + 13
(lldb) 

@rxseger
Copy link
Owner Author

rxseger commented Jul 15, 2016

Some progress, there is now stream output, but when played through play it sounds like only a regularly repeating buzz, off and on. The demodulator output is irregular, as if it is being starved. Emits at about every 61-63 readStream calls.

When piped through to sox to convert to wav:

 ./rtl_fm -M wbfm -f 107.1M | sox -r 32k -t raw -e s -b 16 -c 1 -V1 - -t wav /tmp/a.wav

it is a solid buzzing sound: https://soundcloud.com/user-223812058/rtl-fm-fail-1

Audacity plot spectrum shows:

screen shot 2016-07-14 at 9 04 46 pm

@rxseger
Copy link
Owner Author

rxseger commented Jul 15, 2016

Slight problem: SoapyRTLSDR reports native format as SOAPY_SDR_CS8 (complex signed 8-bit pairs), but is actually CU8 (unsigned), the driver actually subtracts 127 https://github.com/pothosware/SoapyRTLSDR/blob/a44abc3f1ac2c62d707902551f1c2c1e70551c84/Streaming.cpp#L372-L391 - although the direct stream access API uses CU8, see pothosware/SoapyRTLSDR#15.

But even then, something seems wrong with the samples:

Found Rafael Micro R820T tuner
0000ff40  66 00 fc ff cb 00 27 00  76 00 2f 00 5b 00 a4 00  |f.....'.v./.[...|
0000ff50  b5 00 d5 ff 6d 00 89 00  33 00 77 ff f0 ff ab ff  |....m...3.w.....|
0000ff60  50 00 c9 ff 5a 00 a4 00  b5 00 d5 ff 6d 00 54 00  |P...Z.......m.T.|
0000ff70  db ff a0 ff 18 00 c7 ff  68 00 bf ff 57 00 e3 ff  |........h...W...|
0000ff80  44 00 82 ff 4c 00 2d 00  3a 00 7f ff 02 00 c6 ff  |D...L.-.:.......|
0000ff90  4e 00 b9 ff 4e 00 b6 00  9f 00 f5 ff 2d 00 db ff  |N...N.......-...|
0000ffa0  67 00 11 00 9f 00 2f 00  9f 00 e3 ff 6a 00 e6 ff  |g...../.....j...|
0000ffb0  63 00 c4 ff df ff 68 00  50 00 6a ff 02 00 91 ff  |c.....h.P.j.....|
0000ffc0  bc ff cc ff 5d 00 5b 00  f7 00 00 00 f9 ff 5a 00  |....].[.......Z.|
0000ffd0  7d 00 71 ff f0 ff 4c 00  f9 ff e0 ff 32 00 ca ff  |}.q...L.....2...|
0000ffe0  2b 00 da ff d7 ff 43 00  6b 00 68 ff ea ff 95 ff  |+.....C.k.h.....|
0000fff0  62 00 f8 ff e6 ff d3 00  c3 00 c7 ff 3c 00 1c 00  |b...........<...|
00010000  00 00 4b 00 95 00 09 00  ba 00 24 00 04 00 d5 00  |..K.......$.....|
00010010  e6 00 d0 ff 35 00 bb 00  19 00 fb ff 89 ff a6 ff  |....5...........|
00010020  1b 00 f3 ff 06 00 b0 00  f5 00 d9 ff 3d 00 64 00  |............=.d.|
00010030  d3 ff 7c 00 08 00 6b 00  ab 00 65 00 d3 ff 7c 00  |..|...k...e...|.|
00010040  08 00 3d 00 44 00 1e 00  8c 00 10 00 53 ff bd ff  |..=.D.......S...|

actually not necessarily, rtlsdr_callback first thing does:

        /* 1st: convert to 16 bit - to allow easier calculation of DC */
        for (i=0; i<(int)len; i++) {
                s->buf16[i] = ( (int16_t)buf[i] - 127 );
        }

unnecessary conversions (-127 in SoapyRTLSDR, +127 in rtl_fm dongle_thread_fn, then -127 in rtl_fm rtlsdr_callback!) but now fixing the buffer types, get choppy static

@rxseger rxseger changed the title [Incomplete] rtl_fm: port from librtlsdr to SoapySDR API [Incomplete] rtl_fm, rtl_sdr: port from librtlsdr to SoapySDR API Jul 15, 2016
@rxseger
Copy link
Owner Author

rxseger commented Jul 15, 2016

It may be more straightforward to port rtl_sdr first, since it outputs raw I/Q samples, without the complexity of FM demodulation to debug. Pushed what I have so far, but quite broken, curiously readStream returns 8192 bytes but all bytes are 00 or 01(?), not sure what is happening here.

@rxseger
Copy link
Owner Author

rxseger commented Jul 16, 2016

Turning up the gain (-g 40 with rtl_sdr) gives more interesting results:

rtl_tools $ ./rtl_sdr -g 40 -f 107.7M -|hexdump -C|head -20

00000070  00 05 04 02 03 00 00 fb  fa ff fe 03 02 00 00 ff  |................|
00000080  ff fb fa fe fd 04 04 02  02 fe 00 fb fa ff ff 04  |................|
00000090  04 00 02 fe 01 fd fa fd  fc 05 04 02 01 fd ff fd  |................|
000000a0  fb ff fb 04 04 03 01 fc  00 fe fb 00 fd 05 03 04  |................|
…

at a lower gain, 00 and 01 similar to 7f and 80 (signed/unsigned offset) with librtlsdr rtl_sdr. However, with no gain specified, librtlsdr rtl_sdr works as expected, but this branch rtl_sdr only shows 00/01's.

Auto-gain is not set, in src/convenience/convenience.c verbose_auto_gain (and verbose_gain_set), couldn't find an equivalent for rtlsdr_set_tuner_gain_mode(dev, 0); (= set auto gain) and rtlsdr_set_tuner_gain_mode(dev, 1); (= set manual gain). Not called in SoapyRTLSDR: https://github.com/pothosware/SoapyRTLSDR/search?utf8=✓&q=rtlsdr_set_tuner_gain_mode - so for now, will need to set gain explicitly (or hardcode a reasonable default?).

update: filed pothosware/SoapyRTLSDR#21 Automatic gain mode setting, rtlsdr_set_tuner_gain_mode(dev, 0) equivalent

@rxseger
Copy link
Owner Author

rxseger commented Jul 16, 2016

With gain, something can now be heard, just barely:

./rtl_fm  -M wbfm -f 107.7M | play -r 32k -t raw -e s -b 16 -c 1 -V1 -
rtl_fm  -M wbfm -f 107.7M | play -r 32k -t raw -e s -b 16 -c 1 -V1 -

but it is still choppy (the rtl_fm stream starts and stops). Recording to a .wav file on disk with sox (therefore removing the gaps):

https://soundcloud.com/user-223812058/rtl-fm-progress-1

it sounds noisy but also as if it was sped up. Halving the sample rate (./rtl_fm -M wbfm -f 107.7M | play -r 16k -t raw -e s -b 16 -c 1 -V1 -) it can play smoothly, but still very noisily. Maybe a bug in setting the sampling rate?


Another potential problem: setting the bandwidth?

"Bandwidth set to automatic resulted in 0 Hz." vs expected "Bandwidth set to automatic resulted in 350000 Hz.". verbose_set_bandwidth uses rtlsdr_set_and_get_tuner_bandwidth, to both set and get the bandwidth. 0 indicates "automatic", meaning equal to the sampling rate (bw > 0 ? bw : dev->rate in rtlsdr_set_and_get_tuner_bandwidth). But SoapyRTLSDR https://github.com/pothosware/SoapyRTLSDR/blob/master/Settings.cpp#L448 just calls out to SoapySDR::Device::setBandwidth, no calls to rtlsdr_set_and_get_tuner_bandwidth?!. https://github.com/pothosware/SoapySDR/blob/master/lib/Device.cpp#L486 setBandwidth() is a no-op and getBandwidth() returns 0.0, I guess it is not implemented.


Setting bandwidth is relatively "new", rtlsdr_set_tuner_bandwidth was added in librtlsdr/librtlsdr@92df068 Mar 17, 2015 see http://lists.osmocom.org/pipermail/osmocom-sdr/2015-June/000077.html. Other projects have ran into problems relying on this function - jpoirier/gortlsdr#14 - since it "is not present in Linux distro releases of rtl-sdr package" (Feb 28). There was a post to osmocom-sdr requesting a new official librtlsdr release: http://lists.osmocom.org/pipermail/osmocom-sdr/2016-February/001373.html but meanwhile "A project I've been using this package in simply requires users to build rtlsdr from source" hmm.. may be premature to update SoapyRTLSDR setBandwidth to call rtlsdr_set_tuner_bandwidth(). Either way, not likely the problem here.

update: filed pothosware/SoapyRTLSDR#20 setBandwidth/getBandwidth no-ops, should call rtlsdr_set_and_get_tuner_bandwidth?


Possibly has to do with rtlsdr_read_async: - rtl_sdr has a sync mode, but rtl_fm is only async, calls the callback, when porting I replaced with SoapySDRDevice_readStream, sync. There is significant logic in librtlsdr to buffer async, and then call the callback when appropriate: https://github.com/librtlsdr/librtlsdr/blob/development/src/librtlsdr.c#L1840 - although SoapyRTLSDR https://github.com/pothosware/SoapyRTLSDR/blob/master/Streaming.cpp does call rtlsdr_read_async under the hood. What parameters are there to adjust how fast I can read from the SDR, via SoapySDR? ...

There is a "buflen" kwarg argument for setupStream in SoapyRTLSDR, but it defaults to 16384, same as DEFAULT_BUF_LENGTH in rtl_fm (but rtl_sdr has 16 * 16384), changing no effect, still reads 8192.

The original rtl_fm's rtlsdr_callback gets called consistently with len=262144 (= 16384 * 16).

@rxseger
Copy link
Owner Author

rxseger commented Jul 16, 2016

Turned out to be a simple problem, readStream returns the number of elements read, aka the number of samples, which consist of both I and Q values, each 8-bit, forming a complex number. s->buf_len is expected to be the length of the buffer in bytes, so it is double the return value of readStream(). After fixing this, can play and demodulate FM broadcast audio without choppiness.

Instead of having rtl_fm and rtl_sdr convert CS8 to CU8, add the
read_samples_cu8() convenience function. This reads from SoapySDR using
CS8 format, and adds 127 to convert to CU8, the actual native format - a
redundant but necessary conversion, could be improved after
pothosware/SoapyRTLSDR#15 if SoapyRTLSDR adds
native CU8 support (remove the loop in read_samples_cu8()).
@rxseger rxseger changed the title [Incomplete] rtl_fm, rtl_sdr: port from librtlsdr to SoapySDR API [Incomplete] rtl_fm, rtl_sdr, rtl_power: port from librtlsdr to SoapySDR API Jul 16, 2016
@rxseger rxseger changed the title [Incomplete] rtl_fm, rtl_sdr, rtl_power: port from librtlsdr to SoapySDR API rtl_fm, rtl_sdr, rtl_power: port from librtlsdr to SoapySDR API Jul 16, 2016
@rxseger
Copy link
Owner Author

rxseger commented Jul 17, 2016

rtl_fm and rtl_sdr basically work now (rtl_power potentially as well but needs more longer-term testing), with an RTL-SDR. Not compatible with a bladeRF yet due to higher bit-rate #8. With a HackRF, rtl_sdr emits output, but only silence (0x7f - 0x83), need to adjust the multiple gains?


SoapyHackRF::setGain https://github.com/pothosware/SoapyHackRF/blob/master/HackRF_Settings.cpp#L358 has some.. interesting logic, to set a AMP, VGA, and LNA (comments mine):

        if ( gain <= 0 )
        {
            _rx_stream->lna_gain    = 0;
            _rx_stream->vga_gain    = 0;
            _current_amp        = 0;
        }else if ( gain <= /*51*/ (HACKRF_RX_LNA_MAX_DB / 2) + (HACKRF_RX_VGA_MAX_DB / 2) )
        {
            _rx_stream->vga_gain    = (gain / 3) & ~0x1;
            _rx_stream->lna_gain    = gain - _rx_stream->vga_gain;
            _current_amp        = 0;
        }else if ( gain <= /*65*/ ( (HACKRF_RX_LNA_MAX_DB / 2) + (HACKRF_RX_VGA_MAX_DB / 2) + HACKRF_AMP_MAX_DB) )
        {
            _current_amp        = HACKRF_AMP_MAX_DB;
            _rx_stream->vga_gain    = ( (gain - _current_amp) / 3) & ~0x1;
            _rx_stream->lna_gain    = gain -_current_amp - _rx_stream->vga_gain;
        }else if ( gain <= /*116*/ HACKRF_RX_LNA_MAX_DB + HACKRF_RX_VGA_MAX_DB + HACKRF_AMP_MAX_DB )
        {
            _current_amp        = HACKRF_AMP_MAX_DB;
            _rx_stream->vga_gain    = (gain - _current_amp) * double(HACKRF_RX_LNA_MAX_DB) / double(HACKRF_RX_VGA_MAX_DB);
            _rx_stream->lna_gain    = gain - _current_amp - _rx_stream->vga_gain;
        }

        _rx_stream->amp_gain=_current_amp;

        ret = hackrf_set_lna_gain( _dev, _rx_stream->lna_gain );
        ret |= hackrf_set_vga_gain( _dev, _rx_stream->vga_gain );
        ret |= hackrf_set_amp_enable( _dev, (_current_amp > 0) ? 1 : 0 );

and in the headers:

#define HACKRF_RX_VGA_MAX_DB 62
#define HACKRF_TX_VGA_MAX_DB 47
#define HACKRF_RX_LNA_MAX_DB 40
#define HACKRF_AMP_MAX_DB 14

setGain(40) results in LNA=28, VGA=12, AMP=0 (from adding logging to SoapyHackRF), which I can copy in CubicSDR.

Seeing strange return values from readStream with the HackRF:

readStream ret=-1, flags=0, timeNs=4362240
readStream ret=131072, flags=0, timeNs=4362240
readStream ret=-1, flags=0, timeNs=4362240

then it stops. Why the error? -1 is SOAPY_SDR_TIMEOUT, fixed by increasing timeout, but only hear static with the HackRF. readStream ret=131072 (=16384 * 8)

Setting bandwidth and sampling rate 20e6, no difference.

SoapySDRUtil --probe shows:

----------------------------------------------------
-- RX Channel 0
----------------------------------------------------
  Full-duplex: NO
  Supports AGC: NO
  Stream formats: CS8, CS16, CF32, CF64
  Native format: CS8 [full-scale=128]
  Stream args:
     * Buffer Count - Number of buffers per read.
       [key=buffers, units=buffers, default=15, type=int]
  Antennas: TX/RX
  Full gain range: [0, 116] dB
    LNA gain range: [0, 40] dB
    VGA gain range: [0, 62] dB
    AMP gain range: [0, 14] dB
  Full freq range: [0, 7250] MHz
    RF freq range: [0, 7250] MHz
  Sample rates: [1, 20] MHz
  Filter bandwidths: [1.75, 28] MHz

with -g 116, I can sometimes sort of hear something above the static with the HackRF, even though it logs invalid parameter:

HackRF device setting gains: LNA=36, VGA=63, AMP=14
[ERROR] setGain(113.000000) returned HACKRF_ERROR_INVALID_PARAM

@rxseger rxseger changed the title rtl_fm, rtl_sdr, rtl_power: port from librtlsdr to SoapySDR API Port from librtlsdr to SoapySDR API (rtl_fm, rtl_sdr, and rtl_power) Jul 17, 2016
@rxseger rxseger merged commit 4109f2c into master Jul 17, 2016
@rxseger rxseger deleted the soapy branch July 17, 2016 02:07
@rxseger rxseger mentioned this pull request Jul 17, 2016
@rxseger
Copy link
Owner Author

rxseger commented Jul 17, 2016

Merged what I have so far since it works well with rtlsdr driver, new issue for HackRF: #9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant