Achieving stable latency between calls of "playrec"

[davircarvalho]

Hello,

I'm just looking for advice on the best way to implement this.

I have a system I want to do calls of simutaneous playback and recording. When I do multiple (non-concurrent) calls of sd.playrec() each one ends up with different latency.

I believe this happens because the stream within sd.playrec() is closed after each playrec call, then after subsequent call portaudio will deliver a slight different latency.

If I want to achieve constant latency which is sample accurate across all calls of a "playrec" function, I suppose I need to start one stream and use it for all the "playrec" calls? Is such goal even realistic?

Any help is appreciated!

[mgeier]

I think this depends on your hardware and maybe also on the host API (none of which you disclosed).

Some devices reliably (and sample-accurately) provide the same latency (given the same blocksize parameter) each time a stream is created, others might not.

But yes, either way, during the runtime of a single stream, the latency should stay mostly constant, except maybe for some clock drift.

Some suggestions:

• try a different host API, maybe the one you are using sucks
• buy a more expensive sound card, maybe the one you are using sucks
• write your own multi-playrec-thing using a single stream and hope for the best

[HaHeho]

Some suggestions:

* try a different host API, maybe the one you are using sucks

* buy a more expensive sound card, maybe the one you are using sucks

* write your own multi-playrec-thing using a single stream and hope for the best

@davircarvalho alternatively you could try to compensate the varying delays after capture. However, it depends on your application how this could be done, or if at all.

[davircarvalho]

Hi sorry for the late reply,

I've been trying with MME and WASAPI, but behaviour is similar. I'm testing with virtual audio cable and with an rme madiface pro interface.

The actual reported latency is always the same. I'm doing a loopback measurement, where I call play rec multiple times and have a sleep time in between the call. When I overlay the outputs however they don't align perfectly. Am I just being naive to expect MME / WASAPI to deliver sample accurate stability, or could there be something else I'm not really understanding?

This is my current test script:

import numpy as np
import sounddevice as sd


def custom_playrec(playback_data,
                        fs,
                        N_in_channels,
                        N_out_channels,
                        out_device_ID,
                        in_device_ID,
                        bufferSize=1024,
                        stream=None):

    data = playback_data[:]

    # Open Streams
    if stream is None:
        stream = sd.Stream(
                samplerate=fs,
                blocksize=bufferSize,
                device=[in_device_ID, out_device_ID],
                channels=[N_in_channels, N_out_channels])
    stream.start()


    # Initialize stream variables
    frame_start = 0
    frame_end = frame_start + bufferSize


    # Parse playback data  ---------------------------------------------------------------------
    # Ensure data has the  same data format as the stream
    if np.ndim(data) == 1:
        data = np.expand_dims(data, axis=-1)

    totalLatency = int(np.sum(np.array(stream.latency)) * fs)
    padding = np.zeros((totalLatency+bufferSize, N_in_channels), dtype=np.float32)
    data = np.append(data, padding, axis=0)
    data = data.astype(np.float32)

    # Playback / Record data
    sigLenSamples = len(data)
    buffer_rec = np.zeros((sigLenSamples, N_in_channels), dtype=np.float32)  # data which will be recorded
    buffer_play = np.zeros((bufferSize, N_out_channels))  # data to playback


    while frame_end < max(data.shape):
        # Write data to the output buffer .......................................................
        buffer_play[0:int(frame_end-frame_start), :] = data[frame_start : frame_end, :]

        # output data
        cont_buffer = np.ascontiguousarray(buffer_play, dtype=np.float32)
        stream.write(cont_buffer)

        # Read data from the input buffer .......................................................
        data_in, overflowed = stream.read(bufferSize)

        buffer_rec[frame_start:frame_start+bufferSize, :] = data_in


        # update reading positions  ............................................................
        frame_start = frame_end
        frame_end = frame_start + bufferSize
        if frame_end > len(data):
            frame_end = len(data)

    stream.stop()

    return buffer_rec, stream


if __name__ == "__main__":
    import matplotlib.pyplot as plt

    # create input signal
    duration = 1
    fs = 48000
    t_vec = np.arange(0, duration, 1/fs)
    freq = 4
    gain = 1
    playback_data = gain * np.cos(2*np.pi * freq * t_vec)

    # define interface ID (loopback)
    in_device_ID = 2
    out_device_ID = 9
    bufferSize = 2048
    stream = None

    for k in range(10):  # multiple calls of playrec function
        sd.sleep(1000)  # actual user module requires for user input betweeeb playrec calls
        data_out, stream = custom_playrec(playback_data,
                                            fs,
                                            N_in_channels=1,
                                            N_out_channels=1,
                                            out_device_ID=out_device_ID,
                                            in_device_ID=in_device_ID,
                                            bufferSize=bufferSize,
                                            stream=stream)

        plt.figure(0, figsize=(6, 6))
        plt.plot(data_out, alpha=0.8)
        plt.xlim([1e4, 3.5e4])
        plt.title("Recorded data")


    stream.stop()
    stream.close()

and this is what I'm observing here, all the curves shold be a single line:

Thanks for your help :)

[mgeier]

I've been trying with MME and WASAPI, but behaviour is similar.

Please try all host APIs and report back.

Please also provide your device list. Otherwise the device IDs in your code are meaningless. Also, it would tell me which host APIs are even available.

MME isn't made for low latency, so I wouldn't expect stable latency either.

With WASAPI, I would only expect good behavior in "exclusive" mode, see https://python-sounddevice.readthedocs.io/en/0.4.6/api/platform-specific-settings.html#sounddevice.WasapiSettings

ASIO normally gives the best (i.e. lowest) latency values, maybe also the most stable ones.

I'm testing with virtual audio cable and with an rme madiface pro interface.

I don't understand, both at the same time?

Please elaborate.

[davircarvalho]

MME isn't made for low latency

That's the thing, I don't really mind how large is the latency, as long as it is stable.

I don't understand, both at the same time?

As mentioned in my previous message the script is run in loopback (the output is connected to the input of the same interface), either digital interface in the case of Virtual Audio Cable (VAC), or a hardware interface in the case of the madiface (here a cable is used to connect the output to the input of the same interface). This setup is done for testing latency only not in the real use case.

In the test code I sent above I was using virtual audio cable interface, so output of VAC goes into the input of VAC.

ASIO normally gives the best (i.e. lowest) latency values, maybe also the most stable ones.

I'd love to use ASIO but unfortunately my signal flow wouldn't allow to use ASIO because I'd need to mix APIs (which I don't it's possible, please correct me if I'm wrong)

this is my current device list, but please keep in mind the IDs are different now from the ones I used in that code. If I were to run it with the current config I'd use output ID 11 and input ID 2 in the case of MME

 0 Microsoft Sound Mapper - Input, MME (2 in, 0 out)
>  1 Line 3 (Virtual Audio Cable), MME (2 in, 0 out)
   2 Line 1 (Virtual Audio Cable), MME (2 in, 0 out)
   3 Microphone (Realtek Audio), MME (2 in, 0 out)
   4 Line 2 (Virtual Audio Cable), MME (2 in, 0 out)
   5 Microsoft Sound Mapper - Output, MME (0 in, 2 out)
<  6 Headphones (Galaxy Buds Live (6, MME (0 in, 2 out)
   7 Speakers 3 (Virtual Audio Cable, MME (0 in, 2 out)
   8 Speakers 2 (Virtual Audio Cable, MME (0 in, 2 out)
   9 Speakers / Headphones (Realtek , MME (0 in, 2 out)
  10 Headset (Galaxy Buds Live (6D70, MME (0 in, 2 out)
  11 Speakers (Virtual Audio Cable), MME (0 in, 2 out)
  12 Primary Sound Capture Driver, Windows DirectSound (2 in, 0 out)
  13 Line 3 (Virtual Audio Cable), Windows DirectSound (2 in, 0 out)
  14 Line 1 (Virtual Audio Cable), Windows DirectSound (2 in, 0 out)
  15 Microphone (Realtek Audio), Windows DirectSound (2 in, 0 out)
  16 Line 2 (Virtual Audio Cable), Windows DirectSound (2 in, 0 out)
  17 Primary Sound Driver, Windows DirectSound (0 in, 2 out)
  18 Headphones (Galaxy Buds Live (6D70) Stereo), Windows DirectSound (0 in, 2 out)
  19 Speakers 3 (Virtual Audio Cable), Windows DirectSound (0 in, 2 out)
  20 Speakers 2 (Virtual Audio Cable), Windows DirectSound (0 in, 2 out)
  21 Speakers / Headphones (Realtek Audio), Windows DirectSound (0 in, 2 out)
  22 Headset (Galaxy Buds Live (6D70) Hands-Free AG Audio), Windows DirectSound (0 in, 1 out)
  23 Speakers (Virtual Audio Cable), Windows DirectSound (0 in, 2 out)
  24 ASIO4ALL v2, ASIO (2 in, 2 out)
  25 FL Studio ASIO, ASIO (2 in, 2 out)
  26 FlexASIO, ASIO (0 in, 2 out)
  27 Realtek ASIO, ASIO (2 in, 2 out)
  28 Speakers 3 (Virtual Audio Cable), Windows WASAPI (0 in, 2 out)
  29 Speakers 2 (Virtual Audio Cable), Windows WASAPI (0 in, 2 out)
  30 Headphones (Galaxy Buds Live (6D70) Stereo), Windows WASAPI (0 in, 2 out)
  31 Speakers / Headphones (Realtek Audio), Windows WASAPI (0 in, 2 out)
  32 Headset (Galaxy Buds Live (6D70) Hands-Free AG Audio), Windows WASAPI (0 in, 1 out)
  33 Speakers (Virtual Audio Cable), Windows WASAPI (0 in, 2 out)
  34 Line 1 (Virtual Audio Cable), Windows WASAPI (2 in, 0 out)
  35 Microphone (Realtek Audio), Windows WASAPI (2 in, 0 out)
  36 Line 2 (Virtual Audio Cable), Windows WASAPI (2 in, 0 out)
  37 Line 3 (Virtual Audio Cable), Windows WASAPI (2 in, 0 out)
  38 Speakers 1 (Realtek HD Audio output with SST), Windows WDM-KS (0 in, 2 out)
  39 Speakers 2 (Realtek HD Audio output with SST), Windows WDM-KS (0 in, 6 out)
  40 PC Speaker (Realtek HD Audio output with SST), Windows WDM-KS (2 in, 0 out)
  41 Stereo Mix (Realtek HD Audio Stereo input), Windows WDM-KS (2 in, 0 out)
  42 Microphone (Realtek HD Audio Mic input), Windows WDM-KS (2 in, 0 out)
  43 Speakers (Virtual Cable 1), Windows WDM-KS (0 in, 2 out)
  44 Speakers (Virtual Cable 2), Windows WDM-KS (0 in, 2 out)
  45 Speakers (Virtual Cable 3), Windows WDM-KS (0 in, 2 out)
  46 Mic 1 (Virtual Cable 1), Windows WDM-KS (2 in, 0 out)
  47 Line 1 (Virtual Cable 1), Windows WDM-KS (2 in, 0 out)
  48 S/PDIF 1 (Virtual Cable 1), Windows WDM-KS (2 in, 0 out)
  49 Mic 2 (Virtual Cable 2), Windows WDM-KS (2 in, 0 out)
  50 Line 2 (Virtual Cable 2), Windows WDM-KS (2 in, 0 out)
  51 S/PDIF 2 (Virtual Cable 2), Windows WDM-KS (2 in, 0 out)
  52 Mic 3 (Virtual Cable 3), Windows WDM-KS (2 in, 0 out)
  53 Line 3 (Virtual Cable 3), Windows WDM-KS (2 in, 0 out)
  54 S/PDIF 3 (Virtual Cable 3), Windows WDM-KS (2 in, 0 out)
  55 Headset (@System32\drivers\bthhfenum.sys,#2;%1 Hands-Free AG Audio%0
;(Galaxy Buds2 Pro)), Windows WDM-KS (0 in, 1 out)
  56 Headset (@System32\drivers\bthhfenum.sys,#2;%1 Hands-Free AG Audio%0
;(Galaxy Buds2 Pro)), Windows WDM-KS (1 in, 0 out)
  57 Room Speaker (), Windows WDM-KS (0 in, 2 out)
  58 Headset Earphone (@System32\drivers\bthhfenum.sys,#2;%1 Hands-Free AG Audio%0
;(JBL Flip 3)), Windows WDM-KS (0 in, 1 out)
  59 Headset Microphone (@System32\drivers\bthhfenum.sys,#2;%1 Hands-Free AG Audio%0
;(JBL Flip 3)), Windows WDM-KS (1 in, 0 out)
  60 Speakers (), Windows WDM-KS (0 in, 2 out)
  61 Headphones (), Windows WDM-KS (0 in, 2 out)
  62 Headphones (), Windows WDM-KS (0 in, 2 out)
  63 Headset (@System32\drivers\bthhfenum.sys,#2;%1 Hands-Free AG Audio%0
;(Galaxy Buds Live (6D70))), Windows WDM-KS (0 in, 1 out)
  64 Headset (@System32\drivers\bthhfenum.sys,#2;%1 Hands-Free AG Audio%0
;(Galaxy Buds Live (6D70))), Windows WDM-KS (1 in, 0 out)
  65 Headphones (), Windows WDM-KS (0 in, 2 out)

[mgeier]

MME isn't made for low latency

That's the thing, I don't really mind how large is the latency, as long as it is stable.

Well, typically when someone cares about latency, it is short and stable, and if not it's none of those.

I'd love to use ASIO but unfortunately my signal flow wouldn't allow to use ASIO because I'd need to mix APIs (which I don't it's possible, please correct me if I'm wrong)

Did you try?

I'm not seeing your RME sound card in your device list, am I missing something?

Can you please try all available host APIs?

[davircarvalho]

typically when someone cares about latency, it is short and stable, and if not it's none of those.

Being able to isolate the effect of the latency is a very common use case in acoustic measurements. If you can remove the latency caused by the equipment it's possible to extract lots of useful information from what you're measuring, for example the propagation time from a speaker to a microphone. But more importantly you're able to sync multiple measurements.

If I measure the latency before a measurement I can post process all subsequent measurements to remove the delay caused by the latency, however if the latency keeps changing as shown in my previous reply, then post processing to remove the system's latency is becomes unviable.

I'm not seeing your RME sound card in your device list, am I missing something?

I just didn't have it pluged when I called sd.device_query

0 Microsoft Sound Mapper - Input, MME (2 in, 0 out)
>  1 MADI (1-8) (RME MADIface Pro), MME (2 in, 0 out)
   2 MADI (17-24) (RME MADIface Pro), MME (2 in, 0 out)
   3 Line 1 (Virtual Audio Cable), MME (2 in, 0 out)
   4 Microphone Array (Intel® Smart , MME (2 in, 0 out)
   5 MADI (9-16) (RME MADIface Pro), MME (2 in, 0 out)
   6 MADI (25-32) (RME MADIface Pro), MME (2 in, 0 out)
   7 Microsoft Sound Mapper - Output, MME (0 in, 2 out)
<  8 Speakers (Realtek(R) Audio), MME (0 in, 2 out)
   9 MADI (9-16) (RME MADIface Pro), MME (0 in, 2 out)
  10 MADI (17-24) (RME MADIface Pro), MME (0 in, 2 out)
  11 Speakers (Virtual Audio Cable), MME (0 in, 2 out)
  12 MADI (25-32) (RME MADIface Pro), MME (0 in, 2 out)
  13 MADI (1-8) (RME MADIface Pro), MME (0 in, 2 out)
  14 Primary Sound Capture Driver, Windows DirectSound (2 in, 0 out)
  15 MADI (1-8) (RME MADIface Pro), Windows DirectSound (2 in, 0 out)
  16 MADI (17-24) (RME MADIface Pro), Windows DirectSound (2 in, 0 out)
  17 Line 1 (Virtual Audio Cable), Windows DirectSound (2 in, 0 out)
  18 Microphone Array (Intel® Smart Sound Technology for Digital Microphones), Windows DirectSound (2 in, 0 out)
  19 MADI (9-16) (RME MADIface Pro), Windows DirectSound (2 in, 0 out)
  20 MADI (25-32) (RME MADIface Pro), Windows DirectSound (2 in, 0 out)
  21 Primary Sound Driver, Windows DirectSound (0 in, 2 out)
  22 Speakers (Realtek(R) Audio), Windows DirectSound (0 in, 2 out)
  23 MADI (9-16) (RME MADIface Pro), Windows DirectSound (0 in, 8 out)
  24 MADI (17-24) (RME MADIface Pro), Windows DirectSound (0 in, 8 out)
  25 Speakers (Virtual Audio Cable), Windows DirectSound (0 in, 8 out)
  26 MADI (25-32) (RME MADIface Pro), Windows DirectSound (0 in, 8 out)
  27 MADI (1-8) (RME MADIface Pro), Windows DirectSound (0 in, 8 out)
  28 ASIO MADIface USB, ASIO (68 in, 68 out)
  29 ASIO4ALL v2, ASIO (12 in, 10 out)
  30 Realtek ASIO, ASIO (2 in, 2 out)
  31 MADI (9-16) (RME MADIface Pro), Windows WASAPI (0 in, 8 out)
  32 MADI (17-24) (RME MADIface Pro), Windows WASAPI (0 in, 8 out)
  33 Speakers (Virtual Audio Cable), Windows WASAPI (0 in, 8 out)
  34 MADI (25-32) (RME MADIface Pro), Windows WASAPI (0 in, 8 out)
  35 Speakers (Realtek(R) Audio), Windows WASAPI (0 in, 2 out)
  36 MADI (1-8) (RME MADIface Pro), Windows WASAPI (0 in, 8 out)
  37 MADI (17-24) (RME MADIface Pro), Windows WASAPI (8 in, 0 out)
  38 Line 1 (Virtual Audio Cable), Windows WASAPI (8 in, 0 out)
  39 Microphone Array (Intel® Smart Sound Technology for Digital Microphones), Windows WASAPI (2 in, 0 out)
  40 MADI (9-16) (RME MADIface Pro), Windows WASAPI (8 in, 0 out)
  41 MADI (25-32) (RME MADIface Pro), Windows WASAPI (8 in, 0 out)
  42 MADI (1-8) (RME MADIface Pro), Windows WASAPI (8 in, 0 out)
  43 MADI (25-32) (MADIface MADI (25-32)), Windows WDM-KS (0 in, 8 out)
  44 MADI (25-32) (MADIface MADI (25-32)), Windows WDM-KS (8 in, 0 out)
  45 MADI (1-8) (MADIface MADI (1-8)), Windows WDM-KS (0 in, 8 out)
  46 MADI (1-8) (MADIface MADI (1-8)), Windows WDM-KS (8 in, 0 out)
  47 MADI (9-16) (MADIface MADI (9-16)), Windows WDM-KS (0 in, 8 out)
  48 MADI (9-16) (MADIface MADI (9-16)), Windows WDM-KS (8 in, 0 out)
  49 MADI (17-24) (MADIface MADI (17-24)), Windows WDM-KS (0 in, 8 out)
  50 MADI (17-24) (MADIface MADI (17-24)), Windows WDM-KS (8 in, 0 out)
  51 Headphones (Realtek USB Audio), Windows WDM-KS (0 in, 2 out)
  52 Line (Realtek USB Audio), Windows WDM-KS (0 in, 2 out)
  53 Microphone (Realtek USB Audio), Windows WDM-KS (2 in, 0 out)
  54 Speakers 1 (Realtek HD Audio output with SST), Windows WDM-KS (0 in, 2 out)
  55 Speakers 2 (Realtek HD Audio output with SST), Windows WDM-KS (0 in, 2 out)
  56 PC Speaker (Realtek HD Audio output with SST), Windows WDM-KS (2 in, 0 out)
  57 Stereo Mix (Realtek HD Audio Stereo input), Windows WDM-KS (2 in, 0 out)
  58 Headphones 1 (Realtek HD Audio 2nd output with SST), Windows WDM-KS (0 in, 2 out)
  59 Headphones 2 (Realtek HD Audio 2nd output with SST), Windows WDM-KS (0 in, 2 out)
  60 PC Speaker (Realtek HD Audio 2nd output with SST), Windows WDM-KS (2 in, 0 out)
  61 Microphone (Mic in at front panel (black)), Windows WDM-KS (2 in, 0 out)
  62 Speakers (Virtual Cable 1), Windows WDM-KS (0 in, 8 out)
  63 Mic 1 (Virtual Cable 1), Windows WDM-KS (8 in, 0 out)
  64 Line 1 (Virtual Cable 1), Windows WDM-KS (8 in, 0 out)
  65 S/PDIF 1 (Virtual Cable 1), Windows WDM-KS (8 in, 0 out)
  66 Microphone Array 1 (), Windows WDM-KS (2 in, 0 out)
  67 Microphone Array 2 (), Windows WDM-KS (2 in, 0 out)
  68 Microphone Array 3 (), Windows WDM-KS (4 in, 0 out)

[HaHeho]

Being able to isolate the effect of the latency is a very common use case in acoustic measurements. If you can remove the latency caused by the equipment it's possible to extract lots of useful information from what you're measuring, for example the propagation time from a speaker to a microphone. But more importantly you're able to sync multiple measurements.

If I measure the latency before a measurement I can post process all subsequent measurements to remove the delay caused by the latency, however if the latency keeps changing as shown in my previous reply, then post processing to remove the system's latency is becomes unviable.

Of course, it may be very relevant to compensate for the delay correctly.

I have done extensive measurement series utilizing playrec(). However, I was doing sweep-based impulse response measurements and could create a physical loopback connection. The loopback signal can then be used for deconvolution, which renders potential variances in the latency from playrec() irrelevant. This cannot be implemented in your intended use case?

EDIT: To clarify, the (second) loopback channel is output and recorded simultaneously each time, of course.