Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGABRT pure virtual method called in roc_pipeline benchmark #537

Closed
gavv opened this issue May 11, 2023 · 4 comments
Closed

SIGABRT pure virtual method called in roc_pipeline benchmark #537

gavv opened this issue May 11, 2023 · 4 comments
Labels
benchmarks Benchmarks and micro-benchmarks easy hacks The solution is expected to be straightforward even if you are new to the project help wanted An important and awaited task but we have no human resources for it yet tests
Milestone
next

Comments

@gavv
Copy link
Member

gavv commented May 11, 2023

Reported here: #531 (comment)

just tested the roc-toolkit build on develop branch with developer codebook instruction:

scons -Q --build-3rdparty=all \                                                
  --enable-werror --enable-debug --enable-tests --enable-benchmarks \
  --enable-examples --enable-doxygen test bench

and build was successful and tests got completed. In benchmarks, caught SigABRT

    BENCH   roc_core
2023-05-11T20:27:38+05:30
Running bin/x86_64-pc-linux-gnu/roc-bench-core
Run on (12 X 4500 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 256 KiB (x6)
  L3 Unified 12288 KiB (x1)
Load Average: 6.44, 3.69, 1.95
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------------
Benchmark                                                    Time             CPU   Iterations
----------------------------------------------------------------------------------------------
BM_MpscQueue/PushBack/iterations:5000000/threads:1       0.027 us        0.027 us      5000000
BM_MpscQueue/PushBack/iterations:5000000/threads:2       0.053 us        0.106 us     10000000
BM_MpscQueue/PushBack/iterations:5000000/threads:4       0.040 us        0.161 us     20000000
BM_MpscQueue/PushBack/iterations:5000000/threads:8       0.034 us        0.275 us     40000000
BM_MpscQueue/PushBack/iterations:5000000/threads:16      0.030 us        0.372 us     80000000
BM_MpscQueue/TryPopFront/1/iterations:5000000            0.089 us        0.089 us      5000000
BM_MpscQueue/TryPopFront/2/iterations:5000000            0.064 us        0.064 us      5000000
BM_MpscQueue/TryPopFront/4/iterations:5000000            0.114 us        0.114 us      5000000
BM_MpscQueue/TryPopFront/8/iterations:5000000            0.181 us        0.181 us      5000000
BM_MpscQueue/TryPopFront/16/iterations:5000000           0.421 us        0.260 us      5000000
BM_MpscQueue/PopFront/1/iterations:5000000               0.082 us        0.082 us      5000000
BM_MpscQueue/PopFront/2/iterations:5000000               0.066 us        0.066 us      5000000
BM_MpscQueue/PopFront/4/iterations:5000000               0.109 us        0.109 us      5000000
BM_MpscQueue/PopFront/8/iterations:5000000               0.201 us        0.201 us      5000000
BM_MpscQueue/PopFront/16/iterations:5000000              0.417 us        0.270 us      5000000
BM_Random_Fast                                            27.2 ns         27.1 ns     26086588
    BENCH   roc_pipeline
2023-05-11T20:28:23+05:30
Running bin/x86_64-pc-linux-gnu/roc-bench-pipeline
Run on (12 X 4500 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 256 KiB (x6)
  L3 Unified 12288 KiB (x1)
Load Average: 7.43, 4.28, 2.23
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
--------------------------------------------------------------------------------------------------------
Benchmark                                                              Time             CPU   Iterations
--------------------------------------------------------------------------------------------------------
BM_PipelineContention/Schedule/iterations:1000000/threads:1        0.131 us        0.130 us      1000000
BM_PipelineContention/Schedule/iterations:1000000/threads:2        0.074 us        0.148 us      2000000
BM_PipelineContention/Schedule/iterations:1000000/threads:4        0.053 us        0.210 us      4000000
BM_PipelineContention/Schedule/iterations:1000000/threads:8        0.035 us        0.280 us      8000000
BM_PipelineContention/Schedule/iterations:1000000/threads:16       0.033 us        0.372 us     16000000
BM_PipelinePeakLoad_NoTasks/iterations:3000/real_time               4999 us         3009 us         3000 fa_avg=1.176 fa_p95=20 fb_avg=2.128 fb_p95=20 pr=0 sc=0 ss=0 t_avg=-nan t_p95=inf tp_frm=-nan tp_plc=-nan
BM_PipelinePeakLoad_PreciseSchedOff/iterations:3000/real_time       4999 us         3017 us         3000 fa_avg=2.344 fa_p95=20 fb_avg=0.58 fb_p95=20 pr=1.535k sc=0 ss=3.002k t_avg=0.096 t_p95=20 tp_frm=0 tp_plc=0.291
pure virtual method called
terminate called without an active exception

ERROR: caught SIGABRT

#1: 0x55e3871fb199 _ZN3roc4core20print_backtrace_safeEv+0x17
#2: 0x55e3871f960a _ZN3roc4core14die_gracefullyEPKcb+0x87
#3: 0x55e3871f90cb _ZN3roc4core12_GLOBAL__N_114signal_handlerEiP9siginfo_tPv+0x31
#4: 0x7f142be4fab0 __sigaction+0x50
#5: 0x7f142be9f26c pthread_key_delete+0x14c
#6: 0x7f142be4fa08 gsignal+0x18
#7: 0x7f142be38538 abort+0xd7
#8: 0x7f142c09ca6f _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold+0x66
#9: 0x7f142c0b011c _ZN10__cxxabiv111__terminateEPFvvE+0xc
#10: 0x7f142c0b0189 _ZSt9terminatev+0x19
#11: 0x7f142c0b0ec7 __cxa_pure_virtual+0x27
#12: 0x55e3871f33c8 _ZN3roc8pipeline12PipelineLoop13process_task_ERNS0_12PipelineTaskEb+0x8e
#13: 0x55e3871f2cf4 _ZN3roc8pipeline12PipelineLoop20maybe_process_tasks_Ev+0xf8
#14: 0x55e3871f2bcc _ZN3roc8pipeline12PipelineLoop13process_tasksEv+0x18
#15: 0x55e3871ec8c7 _ZN3roc8pipeline12_GLOBAL__N_112TestPipeline14do_processing_ERNS_3ctl11ControlTaskE+0x23
#16: 0x55e3871ed4cc _ZN3roc3ctl19ControlTaskExecutorINS_8pipeline12_GLOBAL__N_112TestPipelineEE12execute_taskERNS0_11Co+0x0
#17: 0x55e3871f04f7 _ZN3roc3ctl16ControlTaskQueue13execute_task_ERNS0_11ControlTaskE+0x187
#18: 0x55e3871f07b6 _ZN3roc3ctl16ControlTaskQueue14process_tasks_Ev+0xaa
#19: 0x55e3871ef319 _ZN3roc3ctl16ControlTaskQueue3runEv+0x7d
#20: 0x55e3871fa189 _ZN3roc4core6Thread14thread_runner_EPv+0x23
#21: 0x7f142be9d44b pthread_condattr_setpshared+0x51b
#22: 0x7f142bf20e40 __clone+0x120
scons: *** [bench/roc_pipeline] Error -6
@gavv gavv added tests help wanted An important and awaited task but we have no human resources for it yet easy hacks The solution is expected to be straightforward even if you are new to the project benchmarks Benchmarks and micro-benchmarks labels May 11, 2023
@gavv gavv mentioned this issue May 11, 2023
Closed
@Hrick87
Copy link
Contributor

Hrick87 commented Nov 7, 2023 

    BENCH   roc_core
  2023-11-06T21:43:07-08:00
  Running bin/x86_64-pc-linux-gnu/roc-bench-core
  Run on (8 X 3900 MHz CPU s)
  CPU Caches:
    L1 Data 48 KiB (x4)
    L1 Instruction 32 KiB (x4)
    L2 Unified 512 KiB (x4)
    L3 Unified 8192 KiB (x1)
  Load Average: 5.17, 3.75, 2.69
  ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
  ----------------------------------------------------------------------------------------------
  Benchmark                                                    Time             CPU   Iterations
  ----------------------------------------------------------------------------------------------
  BM_MpscQueue/PushBack/iterations:5000000/threads:1       0.021 us        0.021 us      5000000
  BM_MpscQueue/PushBack/iterations:5000000/threads:2       0.064 us        0.127 us     10000000
  BM_MpscQueue/PushBack/iterations:5000000/threads:4       0.055 us        0.222 us     20000000
  BM_MpscQueue/PushBack/iterations:5000000/threads:8       0.031 us        0.245 us     40000000
  BM_MpscQueue/PushBack/iterations:5000000/threads:16      0.026 us        0.218 us     80000000
  BM_MpscQueue/TryPopFront/1/iterations:5000000            0.058 us        0.058 us      5000000
  BM_MpscQueue/TryPopFront/2/iterations:5000000            0.132 us        0.132 us      5000000
  BM_MpscQueue/TryPopFront/4/iterations:5000000            0.161 us        0.161 us      5000000
  BM_MpscQueue/TryPopFront/8/iterations:5000000            0.160 us        0.150 us      5000000
  BM_MpscQueue/TryPopFront/16/iterations:5000000           0.324 us        0.166 us      5000000
  BM_MpscQueue/PopFront/1/iterations:5000000               0.080 us        0.080 us      5000000
  BM_MpscQueue/PopFront/2/iterations:5000000               0.125 us        0.125 us      5000000
  BM_MpscQueue/PopFront/4/iterations:5000000               0.184 us        0.184 us      5000000
  BM_MpscQueue/PopFront/8/iterations:5000000               0.196 us        0.180 us      5000000
  BM_MpscQueue/PopFront/16/iterations:5000000              0.353 us        0.177 us      5000000
  BM_Random_Fast                                            18.0 ns         18.0 ns     38993209
      BENCH   roc_pipeline
  2023-11-06T21:43:44-08:00
  Running bin/x86_64-pc-linux-gnu/roc-bench-pipeline
  Run on (8 X 3900 MHz CPU s)
  CPU Caches:
    L1 Data 48 KiB (x4)
    L1 Instruction 32 KiB (x4)
    L2 Unified 512 KiB (x4)
    L3 Unified 8192 KiB (x1)
  Load Average: 6.14, 4.14, 2.86
  ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
  --------------------------------------------------------------------------------------------------------
  Benchmark                                                              Time             CPU   Iterations
  --------------------------------------------------------------------------------------------------------
  BM_PipelineContention/Schedule/iterations:1000000/threads:1        0.171 us        0.170 us      1000000
  BM_PipelineContention/Schedule/iterations:1000000/threads:2        0.075 us        0.151 us      2000000
  BM_PipelineContention/Schedule/iterations:1000000/threads:4        0.051 us        0.202 us      4000000
  BM_PipelineContention/Schedule/iterations:1000000/threads:8        0.030 us        0.227 us      8000000
  BM_PipelineContention/Schedule/iterations:1000000/threads:16       0.029 us        0.240 us     16000000
  BM_PipelinePeakLoad_NoTasks/iterations:3000/real_time               4999 us         3050 us         3000 fa_avg=1.136 fa_p95=20 fb_avg=2.528 fb_p95=20 pr=0 sc=0 ss=0 t_avg=-nan t_p95=inf tp_frm=-nan tp_plc=-nan
  pure virtual method called
  terminate called without an active exception
  
  ERROR: caught SIGABRT
  
  #1: 0x557b3a586ec4 _ZN3roc4core20print_backtrace_safeEv+0x1b
  #2: 0x557b3a5852f2 _ZN3roc4core14die_gracefullyEPKcb+0x82
  #3: 0x557b3a584db5 _ZN3roc4core12_GLOBAL__N_114signal_handlerEiP9siginfo_tPv+0x35
  #4: 0x7fe29f6b6420 funlockfile+0x60
  #5: 0x7fe29f1a700b gsignal+0xcb
  #6: 0x7fe29f186859 abort+0x12b
  #7: 0x7fe29f55e8d1 __cxa_throw_bad_array_new_length+0x560
  #8: 0x7fe29f56a37c _ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE+0x7c
  #9: 0x7fe29f56a3e7 _ZSt9terminatev+0x17
  #10: 0x7fe29f56b145 __cxa_pure_virtual+0x25
  #11: 0x557b3a57f048 _ZN3roc8pipeline12PipelineLoop13process_task_ERNS0_12PipelineTaskEb+0x92
  #12: 0x557b3a57e95e _ZN3roc8pipeline12PipelineLoop20maybe_process_tasks_Ev+0xfc
  #13: 0x557b3a57e832 _ZN3roc8pipeline12PipelineLoop13process_tasksEv+0x1c
  #14: 0x557b3a57826f _ZN3roc8pipeline12_GLOBAL__N_112TestPipeline14do_processing_ERNS_3ctl11ControlTaskE+0x27
  #15: 0x557b3a578f18 _ZN3roc3ctl19ControlTaskExecutorINS_8pipeline12_GLOBAL__N_112TestPipelineEE12execute_taskERNS0_11Co+0x0
  #16: 0x557b3a57c065 _ZN3roc3ctl16ControlTaskQueue13execute_task_ERNS0_11ControlTaskE+0x16d
  #17: 0x557b3a57c31c _ZN3roc3ctl16ControlTaskQueue14process_tasks_Ev+0xae
  #18: 0x557b3a57aed8 _ZN3roc3ctl16ControlTaskQueue3runEv+0x7e
  #19: 0x557b3a585e3b _ZN3roc4core6Thread14thread_runner_EPv+0x27
  #20: 0x7fe29f6aa609 start_thread+0xd9
  #21: 0x7fe29f283133 clone+0x43
  Aborted (core dumped)
  scons: *** [bench/roc_pipeline] Error 134

I've come across a very similar error while running the same command although the error code is different on Lubuntu 20.04. I figured this would help confirm it is indeed not an issue isolated to just Arch as was mentioned in issue #531

@nolan-veed
Copy link
Contributor

Is this issue still valid?
I'm actually seeing a different error.

nolan@nolan-veed:~/nolan-veed/roc-toolkit$ scons -Q --compiler-launcher=ccache --build-3rdparty=all   --enable-werror --enable-debug --enable-tests --enable-benchmarks   --enable-examples --enable-doxygen bench
    BENCH   roc_core
2023-12-27T20:51:13+00:00
Running bin/x86_64-pc-linux-gnu/roc-bench-core
Run on (16 X 4900 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1280 KiB (x8)
  L3 Unified 24576 KiB (x1)
Load Average: 0.50, 0.76, 1.12
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------------
Benchmark                                                    Time             CPU   Iterations
----------------------------------------------------------------------------------------------
BM_MpscQueue/PushBack/iterations:5000000/threads:1       0.027 us        0.027 us      5000000
BM_MpscQueue/PushBack/iterations:5000000/threads:2       0.058 us        0.117 us     10000000
BM_MpscQueue/PushBack/iterations:5000000/threads:4       0.049 us        0.195 us     20000000
BM_MpscQueue/PushBack/iterations:5000000/threads:8       0.043 us        0.341 us     40000000
BM_MpscQueue/PushBack/iterations:5000000/threads:16      0.023 us        0.370 us     80000000
BM_MpscQueue/TryPopFront/1/iterations:5000000            0.060 us        0.060 us      5000000
BM_MpscQueue/TryPopFront/2/iterations:5000000            0.134 us        0.134 us      5000000
BM_MpscQueue/TryPopFront/4/iterations:5000000            0.230 us        0.230 us      5000000
BM_MpscQueue/TryPopFront/8/iterations:5000000            0.319 us        0.319 us      5000000
BM_MpscQueue/TryPopFront/16/iterations:5000000           0.308 us        0.209 us      5000000
BM_MpscQueue/PopFront/1/iterations:5000000               0.084 us        0.084 us      5000000
BM_MpscQueue/PopFront/2/iterations:5000000               0.126 us        0.126 us      5000000
BM_MpscQueue/PopFront/4/iterations:5000000               0.240 us        0.240 us      5000000
BM_MpscQueue/PopFront/8/iterations:5000000               0.370 us        0.370 us      5000000
BM_MpscQueue/PopFront/16/iterations:5000000              0.386 us        0.315 us      5000000
BM_Random_Fast                                            16.3 ns         16.3 ns     43217227
    BENCH   roc_pipeline
2023-12-27T20:51:57+00:00
Running bin/x86_64-pc-linux-gnu/roc-bench-pipeline
Run on (16 X 4900 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1280 KiB (x8)
  L3 Unified 24576 KiB (x1)
Load Average: 1.62, 1.06, 1.20
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
--------------------------------------------------------------------------------------------------------
Benchmark                                                              Time             CPU   Iterations
--------------------------------------------------------------------------------------------------------
BM_PipelineContention/Schedule/iterations:1000000/threads:1        0.141 us        0.141 us      1000000
BM_PipelineContention/Schedule/iterations:1000000/threads:2        0.075 us        0.150 us      2000000
BM_PipelineContention/Schedule/iterations:1000000/threads:4        0.059 us        0.234 us      4000000
BM_PipelineContention/Schedule/iterations:1000000/threads:8        0.047 us        0.376 us      8000000
BM_PipelineContention/Schedule/iterations:1000000/threads:16       0.035 us        0.524 us     16000000
	BM_PipelinePeakLoad_NoTasks/iterations:3000/real_time              15005 us        15003 us         3000 fa_avg=0.216 fa_p95=20 fb_avg=22.8137G fb_p95=inf pr=0 sc=0 ss=0 t_avg=-nan t_p95=inf tp_frm=-nan tp_plc=-nan
BM_PipelinePeakLoad_PreciseSchedOff/iterations:3000/real_time       4999 us         3027 us         3000 fa_avg=2.11 fa_p95=20 fb_avg=7.699 fb_p95=70 pr=1.223k sc=0 ss=3.001k t_avg=1087.58 t_p95=2.87k tp_frm=0 tp_plc=0.302

src/internal_modules/roc_ctl/control_task_queue.cpp:408: error: roc_panic()

ERROR: roc_ctl: control task queue: unexpected already cancelled task in cancel

Backtrace:
#1: 0x562F5D334FFE roc::core::print_backtrace_full()+0x6E
#2: 0x562F5D3333B2 roc::core::die_gracefully(char const*, bool)+0x72
#3: 0x562F5D32E9BD roc::core::panic(char const*, char const*, int, char const*, ...)+0x18D
#4: 0x562F5D327240 roc::ctl::ControlTaskQueue::cancel_task_(roc::ctl::ControlTask&, unsigned int)+0xC0
#5: 0x562F5D326E48 roc::ctl::ControlTaskQueue::renew_scheduling_(roc::ctl::ControlTask&, unsigned int, long, unsigned int)+0x158
#6: 0x562F5D327E4E roc::ctl::ControlTaskQueue::fetch_ready_task_()+0x25E
#7: 0x562F5D327B6F roc::ctl::ControlTaskQueue::fetch_task_()+0x2F
#8: 0x562F5D32688D roc::ctl::ControlTaskQueue::process_tasks_()+0x2D
#9: 0x562F5D3267E3 roc::ctl::ControlTaskQueue::run()+0x83
#10: 0x562F5D333B19 roc::core::Thread::thread_runner_(void*)+0x19
#11: 0x7F0C64E94AC3 pthread_condattr_setpshared+0x513
#12: 0x7F0C64F26660 __xmknodat+0x230
Aborted (core dumped)
scons: *** [bench/roc_pipeline] Error 134

@gavv
Copy link
Member Author

gavv commented Dec 28, 2023

Is this issue still valid?
I'm actually seeing a different error.

@nolan-veed this is a different problem I think.

I often see similar panic on CI (where we don't run benchmarks currently, only tests). I guess it's a race either in queue, or in both tests & benchmarks. Probably it's related to #435, but I'm not sure. Never had time to debug it yet..

@gavv gavv added this to Roc Toolkit Jul 6, 2024
@gavv gavv moved this to Help wanted in Roc Toolkit Jul 6, 2024
sm00th added a commit to sm00th/roc-toolkit that referenced this issue Dec 15, 2024
@sm00th
roc_pipeline benchmark: fix a task completer race
21412ee 
In peakload benchmarks it is possible for the TaskThread object to be
removed along with the pipeline_task_completed() callback before all
tasks are completed. This results in a SIGABRT with "pure virtual method
called" message.

To fix this move pipeline_task_completed() to TestPipeline class that has
a longer lifespan.

Fixes: roc-streaming#537
sm00th added a commit to sm00th/roc-toolkit that referenced this issue Dec 15, 2024
@sm00th
roc_pipeline benchmark: fix a task completer race
063b1bc 
In peakload benchmarks it is possible for the TaskThread object to be
removed along with the pipeline_task_completed() callback before all
tasks are completed. This results in a SIGABRT with "pure virtual method
called" message.

To fix this move pipeline_task_completed() to TestPipeline class that has
a longer lifespan.

Fixes: roc-streaming#537
@sm00th sm00th mentioned this issue Dec 15, 2024
Merged
@gavv gavv added this to the next milestone Dec 19, 2024
gavv pushed a commit to sm00th/roc-toolkit that referenced this issue Dec 19, 2024
@sm00th
roc-streaminggh-537: roc_pipeline benchmark: fix a task completer race
ee3f557 
In peakload benchmarks it is possible for the TaskThread object to be
removed along with the pipeline_task_completed() callback before all
tasks are completed. This results in a SIGABRT with "pure virtual method
called" message.

To fix this move pipeline_task_completed() to TestPipeline class that has
a longer lifespan.

Fixes: roc-streaming#537
gavv pushed a commit that referenced this issue Dec 19, 2024
@sm00th @gavv
gh-537: roc_pipeline benchmark: fix a task completer race
d3026ed 
In peakload benchmarks it is possible for the TaskThread object to be
removed along with the pipeline_task_completed() callback before all
tasks are completed. This results in a SIGABRT with "pure virtual method
called" message.

To fix this move pipeline_task_completed() to TestPipeline class that has
a longer lifespan.

Fixes: #537
@gavv
Copy link
Member Author

gavv commented Dec 19, 2024

Fix is merged

@gavv gavv closed this as completed Dec 19, 2024
@github-project-automation github-project-automation bot moved this from Help wanted to Done in Roc Toolkit Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
benchmarks Benchmarks and micro-benchmarks easy hacks The solution is expected to be straightforward even if you are new to the project help wanted An important and awaited task but we have no human resources for it yet tests
Projects
Roc Toolkit
Status: Done
Milestone
next
Development

No branches or pull requests
3 participants
@gavv @Hrick87 @nolan-veed