debug: add scripts to help debug FW crashes on HW and on qemu by lgirdwood · Pull Request #10583 · thesofproject/sof

lgirdwood · 2026-02-27T15:32:03Z

START - test_ipc4_pipeline_with_dp
[    0.241300] <inf> sof_boot_test: Starting IPC4 pipeline with DP test
[    0.241321] <inf> pipe: pipeline new pipe_id 3 priority 0

    Assertion failed at WEST_TOPDIR/sof/zephyr/test/userspace/test_ipc4_pipeline.c:156: userspace_ipc4_pipeline_test_ipc4_pipeline_with_dp: (NULL is NULL)
[    0.241512] <err> os:  ** FATAL EXCEPTION
[    0.241585] <err> os:  ** CPU 0 EXCCAUSE 28 (load prohibited)
[    0.241619] <err> os:  **  PC 0x1081c2 VADDR 0x3
[    0.241649] <err> os:  **  PS 0x60120
[    0.241684] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:1 CALLINC:2)
[    0.241729] <err> os:  **  A0 0x8010b05e  SP 0x1b6d60  A2 0x2  A3 (nil)
[    0.241764] <err> os:  **  A4 0x20  A5 0xffffffff  A6 0x4  A7 (nil)
[    0.241798] <err> os:  **  A8 (nil)  A9 0x1 A10 0x3 A11 0x1b6eb0
[    0.241833] <err> os:  ** A12 (nil) A13 0x1 A14 0x16e284 A15 (nil)
[    0.241867] <err> os:  ** LBEG 0x142030 LEND 0x142034 LCOUNT (nil)
[    0.241899] <err> os:  ** SAR 0x18
[    0.241931] <err> os:  **  THREADPTR (nil)


Backtrace:0x001081bf:0x001b6d60 0x0010b05b:0x001b6de0 0x0010aa7a:0x001b6e30 0x00108c46:0x001b6e70 0x001401d1:0x001b6ea0 0x00140785:0x001b6ed0 0x0010cc21:0x001b6fc0 0x00108dc6:0x001b6fe0 

[    0.242622] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[    0.242716] <err> os: Current thread: 0x1ad0f0 (test_ipc4_pipeline_with_dp)
[    0.253529] <err> zephyr: Halting system
FAILED: zephyr/CMakeFiles/run_qemu /home/lrg/work/sof/build-qemu_xtensa_mmu/zephyr/CMakeFiles/run_qemu 
cd /home/lrg/work/sof/build-qemu_xtensa_mmu && /home/lrg/zephyr-sdk-0.17.4/sysroots/x86_64-pokysdk-linux/usr/bin/qemu-system-xtensa -machine sim -semihosting -nographic -cpu dc233c -net none -pidfile qemu.pid -chardev stdio,id=con,mux=on -serial chardev:con -mon chardev=con,mode=readline -icount shift=6,align=off,sleep=off -rtc clock=vm -kernel /home/lrg/work/sof/build-qemu_xtensa_mmu/zephyr/zephyr.elf
ninja: build stopped: subcommand failed.

FATAL ERROR: command exited with status 1: /usr/bin/cmake --build /home/lrg/work/sof/build-qemu_xtensa_mmu --target run --


[sof-qemu-run] QEMU process terminated.

[sof-qemu-run] Detected crash signature in standard output!

====================================
Running sof-crash-decode.py Analysis
====================================

Found 18 registers and 8 backtrace elements in crash dump.
Parsing objdump (this may take a few seconds)...
Running /home/lrg/zephyr-sdk-0.17.4/xtensa-dc233c_zephyr-elf/bin/xtensa-dc233c_zephyr-elf-objdump -d -S -l "/home/lrg/work/sof/build-qemu_xtensa_mmu/zephyr/zephyr.elf" ...

--- Summary ---
PS Register Legend:
  INTLEVEL : Interrupt Level          EXCM : Exception Mode
  UM       : User Mode (1=User)       RING : Privilege Ring
  OWB      : Old Window Base          WOE  : Window Overflow Enable
  CALLINC  : Call Increment

PC   : 0x001081c2 -> <cbvprintf_package>
      /home/lrg/work/sof/zephyr/lib/os/cbprintf_packaged.c:469
      if (*++fmt == '\0') {
      001081c2: 010272        	l8ui	a7, a2, 1

EXCCAUSE: 28 (LoadStoreProhibited)
VADDR: 0x00000003
SP   : 0x001b6d60 -> <unknown, DT: memory@0, Region: RAM, Section: noinit>
PS   : 0x00060120 -> [INTLEVEL:0 | UM:1 | RING:0 | OWB:1 | CALLINC:2 | WOE:1]



--- Physical Windowed Registers (A) ---
A0   : 0x8010b05e -> <z_log_msg_runtime_vcreate>
      /home/lrg/work/sof/zephyr/subsys/logging/log_msg.c:357
      package_flags, fmt, ap2);
      __ASSERT_NO_MSG(plen >= 0);
      0010b05e: 9788      	l32i.n	a8, a7, 36

A2   : 0x00000002 -> <xthal_window_spill_nw>
      /home/lrg/work/sof/modules/hal/xtensa/src/hal/windowspill_asm.S:329
      00000002: //  Here, a3 = original WINDOWBASE;

A4   : 0x00000020 -> <unknown, DT: memory@0, Region: IDT_LIST, Section: .xtensa.info>
A5   : 0xffffffff -> <unknown/no code section>
A6   : 0x00000004 -> <xthal_window_spill_nw>
      /home/lrg/work/sof/modules/hal/xtensa/src/hal/windowspill_asm.S:329
      00000002: //  Here, a3 = original WINDOWBASE;

A9   : 0x00000001 -> <xthal_window_spill_nw>
      /home/lrg/work/sof/modules/hal/xtensa/src/hal/windowspill_asm.S:322
      00000001: addi	a3, a3, -1	// decrement towards original WINDOWBASE

A10  : 0x00000003 -> <xthal_window_spill_nw>
      /home/lrg/work/sof/modules/hal/xtensa/src/hal/windowspill_asm.S:329
      00000002: //  Here, a3 = original WINDOWBASE;

A11  : 0x001b6eb0 -> <unknown, DT: memory@0, Region: RAM, Section: noinit>
A13  : 0x00000001 -> <xthal_window_spill_nw>
      /home/lrg/work/sof/modules/hal/xtensa/src/hal/windowspill_asm.S:322
      00000001: addi	a3, a3, -1	// decrement towards original WINDOWBASE

A14  : 0x0016e284 -> <unknown, DT: memory@0, Region: RAM, Section: .rodata>

--- Saved Stack Registers (AR) ---

--- Backtrace Decode ---
Frame 0: SP = 0x001b6d60
PC   : 0x001081bf -> <cbvprintf_package>
      /home/lrg/work/sof/zephyr/lib/os/cbprintf_packaged.c:679
      s_idx++;
      001081be: 551b      	addi.n	a5, a5, 1

Frame 1: SP = 0x001b6de0
PC   : 0x0010b05b -> <z_log_msg_runtime_vcreate>
      /home/lrg/work/sof/zephyr/subsys/logging/log_msg.c:355
      plen = cbvprintf_package(NULL, Z_LOG_MSG_ALIGN_OFFSET,
      0010b05b: 0008e0        	callx8	a8

Frame 2: SP = 0x001b6e30
PC   : 0x0010aa7a -> <z_log_vprintk>
      /home/lrg/work/sof/zephyr/subsys/logging/log_core.c:225
      z_log_msg_runtime_vcreate(Z_LOG_LOCAL_DOMAIN_ID, NULL,
      0010aa7a: 0008e0        	callx8	a8

Frame 3: SP = 0x001b6e70
PC   : 0x00108c46 -> <vprintk>
      /home/lrg/work/sof/zephyr/lib/os/printk.c:104
      z_log_vprintk(fmt, ap);
      00108c46: 0008e0        	callx8	a8

Frame 4: SP = 0x001b6ea0
PC   : 0x001401d1 -> <z_zassert$constprop$0>
      /home/lrg/work/sof/zephyr/subsys/testsuite/ztest/include/zephyr/ztest_assert.h:89
      vprintk(msg, vargs);
      001401d1: 0008e0        	callx8	a8

Frame 5: SP = 0x001b6ed0
PC   : 0x00140785 -> <_userspace_ipc4_pipeline_test_ipc4_pipeline_with_dp_wrapper>
      /home/lrg/work/sof/sof/zephyr/test/userspace/test_ipc4_pipeline.c:156
      zassert_not_null(NULL, 3, "pipeline 3 id is not 3");
      00140785: 0008e0        	callx8	a8

Frame 6: SP = 0x001b6fc0
PC   : 0x0010cc21 -> <test_cb>
      run_test_functions():
      /home/lrg/work/sof/zephyr/subsys/testsuite/ztest/src/ztest.c:328
      test->test(data);
      0010cc21: 0003e0        	callx8	a3

Frame 7: SP = 0x001b6fe0
PC   : 0x00108dc6 -> <z_thread_entry>
      /home/lrg/work/sof/zephyr/lib/os/thread_entry.c:60
      sys_rand_get((uint8_t *)&stack_guard, sizeof(stack_guard));
      __stack_chk_guard = stack_guard;
      __stack_chk_guard <<= 8;
      #endif	/* CONFIG_STACK_CANARIES */
      entry(p1, p2, p3);
      00108dc6: 0002e0        	callx8	a2

lrg@moth:~/work/sof/build-qemu_xtensa_mmu$

This allows CI to unblock and proceed to check results. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Add a helper to assist in debugging FW crashes. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Utility to help with developer, CI, AI usage. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Copilot

Pull request overview

This pull request adds debugging tools to help analyze firmware crashes on hardware and QEMU environments. The PR introduces automated scripts that run QEMU tests, monitor for crashes, and decode crash dumps into human-readable format with source code context.

Changes:

Added automated QEMU test runner (sof-qemu-run.py) that monitors test execution, detects crashes, and automatically invokes crash analysis
Added crash dump decoder (sof-crash-decode.py) that parses Xtensa/Zephyr crash dumps, correlates register values and backtraces to source code using objdump
Added QEMU exit mechanism to cleanly terminate tests after completion in CI environments

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
scripts/sof-qemu-run.sh	Bash wrapper script that sets up the virtual environment and invokes the Python QEMU runner
scripts/sof-qemu-run.py	Python script that spawns QEMU via west, monitors output for crashes, and triggers crash decoding
scripts/sof-crash-decode.py	Python script that parses crash dumps, decodes registers/backtraces, and maps addresses to source code
app/src/main.c	Added inline assembly function to cleanly exit QEMU using simcall after boot tests complete
app/boards/qemu_xtensa_dc233c_mmu.conf	Enabled CONFIG_REBOOT for QEMU shutdown support
app/boards/qemu_xtensa_dc233c.conf	Enabled CONFIG_REBOOT for QEMU shutdown support

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/sof-qemu-run.py

Copilot · 2026-02-27T15:37:55Z

scripts/sof-qemu-run.py

+    print(f"Starting QEMU test runner. Monitoring for crashes (Build Dir: {args.build_dir})...")
+
+    # We will use pexpect to spawn the west command to get PTY features
+    import shutil


The shutil module is imported inside the main() function on line 87, but it's only used once. For better readability and consistency with other imports, consider moving it to the top of the file with the other imports.

Copilot · 2026-02-27T15:37:55Z

scripts/sof-crash-decode.py

+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Intel Corporation. All rights reserved.
+"""
+decode_crash.py - Zephyr Xtensa Crash Dump Decoder


The docstring on line 5 refers to the script as "decode_crash.py", but the actual filename is "sof-crash-decode.py". Update the docstring to match the actual filename.

scripts/sof-crash-decode.py

scripts/sof-qemu-run.sh

scripts/sof-qemu-run.py

app/src/main.c

scripts/sof-qemu-run.sh

scripts/sof-qemu-run.py

Apply changes from review. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood <lgirdwood@gmail.com>

lrgirdwo added 3 commits February 27, 2026 15:20

qemu: quit simulation when tests are completed

f740cb5

This allows CI to unblock and proceed to check results. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

scripts: add a firmware crash decode helper

82e5e63

Add a helper to assist in debugging FW crashes. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

qemu: add a helper script to assist with qemu debug

4ce2ed0

Utility to help with developer, CI, AI usage. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Copilot AI review requested due to automatic review settings February 27, 2026 15:32

lgirdwood requested review from dbaluta, kv2019i, lbetlej, mmaka1 and plbossart as code owners February 27, 2026 15:32

Copilot started reviewing on behalf of lgirdwood February 27, 2026 15:32 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

cleanup: batch review updates for pr thesofproject#10583

018e5a9

Apply changes from review. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood <lgirdwood@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

debug: add scripts to help debug FW crashes on HW and on qemu#10583

debug: add scripts to help debug FW crashes on HW and on qemu#10583
lgirdwood wants to merge 4 commits intothesofproject:mainfrom
lgirdwood:crash

lgirdwood commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lgirdwood commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants