Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
0
votes
0
answers
65
views
Need help to understand a kernel "deadlock" in armhf kernel 5.4
I'm struggling to understand a condition that occurs rarely (MTBF measured in weeks to months), but results in a fatal kernel lockup when it does. The environment is kernel 5.4.0, running on a dual-core ARM A9 32-bit (armhf) processor in an AMD/Xilinx Zynq 7020 SoC. I've been able to examine stack t...
I'm struggling to understand a condition that occurs rarely (MTBF measured in weeks to months), but results in a fatal kernel lockup when it does.
The environment is kernel 5.4.0, running on a dual-core ARM A9 32-bit (armhf) processor in an AMD/Xilinx Zynq 7020 SoC.
I've been able to examine stack traces on two failed devices using the Xilinx Vitis system debugger. In both cases, both cores were waiting for (attempting to lock) the same spinlock. The spinlock is the
pi_lock
member of the struct task_struct
structure associated with a threaded IRQ handler.
Core #0 is running the idle process, responding to an interrupt, attempting to wake up the IRQ handler thread. Core #1 is reading the /proc//stat
sysfs pseudofile associated with the IRQ handler thread (while reading all of the sysfs pseudofiles of the form /proc//stat
on behalf of pidof
, probably at the request of sessionclean
, invoked by apache2
).
The interrupt being handled on one device is for the BQ25890 battery charger (asserted ~zero to 10 times per second, and on the other device, for a custom kernel module (~100/sec.). Both interrupts use the default primary IRQ handler.
The current hypothesis is that that core #0 got stuck first, followed some time later by core #1, rather than both occurring simultaneously, but this is speculative.
Stack traces for the two cores and threaded IRQ handler source for one of the devices are below.
**Questions**
Is there enough information in the stack traces to provide a clue as to what went wrong? If so, what went wrong? If not, what additional information is needed? One of the devices is still available, and some (not all) variables can be interrogated.
Is the fact that interrupts are disabled (cpsr.i == 1 for both cores) significant?
Please note that because of the extremely long time between failures, black-box testing techniques - that is, making changes in order to observe the result - are not practical.
**ARM Cortex-A9 MPCore #0 (Suspended)**
0x80165d68: ./include/linux/compiler.h, line 199
0x80165d68 arch_spin_lock(): .../arch/arm/include/asm/spinlock.h, line 75
0x8076f774: ./include/linux/spinlock.h, line 193
0x8076f774: ...include/linux/spinlock_api_smp.h, line 119
0x8076f774 _raw_spin_lock_irqsave(): kernel/locking/spinlock.c, line 159
0x8014a394 try_to_wake_up(): kernel/sched/core.c, line 2551
0x8014a724 wake_up_process(): kernel/sched/core.c, line 2667
0x8016f600 __irq_wake_thread(): kernel/irq/handle.c, line 134
0x8016f820 __handle_irq_event_percpu(): kernel/irq/handle.c, line 167
0x8016f888 handle_irq_event_percpu(): kernel/irq/handle.c, line 189
0x8016f924 handle_irq_event(): kernel/irq/handle.c, line 206
0x80174230 handle_level_irq(): kernel/irq/chip.c, line 650
0x8016e808: ./include/linux/irqdesc.h, line 156
0x8016e808 generic_handle_irq(): kernel/irq/irqdesc.c, line 644
0x80447a20: drivers/gpio/gpio-zynq.c, line 632
0x80447a20 zynq_gpio_irqhandler(): drivers/gpio/gpio-zynq.c, line 661
0x8016e808: ./include/linux/irqdesc.h, line 156
0x8016e808 generic_handle_irq(): kernel/irq/irqdesc.c, line 644
0x8016eec4 __handle_domain_irq(): kernel/irq/irqdesc.c, line 681
0x80102300: ./include/linux/irqdesc.h, line 174
0x80102300 gic_handle_irq(): drivers/irqchip/irq-gic.c, line 383
0x80101a70 __irq_svc(): arch/arm/kernel/entry-armv.S, line 212
0x805827c0 cpuidle_enter_state(): .../arch/arm/include/asm/irqflags.h, line 39
0x805829d8 cpuidle_enter(): drivers/cpuidle/cpuidle.c, line 344
0x8014ecf0: kernel/sched/idle.c, line 117
0x8014ecf0: kernel/sched/idle.c, line 201
0x8014ecf0 do_idle(): kernel/sched/idle.c, line 263
0x8014eeac cpu_startup_entry(): kernel/sched/idle.c, line 355
0x80769828 rest_init(): init/main.c, line 451
0x80c00b30 arch_call_rest_init(): init/main.c, line 572
0x80c00f78 start_kernel(): init/main.c, line 784
0x00000000
0x00000000
**The offending lock**
lock arch_spinlock_t * {{slock=0x94449440, tickets={owner=0x9440, next=0x9444}}}
slock u32 0x94449440
tickets struct __raw_tickets {owner=0x9440, next=0x9444}
owner u16 0x9440
next u16 0x9444
**Threaded IRQ handler source**
Note that this code does not appear in the call stacks, but is the code for the thread that core #0 is attempting to wake up.
On one of the devices:
static irqreturn_t bq25890_irq_handler_thread(int irq, void *private)
{
struct bq25890_device *bq = private;
int ret;
struct bq25890_state state;
ret = bq25890_get_chip_state(bq, &state);
if (ret lock);
bq->state = state;
mutex_unlock(&bq->lock);
power_supply_changed(bq->charger);
handled:
return IRQ_HANDLED;
}
On the other device:
static irqreturn_t core100_irq_handler( int irq, void* data )
{
struct core100_drvdata* p_info;
struct core100_block_header* next;
unsigned long iflags;
p_info = ( struct core100_drvdata* ) data;
if ( ( p_info->state != state_reading ) && ( p_info->state != state_writing ) )
{
return IRQ_HANDLED;
}
next = &p_info->current_block->next->header;
next->count_at_pps = read_register( p_info->count_pps_register );
next->frc_at_pps = read_register( p_info->frc_pps_register );
next->count = adc_counter();
next->frc = free_running_counter();
spin_lock_irqsave( &p_info->state_lock, iflags );
p_info->dma_head += bytes_per_block( p_info );
if ( p_info->dma_head - p_info->dma_tail > bytes_per_buffer( p_info ) )
{
p_info->dma_tail = p_info->dma_head - bytes_per_buffer( p_info );
}
p_info->current_block = p_info->current_block->next;
wake_up_interruptible( &p_info->wq );
spin_unlock_irqrestore( &p_info->state_lock, iflags );
return IRQ_HANDLED;
}
Kernel source:
[struct task_struct]
**ARM Cortex-A9 MPCore #1 (Suspended)**
0x80165d68: ./include/linux/compiler.h, line 199
0x80165d68 arch_spin_lock(): .../arch/arm/include/asm/spinlock.h, line 75
0x8076f774: ./include/linux/spinlock.h, line 193
0x8076f774: ...include/linux/spinlock_api_smp.h, line 119
0x8076f774 _raw_spin_lock_irqsave(): kernel/locking/spinlock.c, line 159
0x80148914 task_rq_lock(): kernel/sched/core.c, line 109
0x8014e22c: kernel/sched/cputime.c, line 281
0x8014e22c thread_group_cputime(): kernel/sched/cputime.c, line 326
0x8014e650 thread_group_cputime_adjusted(): kernel/sched/cputime.c, line 678
0x802ccb5c do_task_stat(): fs/proc/array.c, line 510
0x802cdaf4 proc_tgid_stat(): fs/proc/array.c, line 632
0x802c7c0c proc_single_show(): fs/proc/base.c, line 756
0x80281b18 seq_read(): fs/seq_file.c, line 229
0x8025dd1c __vfs_read(): fs/read_write.c, line 425
0x8025de60 vfs_read(): fs/read_write.c, line 461
0x8025e088 ksys_read(): fs/read_write.c, line 587
0x8025e0ec: fs/read_write.c, line 597
0x8025e0ec __se_sys_read(): fs/read_write.c, line 595
0x80101000 __idmap_text_end()
0x76e627e6
0x76edc616
Here's an alternative view, showing the calls stacks associated with their calling processes. "Swapper" is an archaic term for the idle process:
Kernel
0 swapper/0 (Suspended), ARM Cortex-A9 MPCore #0
0x80165d68: ./include/linux/compiler.h, line 199
0x80165d68 arch_spin_lock(): .../arch/arm/include/asm/spinlock.h, line 75
0x8076f774: ./include/linux/spinlock.h, line 193
0x8076f774: ...include/linux/spinlock_api_smp.h, line 119
0x8076f774 _raw_spin_lock_irqsave(): kernel/locking/spinlock.c, line 159
11950 pidof
11950 (Suspended), ARM Cortex-A9 MPCore #1
0x80165d68: ./include/linux/compiler.h, line 199
0x80165d68 arch_spin_lock(): .../arch/arm/include/asm/spinlock.h, line 75
0x8076f774: ./include/linux/spinlock.h, line 193
0x8076f774: ...include/linux/spinlock_api_smp.h, line 119
0x8076f774 _raw_spin_lock_irqsave(): kernel/locking/spinlock.c, line 159
**RCU Configuration**
# RCU Subsystem
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
# end of RCU Subsystem
# RCU Debugging
# CONFIG_RCU_PERF_TEST is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
# CONFIG_RCU_TRACE is not set
# CONFIG_RCU_EQS_DEBUG is not set
# end of RCU Debugging
**Kernel Thread Prioities**
# ps -eo pid,tid,class,rtprio,ni,pri,wchan:14,comm | grep -E '(PID|bq25890|raw_dma)'
PID TID CLS RTPRIO NI PRI WCHAN COMMAND
214 214 FF 50 - 90 irq_thread irq/59-bq25890_
229 229 FF 50 - 90 irq_thread irq/60-raw_dma
**Processor Status Registers**
On at least one failed system, the interrupt disable
bit is set on both processors. That seems wrong, although the fast interrupt disable
bit is not set. If I set a breakpoint in arch_spin_lock()
on a working system, the interrupt disable bit is not set.
It would seem that the only way for the spinlock to be released is in response to an interrupt, since the active threads on both cores aren't going to do it.
Core #0
cpsr 200f0193 537854355
n 0 0 Negative condition code flag
z 0 0 Zero condition code flag
c 1 1 Carry condition code flag
v 0 0 Overflow condition code flag
q 0 0 Cumulative saturation flag
it 00 0 If-Then execution state bits
j 0 0 Jazelle bit
ge f 15 SIMD Greater than or Equal flags
e 0 0 Endianness execution state bit
a 1 1 Asynchronous abort disable bit
i 1 1 Interrupt disable bit
f 0 0 Fast interrupt disable bit
t 0 0 Thumb execution state bit
m 13 19 Mode field
irq
sp 80d94400 2161722368
lr 80101e20 2148539936
spsr 200f0193 537854355
n 0 0 Negative condition code flag
z 0 0 Zero condition code flag
c 1 1 Carry condition code flag
v 0 0 Overflow condition code flag
q 0 0 Cumulative saturation flag
it 00 0 If-Then execution state bits
j 0 0 Jazelle bit
ge f 15 SIMD Greater than or Equal flags
e 0 0 Endianness execution state bit
a 1 1 Asynchronous abort disable bit
i 1 1 Interrupt disable bit
f 0 0 Fast interrupt disable bit
t 0 0 Thumb execution state bit
m 13 19 Mode field
Core #1
cpsr 200f0093 537854099
n 0 0 Negative condition code flag
z 0 0 Zero condition code flag
c 1 1 Carry condition code flag
v 0 0 Overflow condition code flag
q 0 0 Cumulative saturation flag
it 00 0 If-Then execution state bits
j 0 0 Jazelle bit
ge f 15 SIMD Greater than or Equal flags
e 0 0 Endianness execution state bit
a 0 0 Asynchronous abort disable bit
i 1 1 Interrupt disable bit
f 0 0 Fast interrupt disable bit
t 0 0 Thumb execution state bit
m 13 19 Mode field
irq
sp 80d94440 2161722432
lr 80101a00 2148538880
spsr 60030193 1610809747
n 0 0 Negative condition code flag
z 1 1 Zero condition code flag
c 1 1 Carry condition code flag
v 0 0 Overflow condition code flag
q 0 0 Cumulative saturation flag
it 00 0 If-Then execution state bits
j 0 0 Jazelle bit
ge 3 3 SIMD Greater than or Equal flags
e 0 0 Endianness execution state bit
a 1 1 Asynchronous abort disable bit
i 1 1 Interrupt disable bit
f 0 0 Fast interrupt disable bit
t 0 0 Thumb execution state bit
m 13 19 Mode field
Omnia tu scis quod malum
(1 rep)
Nov 24, 2023, 10:59 PM
• Last activity: Nov 29, 2023, 05:49 PM
12
votes
2
answers
30815
views
Compiled kernel 4.19 will not boot: "Kernel panic not syncing : System is deadlocked on memory"
I am compiling kernel 4.19 on Ubuntu 14.04 because I have a assignment to add a system call, but when I try to boot this kernel there's a error: Kernel panic - not syncing: System is deadlocked on memory Nothing happens after this message appears [![boot sequence stop at deadlock error][1]][1] 1. I...
I am compiling kernel 4.19 on Ubuntu 14.04 because I have a assignment to add a system call, but when I try to boot this kernel there's a error:
Kernel panic - not syncing: System is deadlocked on memory
Nothing happens after this message appears
1. I have compiled my kernel several times and installed the modules. No errors were shown in the old terminal.
2. I used GParted to enlarge my

/dev/sda1
and I have set up a swap area, all done.
3. The stock Ubuntu 14.04 kernel boots fine. I can log in and use smoothly.
4. Commands I used:
sudo cp /boot/config-**** .config
sudo make menuconfig # I did not change anything here
sudo make -j4
sudo make modules_install
sudo make install
reboot
5. For the syscall, I just add a very simple helloworld in the sys.c
:
asmlinkage int sys_mysyscall(int arg){printk("hello %d\n",arg);return 0;}
and I have added it in the syscalls.h
and syscall_64.tbl
.
wjrforcyber
(371 rep)
Jan 5, 2019, 02:26 PM
• Last activity: Aug 2, 2023, 11:31 AM
0
votes
0
answers
542
views
linux kernel freeze
I am using linux kernel 4.19 on intel (icelake) based server. Sometime immediately after linux boot (at the login prompt) it get stuck(once is 50 reboot or so). It is not responding to anything on serial terminal or monitor. When it get stuck, keyboard caps lock light is not turning on and off. It i...
I am using linux kernel 4.19 on intel (icelake) based server. Sometime immediately after linux boot (at the login prompt) it get stuck(once is 50 reboot or so). It is not responding to anything on serial terminal or monitor. When it get stuck, keyboard caps lock light is not turning on and off. It is not taking even sysrq requests. Only way to recover is to reset the server.
I debugged, tried deadlock debugging, it is not helping. The issue occurs randomly after 10 or sometime 50 reboot. There is no backtrace or oops messages. In what type of situation this can happen? Is it possible that it deadlock in the interrupt context? if yes what is the best way to debug this? Or is may be some type of hardware issue? What else i can do to debug this further?
saurin
(39 rep)
Apr 12, 2022, 04:37 AM
• Last activity: Apr 12, 2022, 04:43 AM
2
votes
0
answers
4647
views
System is deadlocked on memory
I've upgraded my kernel on LinuxLite 4.8 from 4.15.0-74-generic to 5.4.30 (unpatched) using the commands: ``` sudo cp /boot/config-$(uname -r) .config sudo make menuconfig sudo make sudo make modules_install sudo make install reboot ``` I just saved the standard settings in menuconfig without changi...
I've upgraded my kernel on LinuxLite 4.8 from 4.15.0-74-generic to 5.4.30 (unpatched) using the commands:
sudo cp /boot/config-$(uname -r) .config
sudo make menuconfig
sudo make
sudo make modules_install
sudo make install
reboot
I just saved the standard settings in menuconfig without changing anything. After rebooting I faced the following error:
> end Kernel panic - not syncing: System is deadlocked on memory
Would anyone know how to solve this?
I already found https://unix.stackexchange.com/questions/492667/compiled-kernel-4-19-will-not-boot-kernel-panic-not-syncing-system-is-deadlo question, but I cannot afford to use 4 GB of RAM as the motherboard only supports a maximum of 2 GB (which is what I am using currently). I can still boot using the older kernel version.
Thanks in advance!
Daniel
(21 rep)
Apr 7, 2020, 01:44 PM
• Last activity: Apr 7, 2020, 01:50 PM
2
votes
2
answers
280
views
Bash: join pipes without deadlock
I want to list a bunch of filenames via `find`, pipe them through a utility (let's call this `util`) which outputs a new name for each input name, and then rename each file from the old name to the new. The most basic solution would be this: find . -print0 | while IFS= read -d '' -r old_name; do new...
I want to list a bunch of filenames via util does input/output buffering...
So my question is: what's the proper way of doing this in bash?
find
, pipe them through a utility (let's call this util
) which outputs a new name for each input name, and then rename each file from the old name to the new.
The most basic solution would be this:
find . -print0 | while IFS= read -d '' -r old_name; do
new_name="$(echo "$file" | util)"
mv "$old_name" "$new_name"
done
The problem with this approach is that util
is too slow to fire up for each filename separately. So the solution is to launch util
only once and pipe all the filenames through this single process:
find . -print0 >old_names
util new_names
exec {old_fd}new_names &
exec {old_fd}
Tamás Zahola
(143 rep)
Feb 2, 2019, 06:43 PM
• Last activity: Feb 3, 2019, 12:29 AM
0
votes
0
answers
57
views
How to view pending SCSI requests?
If the PC sends a request to read a sector on an optical disc, which turns out to be unreadable, the drive deadlocks the program into responselessness and motionlessness, until it returns the sector or an error. During the time the program is locked by the drive, which makes the program effectively...
If the PC sends a request to read a sector on an optical disc, which turns out to be unreadable, the drive deadlocks the program into responselessness and motionlessness, until it returns the sector or an error.
During the time the program is locked by the drive, which makes the program effectively a slave, how can I see the pending SCSI request with details such as read retry count and requested LBA?
neverMind9
(1720 rep)
Apr 4, 2018, 01:38 PM
• Last activity: Apr 4, 2018, 01:43 PM
4
votes
0
answers
1234
views
Why is Firefox stuck, and how might I get it unstuck?
Firefox is hung. Usually I kill it with -1, -9 or -15, but this time, dang it, I want to find out why. I have an hour or so to waste on this, and seek education above actually solving the problem. Goal: get this instance of firefox running again, without killing it and restarting. Firefox was workin...
Firefox is hung. Usually I kill it with -1, -9 or -15, but this time, dang it, I want to find out why. I have an hour or so to waste on this, and seek education above actually solving the problem.
Goal: get this instance of firefox running again, without killing it and restarting.
Firefox was working fine, though sluggish due to heavy memory usage. I had Avogadro's number of tabs open in about a thousand browser windows. (Some exaggeration may be involved.) I was closing some tabs, when at one point the browser window stopped repainting their contents. Dragging unrelated windows over causes remnants of those windows to sit there forever. Whatever is the X11 equivalent of WM_PAINT, it's being ignored. Attempts to fire up a new Firefox by clicking on URLs in emails bring up only an error popup saying "Firefox is already running, but is not responding."
Firefox has process ID 9297.
Process 526 is the windows manager.
=> ps axl | grep 9297
0 1000 9297 526 13 -7 6080428 4465776 poll_s S
(I was playing with renice to see if priority had anything to do with firefox hanging. Nope. But I left it at -7.)
=> ps -efL | grep 9297
darenw 9297 526 9297 57 61 Sep22 ? 6-23:10:34 firefox
darenw 9297 526 9300 0 61 Sep22 ? 00:00:00 firefox
darenw 9297 526 9301 0 61 Sep22 ? 00:00:00 firefox
... dozens like these ...
darenw 9297 526 7607 0 61 16:17 ? 00:00:00 firefox
darenw 9297 526 7657 0 61 16:17 ? 00:00:00 firefox
darenw 20000 9297 20000 1 1 Sep23 ? 03:34:36 [plugin-containe]
=> ps axl | grep 20000
0 1000 20000 9297 20 0 0 0 exit Z ? 216:31 [plugin-containe]
This one lingers because the main thread 9297 hasn't yet finished off that thread by obtaining its exit code. At least, that's my understanding of "defunct" processes. I'm not sure how to investigate this detail further, or how to determine if this is why Firefox is hung.
=> strace -f -p 9297
...
[pid 13945] ) = 0 (Timeout)
[pid 13945] select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
[pid 13945] select(0, NULL, NULL, NULL, {0, 10000}
[pid 9356] ) = -1 ETIMEDOUT (Connection timed out)
[pid 9356] futex(0x7f28b7df8be8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 9356] futex(0x7f28b7df8c14, FUTEX_WAIT_BITSET_PRIVATE, 1, {6910910, 806929252}, ffffffff
...
Same six lines repeat for as long as I let strace run, though sometimes there's only two instead of three lines for either PID.
I didn't see any mention of other pids, besides these two.
=> ps axl | grep 13945
4 0 4985 23253 20 0 11064 2236 pipe_w S+ pts/19 0:00 grep 13945
=> ps axl | grep 9356
4 0 5111 23253 20 0 11064 2232 pipe_w S+ pts/19 0:00 grep 9356
So what are these two processes, 13945 and 9356?
=> ps -eL | grep 9297 |grep 9356
9297 9356 ? 02:02:50 SoftwareVsyncTh
9297 13945 ? 01:04:15 firefox
So what is SoftwareVsyncTh? Google does not help much. This symbol appears in 'ps' output and other listing of processes and threads, but not in any online source code, or Q&A forums mentioning it in a specific way. For all I know, it, and these two processes, have nothing to do with Firefox being stuck and not painting its windoes.
What further commands could I use to uncover more clues? Is there a way to get a list of windows and tabs with their urls, and close one (the most suspicious/spammy looking url) from a shell command line?
Well, whatever is going on with those, I find I can get some info on the main thread:
=> strace -p 9297
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=9297, si_uid=1000} ---
rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
poll([{fd=23, events=POLLPRI|POLLOUT}], 1, 5000) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
which repeats over and over with no changes.
I am not a genius at threads, processes, mutexes and futexes and all that, but maybe by solving this mystery, I'll become one! I just need to know more commands for investigating further, and understanding of what exactly is going on in the strace results which may relate to the hangedness of Firefox.
Are there any command I could try, to kick Firefox back into action?
System specs: quad core Intel something, 16GB, Arch Linux last updated about a month or two ago. Using icewm, multiple workspaces and many text editors, PDF viewers, browsers and whatever open. Running conky to show RAM and swap usage in upper left corner of screen. I'm usually on the verge of going into swap.
DarenW
(3532 rep)
Oct 5, 2016, 08:37 AM
2
votes
0
answers
122
views
What does Linux wait for when switches from Xorg to text console?
Sometimes my Xorg hangs and locks the console dead, with a “stale frame” of GUI on the monitor. Hardware and driver-related problems are likely (see [here][1] for description of particular configuration), but this question is focused on a particularly annoying effect: inability to switch to text mod...
Sometimes my Xorg hangs and locks the console dead, with a “stale frame” of GUI on the monitor. Hardware and driver-related problems are likely (see here for description of particular configuration), but this question is focused on a particularly annoying effect: inability to switch to text mode. Otherwise the system is fully functioning. The box serves a LAN, I can log in and even run somethings like
chvt
inside, to no avail. Of course, I can terminate the Xorg process to free the console… but it’s so rude. Not the same situation as in that question , where Ctrl+Alt+F1 failed consistently, but, in my case, it fails *sometimes* and only when X is unable to work anyway.
So, the question: **Which exactly step of console switching is vulnerable to deadlocks?** Modern Linux kernels (3.x) are assumed. Many years ago (in the age of Linux 2.4) I had reasonably good understanding of virtual console-related stuff and even wrote some notes about it, but modern vc_screen.c
changed considerably from 10-years-old specimens, with such new calls as console_lock()
and console_unlock()
appeared in it.
Incnis Mrsi
(2076 rep)
Aug 25, 2015, 11:53 AM
• Last activity: Sep 9, 2015, 06:10 PM
1
votes
1
answers
1295
views
Deadlock on read/wait
My process deadlocks. `master` looks like this: p=Popen(cmd, stdin=PIPE, stdout=PIPE) for ....: # a few million p.stdin.write(...) p.stdin.close() out = p.stdout.read() p.stdout.close() exitcode = p.wait() `child` looks something like this: l = list() for line in sys.stdin: l.append(line) sys.stdout...
My process deadlocks.
master
looks like this:
p=Popen(cmd, stdin=PIPE, stdout=PIPE)
for ....: # a few million
p.stdin.write(...)
p.stdin.close()
out = p.stdout.read()
p.stdout.close()
exitcode = p.wait()
child
looks something like this:
l = list()
for line in sys.stdin:
l.append(line)
sys.stdout.write(str(len(l)))
* strace -p PID_master
shows that master
is stuck in wait4(PID_child,...)
.
* strate -p PID_child
shows that child
is stuck in read(0,...)
.
How can that be?!
**I did close
the stdin
, why is child
still reading from it?!**
sds
(1726 rep)
Aug 3, 2015, 01:12 PM
• Last activity: Aug 3, 2015, 03:41 PM
12
votes
2
answers
8970
views
Solving Ethernet watchdog timer deadlocks
I have a debian linux box (Debian Squeeze) that deadlocks every few hours if I run a python script that sniffs an interface... The stack trace is attached to the bottom of this question. Essentially, I have a Broadcom ethernet interface (`bnx2` driver) that seems to die when I start a sniffing sessi...
I have a debian linux box (Debian Squeeze) that deadlocks every few hours if I run a python script that sniffs an interface...
The stack trace is attached to the bottom of this question. Essentially, I have a Broadcom ethernet interface (
bnx2
driver) that seems to die when I start a sniffing session and then it tries to transmit a frame out the same interface.
From what I can tell, a kernel watchdog timer is tripping...
NETDEV WATCHDOG: eth3 (bnx2): transmit queue 0 timed out
I think there is a way to control watchdog timers with ioctl
(ref: [EmbeddedFreak: How to use linux watchdog](http://embeddedfreak.wordpress.com/2010/08/23/howto-use-linux-watchdog/)) .
Questions (Original):
---------------------
How can I find which watchdog timer(s) is controlling eth3? Bonus points if you can tell me how to change the timer or even disable the watchdog...
Questions (Revised):
---------------------
How can I prevent the ethernet watchdog timer from causing problems?
Stack trace
-----------
Apr 30 08:38:44 Hotcoffee kernel: [275460.837147] ------------[ cut here ]------------
Apr 30 08:38:44 Hotcoffee kernel: [275460.837166] WARNING: at /build/buildd-linux-2.6_2.6.32-41squeeze2-amd64-NDo8b7/linux-2.6-2.6.32/debian/build/source_amd64_none/net/sched/sch_generic.c:261 dev_watchdog+0xe2/0x194()
Apr 30 08:38:44 Hotcoffee kernel: [275460.837169] Hardware name: PowerEdge R710
Apr 30 08:38:44 Hotcoffee kernel: [275460.837171] NETDEV WATCHDOG: eth3 (bnx2): transmit queue 0 timed out
Apr 30 08:38:44 Hotcoffee kernel: [275460.837172] Modules linked in: 8021q garp stp parport_pc ppdev lp parport pci_stub vboxpci vboxnetadp vboxnetflt vboxdrv ext2 loop psmouse power_meter button dcdbas evdev pcspkr processor serio_raw ext4 mbcache jbd2 crc16 sg sr_mod cdrom ses ata_generic sd_mod usbhid hid crc_t10dif enclosure uhci_hcd ehci_hcd megaraid_sas ata_piix thermal libata usbcore nls_base scsi_mod bnx2 thermal_sys [last unloaded: scsi_wait_scan]
Apr 30 08:38:44 Hotcoffee kernel: [275460.837202] Pid: 0, comm: swapper Not tainted 2.6.32-5-amd64 #1
Apr 30 08:38:44 Hotcoffee kernel: [275460.837204] Call Trace:
Apr 30 08:38:44 Hotcoffee kernel: [275460.837206] [] ? dev_watchdog+0xe2/0x194
Apr 30 08:38:44 Hotcoffee kernel: [275460.837211] [] ? dev_watchdog+0xe2/0x194
Apr 30 08:38:44 Hotcoffee kernel: [275460.837217] [] ? warn_slowpath_common+0x77/0xa3
Apr 30 08:38:44 Hotcoffee kernel: [275460.837220] [] ? dev_watchdog+0x0/0x194
Apr 30 08:38:44 Hotcoffee kernel: [275460.837223] [] ? warn_slowpath_fmt+0x51/0x59
Apr 30 08:38:44 Hotcoffee kernel: [275460.837228] [] ? try_to_wake_up+0x289/0x29b
Apr 30 08:38:44 Hotcoffee kernel: [275460.837231] [] ? netif_tx_lock+0x3d/0x69
Apr 30 08:38:44 Hotcoffee kernel: [275460.837237] [] ? netdev_drivername+0x3b/0x40
Apr 30 08:38:44 Hotcoffee kernel: [275460.837240] [] ? dev_watchdog+0xe2/0x194
Apr 30 08:38:44 Hotcoffee kernel: [275460.837242] [] ? __wake_up+0x30/0x44
Apr 30 08:38:44 Hotcoffee kernel: [275460.837249] [] ? run_timer_softirq+0x1c9/0x268
Apr 30 08:38:44 Hotcoffee kernel: [275460.837252] [] ? __do_softirq+0xdd/0x1a6
Apr 30 08:38:44 Hotcoffee kernel: [275460.837257] [] ? lapic_next_event+0x18/0x1d
Apr 30 08:38:44 Hotcoffee kernel: [275460.837262] [] ? call_softirq+0x1c/0x30
Apr 30 08:38:44 Hotcoffee kernel: [275460.837265] [] ? do_softirq+0x3f/0x7c
Apr 30 08:38:44 Hotcoffee kernel: [275460.837267] [] ? irq_exit+0x36/0x76
Apr 30 08:38:44 Hotcoffee kernel: [275460.837270] [] ? smp_apic_timer_interrupt+0x87/0x95
Apr 30 08:38:44 Hotcoffee kernel: [275460.837273] [] ? apic_timer_interrupt+0x13/0x20
Apr 30 08:38:44 Hotcoffee kernel: [275460.837274] [] ? acpi_idle_enter_bm+0x27d/0x2af [processor]
Apr 30 08:38:44 Hotcoffee kernel: [275460.837283] [] ? acpi_idle_enter_bm+0x276/0x2af [processor]
Apr 30 08:38:44 Hotcoffee kernel: [275460.837289] [] ? cpuidle_idle_call+0x94/0xee
Apr 30 08:38:44 Hotcoffee kernel: [275460.837293] [] ? cpu_idle+0xa2/0xda
Apr 30 08:38:44 Hotcoffee kernel: [275460.837297] [] ? early_idt_handler+0x0/0x71
Apr 30 08:38:44 Hotcoffee kernel: [275460.837301] [] ? start_kernel+0x3dc/0x3e8
Apr 30 08:38:44 Hotcoffee kernel: [275460.837304] [] ? x86_64_start_kernel+0xf9/0x106
Apr 30 08:38:44 Hotcoffee kernel: [275460.837306] ---[ end trace 92c65e52c9e327ec ]---
Mike Pennington
(2521 rep)
May 1, 2012, 06:57 PM
• Last activity: May 4, 2015, 06:46 PM
1
votes
3
answers
6940
views
System freezes when browsing internet
About a week ago my system began to freeze (deadlock, only manual shutdown with laptop button helped) from time to time. The frequency of deadlock is increasing. Today it was about 10 deadlocks with only browsing the internet and terminal emulator and vim. It usually happens when loading new url and...
About a week ago my system began to freeze (deadlock, only manual shutdown with laptop button helped) from time to time.
The frequency of deadlock is increasing. Today it was about 10 deadlocks with only browsing the internet and terminal emulator and vim. It usually happens when loading new url and then all the system is dead. It's very annoying for me as it happens more often.
My system is
Lubuntu 11.10
with xmonad
window manager.
Could you help me to find the cause of this problem? Are there some logs associated with deadlocks? Could it be hardware problem?
(the system was dead twice while writing this question too)
thank you
xralf
(15189 rep)
Jan 6, 2012, 05:17 PM
• Last activity: Jan 28, 2012, 05:17 AM
Showing page 1 of 11 total questions