I'm aware of Linux softirqs **may** run within specific per-cpu kernel thread context -- ksoftirqd/[cpu_id]. Kernel threads are similar to user-space threads, however they only execute kernel code in kernel mode (they don't have user mode stacks). Suppose that at any given point in time, a ksoftirqd kernel thread is running a softirq (i.e. deferred work from a Top half interrupt handler). As far as I know, kernel threads share the virtual address range to which kernel code (including loaded kernel modules) is mapped to. What would happen if the softirq bottom half kernel code attempted to access a virtual address (VA) within the user space VA range ? I'm not sure an error would actually occur, but I believe the result of such an access would be unpredictable though. Does it make sense ? Thanks. BTW, I run the following trace-cmd command to dig into some details root@eve-ng62-28:~# trace-cmd start -e net:netif_receive_skb_entry root@eve-ng62-28:~# trace-cmd show # tracer: nop # # entries-in-buffer/entries-written: 2/2 #P:48 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | qemu-system-x86-2064265 b.... 599782.975041: netif_receive_skb_entry: dev=vunl0_2_0 napi_id=0x0 queue_mapping=1 skbaddr=00000000b36764b3 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0x0 qemu-system-x86-2064265 b.... 599783.973971: netif_receive_skb_entry: dev=vunl0_2_0 napi_id=0x0 queue_mapping=1 skbaddr=00000000dd8af289 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0x0 As you can check, netif_receive_skb_entry tracepoint within netif_receive_skb() kernel function/routine is triggered. It has the b value for the irqs-off/BH-disabled field. Does it mean netif_receive_skb() is actually running within a BH softirq with Bottom Half(BH) disabled ?

I recently noticed that one of my CPUs was idling at around 85-90% and according to top the usage was coming from interrupts, so just like in this question I used a combination of dmesg and periodically cating /proc/interrupts and found out this: CPU0 CPU1 CPU2 CPU3 0: 17 0 0 0 IR-IO-APIC 2-edge timer 1: 11548 0 2429 0 IR-IO-APIC 1-edge i8042 8: 0 0 0 1 IR-IO-APIC 8-edge rtc0 9: 7 16 0 0 IR-IO-APIC 9-fasteoi acpi 12: 14530 108887 0 0 IR-IO-APIC 12-edge i8042 16: 78464100 0 0 11702812 IR-IO-APIC 16-fasteoi idma64.0, i2c_designware.0, i801_smbus 120: 0 0 0 0 DMAR-MSI 0-edge dmar0 121: 0 0 0 0 DMAR-MSI 1-edge dmar1 As you can see, IRQ #16 is sending interrupts like crazy (every time the CPU wakes up from S3 it seems to start spamming a different CPU), I also found out that my touchpad uses the same IRQ and if the I2C mode is enabled (or *advanced* mode, according to my BIOS), it randomly stops working with the following messages (from dmesg): [ 167.851139] irq 16: nobody cared (try booting with the "irqpoll" option) [ 167.851158] CPU: 2 PID: 3874 Comm: firefox Not tainted 4.15.3-300.fc27.x86_64 #1 [ 167.851160] Hardware name: Acer Aspire E5-575/Ironman_SK , BIOS V1.04 04/26/2016 [ 167.851162] Call Trace: [ 167.851171] [ 167.851185] dump_stack+0x5c/0x85 [ 167.851193] __report_bad_irq+0x30/0xc0 [ 167.851196] note_interrupt+0x235/0x280 [ 167.851198] handle_irq_event_percpu+0x51/0x70 [ 167.851201] handle_irq_event+0x27/0x50 [ 167.851204] handle_fasteoi_irq+0x6b/0x120 [ 167.851209] handle_irq+0xaf/0x120 [ 167.851214] do_IRQ+0x41/0xc0 [ 167.851219] common_interrupt+0xa2/0xa2 [ 167.851222] [ 167.851224] RIP: 0010:_raw_spin_lock+0x10/0x20 [ 167.851226] RSP: 0000:ffffa85a857dfdd0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb [ 167.851230] RAX: 0000000000000000 RBX: ffff8d0a268930c8 RCX: 00003ffffffff000 [ 167.851231] RDX: 0000000000000001 RSI: 8000000000000025 RDI: ffffd21648d7ca70 [ 167.851232] RBP: ffffd2164892e100 R08: 0000000000000000 R09: 0000000000171800 [ 167.851233] R10: 0000000000271800 R11: 0000000000001000 R12: 0000000000000000 [ 167.851234] R13: 8000000224b84867 R14: ffffd21648d7ca70 R15: ffff8d0a35f29810 [ 167.851244] __handle_mm_fault+0xa4c/0x1290 [ 167.851249] handle_mm_fault+0xaa/0x1f0 [ 167.851255] __do_page_fault+0x25d/0x4e0 [ 167.851262] ? SyS_mmap_pgoff+0xfb/0x250 [ 167.851264] do_page_fault+0x32/0x110 [ 167.851267] ? page_fault+0x36/0x60 [ 167.851269] page_fault+0x4c/0x60 [ 167.851272] RIP: 0033:0x7ff86dc0b205 [ 167.851273] RSP: 002b:00007ffe6493e888 EFLAGS: 00010206 [ 167.851276] handlers: [ 167.851291] [] idma64_irq [idma64] [ 167.851296] [] i2c_dw_isr [ 167.851302] [] i801_isr [i2c_i801] [ 167.851304] Disabling IRQ #16 Is this a hardware issue? What can I do? ---------- Finally I have a chance to dig more into this, by running lspci -nnkv I found out 2 devices that are using IRQ 16: 00:15.0 Signal processing controller : Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 [8086:9d60] (rev 21) Subsystem: Acer Incorporated [ALI] Device [1025:1094] Flags: fast devsel, IRQ 16 Memory at a132b000 (64-bit, non-prefetchable) [size=4K] Capabilities: Power Management version 3 Capabilities: Vendor Specific Information: Len=14 Kernel driver in use: intel-lpss Kernel modules: intel_lpss_pci and: 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21) Subsystem: Acer Incorporated [ALI] Device [1025:1094] Flags: medium devsel, IRQ 16 Memory at a132e000 (64-bit, non-prefetchable) [size=256] I/O ports at 4040 [size=32] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 The problem seems to go away if I unload the intel_lpss_pci module i.e. rmmod intel_lpss_pci, but of course the touchpad would stop working. But I guess it's better than having a CPU always at 100%.

I have a problem with high load average (~2) on my (personal laptop) computer for a long time now. I am running Arch Linux. If I remember correctly, the problem started with a certain kernel update, initially I thought it was related to this bug . The problem was not solved though, when the bug was fixed. I did not really care as I thought it is still a bug, because the performance did not seem to suffer. What made me curious is that, recently, I had a moment of super low load average (~0) while idling. After a reboot, everything went back to "normal", with high load average. So I started investigating: % uptime 14:31:04 up 2:22, 1 user, load average: 1.96, 1.98, 1.99 So far nothing new. Then I tried top: % top -b -n 1 top - 14:33:52 up 2:25, 1 user, load average: 2.02, 2.07, 2.02 Tasks: 146 total, 2 running, 144 sleeping, 0 stopped, 0 zombie %Cpu0 : 2.6/0.9 3[|||| ] %Cpu1 : 2.7/0.9 4[|||| ] %Cpu2 : 2.7/1.0 4[|||| ] %Cpu3 : 2.7/0.8 3[|||| ] GiB Mem :228125107552256.0/7.712 [ GiB Swap: 0.0/7.904 [ ] PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND 2 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S kthreadd 404 root 20 0 0.0m 0.0m 0.0 0.0 0:01.09 D `- rtsx_usb_ms_2 1854 root 20 0 0.0m 0.0m 0.0 0.0 0:06.03 D `- kworker/0:2 I cut out all the processes and kernel threads except those two. Here we can see already some suspicious kernel threads (state D). And some suspicious Mem value (see edit)... Looking at CPU: % mpstat Linux 4.13.12-1-ARCH (arch) 30.11.2017 _x86_64_ (4 CPU) 14:36:09 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 14:36:09 all 2.66 0.00 0.88 1.56 0.00 0.01 0.00 0.00 0.00 94.90 % sar -u 1 30 Linux 4.13.12-1-ARCH (arch) 30.11.2017 _x86_64_ (4 CPU) 14:37:04 CPU %user %nice %system %iowait %steal %idle 14:37:05 all 1.00 0.00 0.75 0.00 0.00 98.25 14:37:06 all 1.76 0.00 0.50 0.00 0.00 97.74 14:37:07 all 1.00 0.00 0.25 0.00 0.00 98.75 14:37:08 all 0.50 0.00 0.50 0.00 0.00 99.00 14:37:09 all 0.50 0.00 0.50 0.25 0.00 98.75 14:37:10 all 0.50 0.00 0.50 6.03 0.00 92.96 14:37:11 all 0.75 0.00 0.50 11.75 0.00 87.00 14:37:12 all 0.50 0.00 0.25 0.00 0.00 99.25 [ . . . ] 14:37:21 all 1.26 0.00 0.76 0.00 0.00 97.98 14:37:22 all 0.75 0.00 0.25 2.26 0.00 96.73 14:37:23 all 0.50 0.00 0.50 16.83 0.00 82.16 14:37:24 all 0.75 0.00 0.50 0.00 0.00 98.74 14:37:25 all 0.50 0.00 0.50 0.00 0.00 98.99 14:37:26 all 0.76 0.00 0.50 7.56 0.00 91.18 14:37:27 all 0.25 0.00 0.51 0.00 0.00 99.24 14:37:28 all 1.00 0.00 0.75 0.25 0.00 98.00 14:37:29 all 0.25 0.00 0.76 0.00 0.00 98.99 14:37:30 all 0.75 0.00 0.50 0.00 0.00 98.74 14:37:31 all 0.75 0.00 0.50 3.27 0.00 95.48 14:37:32 all 0.51 0.00 0.51 13.16 0.00 85.82 14:37:33 all 0.75 0.00 0.50 0.25 0.00 98.49 14:37:34 all 1.26 0.00 0.75 0.00 0.00 97.99 Average: all 0.71 0.00 0.56 2.06 0.00 96.67 reveals some peaks in I/O wait. The best guess so far. Looking closer: % iostat -x 1 30 Linux 4.13.12-1-ARCH (arch) 30.11.2017 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 2.60 0.00 0.87 1.55 0.00 94.98 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.93 3.00 3.71 1.94 95.04 102.27 69.91 0.60 106.78 16.56 279.32 14.47 8.17 avg-cpu: %user %nice %system %iowait %steal %idle 0.75 0.00 0.75 0.25 0.00 98.25 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.01 13.00 0.00 13.00 10.00 1.00 [ . . . ] avg-cpu: %user %nice %system %iowait %steal %idle 0.50 0.00 0.50 17.04 0.00 81.95 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 8.00 0.00 2.00 0.00 40.00 40.00 0.69 346.50 0.00 346.50 346.50 69.30 [ . . . ] avg-cpu: %user %nice %system %iowait %steal %idle 0.25 0.00 0.50 7.29 0.00 91.96 [ . . . ] avg-cpu: %user %nice %system %iowait %steal %idle 1.00 0.00 0.75 16.96 0.00 81.30 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 5.00 0.00 2.00 0.00 28.00 28.00 0.71 357.00 0.00 357.00 356.50 71.30 [ . . . ] avg-cpu: %user %nice %system %iowait %steal %idle 0.50 0.00 0.50 0.00 0.00 99.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Looking at processes with uninterruptable sleep: % for x in seq 1 1 10; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done D 404 [rtsx_usb_ms_2] D 1854 [kworker/0:2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 1854 [kworker/0:2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 1854 [kworker/0:2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] ---- D 404 [rtsx_usb_ms_2] D 1854 [kworker/0:2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 1854 [kworker/0:2] D 2877 [kworker/0:0] ---- D 404 [rtsx_usb_ms_2] D 3177 [kworker/u32:4] ---- and last thing I did: % vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 1 0 5010040 123612 1220080 0 0 23 25 111 433 3 1 95 2 0 0 0 0 5006256 123612 1224164 0 0 0 96 186 839 1 1 97 1 0 1 0 0 5006132 123612 1224164 0 0 0 0 175 714 1 0 99 0 0 0 0 0 5003156 123612 1224156 0 0 0 0 234 1009 2 1 98 0 0 0 0 0 5003156 123612 1224156 0 0 0 0 161 680 0 0 99 0 0 0 1 0 5003156 123616 1224156 0 0 0 60 214 786 1 1 94 5 0 0 0 0 5003280 123620 1224156 0 0 0 4 226 776 1 0 88 11 0 1 0 0 5003156 123620 1224156 0 0 0 0 210 733 1 0 99 0 0 0 0 0 5005388 123620 1224156 0 0 0 0 159 747 1 0 99 0 0 0 0 0 5005388 123620 1224156 0 0 0 0 233 803 1 0 99 0 0 0 0 0 5005512 123620 1224156 0 0 0 0 152 670 1 0 99 0 0 0 0 0 5009664 123620 1220060 0 0 0 0 240 914 1 1 99 0 0 0 0 0 5009540 123620 1220060 0 0 0 0 237 833 1 1 99 0 0 0 0 0 5009664 123620 1220060 0 0 0 0 166 999 1 1 99 0 0 0 1 0 5009664 123620 1220060 0 0 0 4 168 700 1 0 88 11 0 0 0 0 5009540 123628 1220060 0 0 0 12 207 778 1 1 91 8 0 0 0 0 5009788 123628 1220064 0 0 0 0 189 717 0 1 99 0 0 0 0 0 5009664 123628 1220064 0 0 0 0 243 1453 1 1 98 0 0 0 0 0 5009044 123628 1220576 0 0 0 0 166 708 1 0 99 0 0 0 0 0 5009168 123628 1220576 0 0 0 0 146 663 1 0 99 0 0 0 0 0 5009540 123628 1220064 0 0 0 0 175 705 1 1 99 0 0 0 1 0 5009292 123632 1220128 0 0 0 8 223 908 1 0 99 0 0 ^C Now I still don't know what the problem is, but it looks like it comes from some peak I/O operations. There are some suspicious kernel threads. Any further ideas? What else could I do to investigate? **edit:** The Mem value seems strange, but it just occured very recently, a week ago or so, everything seemed to be normal. And % free total used free shared buff/cache available Mem: 8086240 1913860 4824764 133880 1347616 6231856 Swap: 8288252 0 8288252 seems to be fine though. **edit2:** First results of testing sar monitoring my system (very frequently, intervals of 1 second, but for a short duration, to get the peaks): Linux 4.13.12-1-ARCH (arch) 01.12.2017 _x86_64_ (4 CPU) 12:36:25 CPU %user %nice %system %iowait %steal %idle 12:36:26 all 0.50 0.00 0.50 0.00 0.00 99.00 12:36:27 all 0.50 0.00 0.50 0.25 0.00 98.74 12:36:28 all 0.50 0.00 0.75 0.00 0.00 98.75 12:36:29 all 0.50 0.00 0.25 7.52 0.00 91.73 12:36:30 all 0.25 0.00 0.75 9.77 0.00 89.22 12:36:31 all 0.25 0.00 0.75 0.00 0.00 98.99 12:36:32 all 1.00 0.00 0.50 0.25 0.00 98.25 12:36:33 all 1.00 0.00 1.00 0.00 0.00 98.00 12:36:34 all 0.25 0.00 0.25 0.25 0.00 99.24 12:36:35 all 0.50 0.25 0.75 33.25 0.00 65.25 12:36:36 all 0.50 0.00 0.75 0.25 0.00 98.50 12:36:37 all 0.75 0.00 0.25 0.00 0.00 99.00 12:36:38 all 0.25 0.00 0.50 0.00 0.00 99.24 12:36:39 all 0.50 0.00 0.50 0.00 0.00 99.00 12:36:40 all 0.50 0.25 0.50 10.75 0.00 88.00 Average: all 0.52 0.03 0.57 4.16 0.00 94.72 Network (-n) seems to be alright. Looking at devices (-d) reveals: Linux 4.13.12-1-ARCH (arch) 01.12.2017 _x86_64_ (4 CPU) 12:36:25 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:36:26 dev8-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:26 dev8-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 [ . . . ] 12:36:29 dev8-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:30 dev8-0 2.00 0.00 88.00 44.00 0.41 355.00 207.00 41.40 12:36:30 dev8-1 2.00 0.00 88.00 44.00 0.41 355.00 207.00 41.40 12:36:30 dev8-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:30 dev8-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:30 dev8-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:30 dev8-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:30 dev8-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:30 dev8-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:31 dev8-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:31 dev8-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 [ . . . ] 12:36:34 dev8-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:35 dev8-0 2.00 0.00 24.00 12.00 0.70 348.50 348.00 69.60 12:36:35 dev8-1 2.00 0.00 24.00 12.00 0.70 348.50 348.00 69.60 12:36:35 dev8-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:35 dev8-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:35 dev8-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:35 dev8-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:35 dev8-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:35 dev8-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:36 dev8-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:36 dev8-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 [ . . . ] 12:36:40 dev8-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-0 0.27 0.00 7.47 28.00 0.12 351.75 455.75 12.15 Average: dev8-1 0.27 0.00 7.47 28.00 0.12 351.75 455.75 12.15 Average: dev8-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 and -b gives: Linux 4.13.12-1-ARCH (arch) 01.12.2017 _x86_64_ (4 CPU) 12:36:25 tps rtps wtps bread/s bwrtn/s 12:36:26 0.00 0.00 0.00 0.00 0.00 12:36:27 0.00 0.00 0.00 0.00 0.00 12:36:28 0.00 0.00 0.00 0.00 0.00 12:36:29 0.00 0.00 0.00 0.00 0.00 12:36:30 2.00 0.00 2.00 0.00 88.00 12:36:31 0.00 0.00 0.00 0.00 0.00 12:36:32 0.00 0.00 0.00 0.00 0.00 12:36:33 0.00 0.00 0.00 0.00 0.00 12:36:34 0.00 0.00 0.00 0.00 0.00 12:36:35 2.00 0.00 2.00 0.00 24.00 12:36:36 0.00 0.00 0.00 0.00 0.00 12:36:37 0.00 0.00 0.00 0.00 0.00 12:36:38 0.00 0.00 0.00 0.00 0.00 12:36:39 0.00 0.00 0.00 0.00 0.00 12:36:40 0.00 0.00 0.00 0.00 0.00 Average: 0.27 0.00 0.27 0.00 7.47 So I assume the issue seems to be related to my hard drive (?). Because the I/O is on partition 1 (my root partition), it should be somewhere outside of /var which has an extra partition. The other partitions are data partitions and not system related. **edit3:** Even more data to that specific peak: paging looks fine (from my perspective with limited knowledge) Linux 4.13.12-1-ARCH (arch) 01.12.2017 _x86_64_ (4 CPU) 12:36:25 pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 12:36:26 0.00 0.00 0.00 0.00 2233.00 0.00 0.00 0.00 0.00 12:36:27 0.00 0.00 0.00 0.00 88.00 0.00 0.00 0.00 0.00 12:36:28 0.00 0.00 766.00 0.00 185.00 0.00 0.00 0.00 0.00 12:36:29 0.00 40.00 0.00 0.00 47.00 0.00 0.00 0.00 0.00 12:36:30 0.00 4.00 0.00 0.00 45.00 0.00 0.00 0.00 0.00 12:36:31 0.00 0.00 1.00 0.00 46.00 0.00 0.00 0.00 0.00 12:36:32 0.00 0.00 5.00 0.00 560.00 0.00 0.00 0.00 0.00 12:36:33 0.00 0.00 2.00 0.00 85.00 0.00 0.00 0.00 0.00 12:36:34 0.00 0.00 2.00 0.00 47.00 0.00 0.00 0.00 0.00 12:36:35 0.00 12.00 0.00 0.00 44.00 0.00 0.00 0.00 0.00 12:36:36 0.00 0.00 0.00 0.00 47.00 0.00 0.00 0.00 0.00 12:36:37 0.00 0.00 2.00 0.00 45.00 0.00 0.00 0.00 0.00 12:36:38 0.00 0.00 0.00 0.00 47.00 0.00 0.00 0.00 0.00 12:36:39 0.00 0.00 0.00 0.00 77.00 0.00 0.00 0.00 0.00 12:36:40 0.00 8.00 0.00 0.00 47.00 0.00 0.00 0.00 0.00 Average: 0.00 4.27 51.87 0.00 242.87 0.00 0.00 0.00 0.00 It looks like files were created during that peak (-v): Linux 4.13.12-1-ARCH (arch) 01.12.2017 _x86_64_ (4 CPU) 12:36:25 dentunusd file-nr inode-nr pty-nr 12:36:26 186520 4480 195468 2 [ . . . ] 12:36:34 186520 4480 195468 2 12:36:35 186520 4512 195468 2 [ . . . ] 12:36:40 186520 4512 195468 2 Average: 186520 4493 195468 2 **edit4:** It looks like some irq's are responsible. Running iotop -o -a (show only processes with i/o and accumulate them, so keep all processes that had i/o since the start of the program) resulted in: Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 7 be/4 root 0.00 B 0.00 B 0.00 % 99.99 % [ksoftirqd/0] 17 be/4 root 0.00 B 0.00 B 0.00 % 99.99 % [ksoftirqd/1] 23 be/4 root 0.00 B 0.00 B 0.00 % 99.99 % [ksoftirqd/2] 29 be/4 root 0.00 B 0.00 B 0.00 % 99.99 % [ksoftirqd/3] 292 rt/4 root 0.00 B 0.00 B 0.00 % 99.99 % [i915/signal:0] [ . . . ] So, is this a thing? How could I continue...?

On the IRQ affinity tuning subject. I've realized that some IRQs cannot be reassigned to other cores. I've read that those are called "managed IRQs". On the Linux Foundation wiki, I've also found a reference about SMI (System managed Interrupts), I had a suspect that those are the same, wish a confirmation on that if possible please. Another thing is: Supposing that they are the same, how can I identify an SMI? I have going through the Kernel docs and man pages, but I didn't find anything on how to interpret the procfs directory: /proc/irq/ which usually contains the files: - /proc/irq//affinity_hint - /proc/irq//effective_affinity - /proc/irq//effective_affinity_list - /proc/irq//node: According to : "The node file on an SMP system shows the node to which the device using the IRQ reports itself as being attached." Does it mean that this represents the physical socket from which CPU the IRQs are being served from? - /proc/irq//spurious I know that these two give me the IRQ affinity in a binary and CPU-list format respectively: - /proc/irq//smp_affinity - /proc/irq//smp_affinity_list And also there is the sysfs directory /sys/kernel/irq/ that seems to be used as source of information for the contents of /proc/interrupts, that usually has these files: - /sys/kernel/irq//actions - /sys/kernel/irq//chip_name - /sys/kernel/irq//hwirq - /sys/kernel/irq//name - /sys/kernel/irq//per_cpu_count - /sys/kernel/irq//type - /sys/kernel/irq//wakeup Despite, there is a man page for /proc/interrrups it doesn't tell much other than what is already obvious by looking at the file, and the sysfs neither the procfs directory for IRQs has a man page. So, Am I missing something? How can I effectively identify an SMI without having to manually try to reassign it and have an Input/Output error (like shown below)?

# echo '7-8' > /proc/irq/53/smp_affinity_list
bash: echo: write error: Input/output error

https://access.redhat.com/solutions/4819541 https://wiki.linuxfoundation.org/realtime/documentation/howto/debugging/smi-latency/smi https://docs.kernel.org/filesystems/proc.html https://docs.kernel.org/admin-guide/kernel-parameters.html#cpu-lists https://man7.org/linux/man-pages/man5/proc_interrupts.5.html

I am running a system with RT-Preempt kernel v6.4.8-rt8 CPU is 12th Gen Intel(R) Core(TM) i7-12800HE This system is receiving data through the CAN Bus. Every 1 millisecond a burst of 3 messages is received in the hardware CAN interface. Most of the time (99.99998%) of the time the messages are available on the socket to be received within 1ms of the previous group. But once per minute approximately, there is a delay of 4-6 ms in which the messages has been received in the hardware but is not available on the socket when calling recv. Things I have checked: - My application is calling deterministically recv every 1ms, the data is not there. - With candump I get the confirmation that the messages are received every 1 ms by checking the hardware timestamp. - The IRQ associated with the CAN device is bound to a CPU core (2) that is not used by my RT application (CPU core 6) ## The hack If I execute stress --cpu 1 and bind the stress CPU core to the same core as the IRQ the delays disappear. So I assume that a delay in waking up the CPU is causing the culprit. I have tried all of the following to try to keep the CPU awake without using stress. - performance /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor - performance /sys/devices/system/cpu/cpu2/cpufreq/energy_performance_preference - idle=poll intel_idle.max_cstate=0 in /etc/default/grub and update-grub; reboot Still have delays with all these applied. I am out of ideas, I cannot switch to acpi-cpufreq because it does not respect the integrated GPU power consumption (which is not being used during these tests). Any other knobs I can turn to decrease that latency or ways I can understand the root issue?

I have a 3-node proxmox cluster running at home with Dell Precision 3620 computers, all interconnected by:

05:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        Subsystem: Hewlett-Packard Company Ethernet 10G 2-port 546SFP+ Adapter

During the upgrade to the most recent version of proxmox, the Linux kernel was upgraded from 6.5.13-6-pve to 6.8.12-1-pve. The 6.8.12-1-pve was stuck after initializing the USB devices. The latest line of the boot is:

[    1.948124] hid-generic 0003:046A:0001.0002: input,hidraw1: USB HID v1.11 Keyboard [Cherry GmbH] on usb-0000:00:14.0-8/input0

It only continues booting when I add "nolapic" as a kernel command line parameter, which then of course, disables SMP and I can only use one CPU / core. Interestingly, the next step (which never comes to execution) would have been the initialization of the Mellanox NIC:

[    7.901026] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[    7.901402] mlx4_core 0000:05:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)

Please find the whole dmesg (of the "nolapic" boot) here: https://pastebin.com/raw/uJb15njQ I also tried to upgrade the BIOS (the version 2.8.1 is installed but 2.32.0 is most recent). Upgrading the BIOS makes the new kernel bootable without the nolapic command line parameter, but the Mellanox NIC unusable (independant of which kernel I use, all show the same error):

[    0.324965] pci 0000:05:00.0: [15b3:1007] type 00 class 0x020000 PCIe Endpoint
[    0.325432] pci 0000:05:00.0: BAR 0 [mem 0xef100000-0xef1fffff 64bit]
[    0.325748] pci 0000:05:00.0: BAR 2 [mem 0xe0000000-0xe1ffffff 64bit pref]
[    0.326284] pci 0000:05:00.0: ROM [mem 0xef000000-0xef0fffff pref]
[    0.328610] pci 0000:05:00.0: VF BAR 2 [mem 0x00000000-0x01ffffff 64bit pref]
[    0.328693] pci 0000:05:00.0: VF BAR 2 [mem 0x00000000-0x1fffffff 64bit pref]: contains BAR 2 for 16 VFs
[    0.331768] pci 0000:05:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
[    0.388844] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: can't assign; no space
[    0.388942] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: failed to assign
[    0.390239] pci 0000:05:00.0: BAR 2 [mem size 0x02000000 64bit pref]: can't assign; no space
[    0.390336] pci 0000:05:00.0: BAR 2 [mem size 0x02000000 64bit pref]: failed to assign
[    0.390432] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: can't assign; no space
[    0.390530] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: failed to assign
[    0.392703] pci 0000:05:00.0: BAR 2 [mem size 0x02000000 64bit pref]: can't assign; no space
[    0.392800] pci 0000:05:00.0: BAR 2 [mem size 0x02000000 64bit pref]: failed to assign
[    0.392897] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: can't assign; no space
[    0.392995] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: failed to assign
[    0.393092] pci 0000:05:00.0: BAR 0 [mem 0xe0100000-0xe01fffff 64bit]: assigned
[    0.393345] pci 0000:05:00.0: ROM [mem 0xe0200000-0xe02fffff pref]: assigned
[    0.393429] pci 0000:05:00.0: BAR 2 [mem size 0x02000000 64bit pref]: can't assign; no space
[    0.393526] pci 0000:05:00.0: BAR 2 [mem size 0x02000000 64bit pref]: failed to assign
[    0.394199] pci 0000:05:00.0: BAR 0 [mem 0xe0100000-0xe01fffff 64bit]: assigned
[    0.394447] pci 0000:05:00.0: ROM [mem 0xe0200000-0xe02fffff pref]: assigned
[    0.394530] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: can't assign; no space
[    0.394628] pci 0000:05:00.0: VF BAR 2 [mem size 0x20000000 64bit pref]: failed to assign
[    0.400419] pci 0000:05:00.0: Adding to iommu group 13
[    0.405601] DMAR: [DMA Read NO_PASID] Request device [05:00.0] fault addr 0xc4da3000 [fault reason 0x06] PTE Read access is not set
[    0.406462] DMAR: [DMA Read NO_PASID] Request device [05:00.0] fault addr 0xc4d21000 [fault reason 0x06] PTE Read access is not set
[    0.991623] mlx4_core: Initializing 0000:05:00.0
[    0.991831] mlx4_core 0000:05:00.0: Missing UAR, aborting

I spent hours and hours trying different kernel parameters (especially related to APIC / ACPI) first with the most recent BIOS to make the NIC usable again, then with the older BIOS to make the system boot without "nolapic" but without any success. For example (all after each other): - pci=nomsi - pci=noioapicquirk - pci=ioapicreroute - pci=noioapicreroute - pci=noacpi - pci=use_crs - pci=nocrs - pci=routeirq - pci=realloc=off - pci=noari - pci=noats - clocksource=hpet - irqpoll - noacpi - noapic - irqfixup - iommu=off The firmware on the Mellanox NIC is the most recent one. I'm thankful for any more ideas / comments. Thanks, Freddy

I am deeply wondering about this top (man page ) output in uptime 5 hours 30 minutes only:

top - 00:41:41 up  5:48,  1 user,  load average: 0.36, 0.44, 0.63
Tasks: 281 total,   1 running, 280 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31827.3 total,  15894.9 free,   1933.8 used,  13998.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  29321.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                     
   4567 vlastim+  20   0 5229056 292040 141472 S   0.5   0.9   9:40.38 cinnamon                                                                                                                                    
    278 root     -51   0       0      0      0 S   0.2   0.0   7:29.32 irq/140-nvidia                                                                                                                              
    746 root     -51   0       0      0      0 S   0.0   0.0   5:52.90 irq/142-iwlwifi                                                                                                                             
   1423 root      20   0   25.1g 181056 118612 S   0.3   0.6   5:48.33 Xorg                                                                                                                                        
    280 root      20   0       0      0      0 S   0.0   0.0   2:23.89 nv_queue                                                                                                                                    
    276 root      20   0       0      0      0 S   0.0   0.0   0:31.17 nvidia-modeset/                                                                                                                             
  17208 root      20   0       0      0      0 I   0.0   0.0   0:15.22 kworker/2:1-events                                                                                                                          
    391 root      20   0       0      0      0 S   0.1   0.0   0:10.72 jbd2/nvme0n1p2-                                                                                                                             
  21613 vlastim+  20   0   32.7g 418536 216740 S   0.1   1.3   0:09.15 brave

--- Laptop specs for question background - Using Linux Mint 21.1 Cinnamon with kernel 5.15.0-76-generic and Nvidia driver version 535.54.03. I am playing one specific HTML5 game in Google Chrome (stable), I have some decent specs of my laptop, it's getting old though, I know: - CPU : Intel Core i7-7700HQ @ 2.80GHz base freq. / 3.80 GHz turbo freq .; (4 cores, 8 threads) - RAM : 32 GB DDR4 2400MHz, 2 sticks in dual-channel , disabled swap file - GPU : NVIDIA , GeForce GTX 1060 , Max-Q Design , 6 GB GDDR5X VRAM - Display: Believe it or not, my laptop has a 15.6" UltraHD = 4K = 3840x2160 resolution, it was the cause for overheating of my laptop even if doing little tasks, so I have added a "_normal_" FullHD, and turned the built-in display off. No overheating now even when hours on the game. --- The actual question - Sad to say that I do not understand how IRQ works / what they are for. So it is impossible for me to understand why my computer spends so much time dealing with it. To clarify, from comment: Can they be avoided or limited to some extent? For example, I could switch to cable, which I do not have right now (standard Cat.6, I mean). Would that eliminate the hours of CPU time or would that cause just another IRQ to consume my CPU? --- Clues - nvidia-smi -

Mon Jul 17 01:41:49 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1060 ...    Off | 00000000:01:00.0  On |                  N/A |
| N/A   48C    P0              23W /  60W |    360MiB /  6144MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1423      G   /usr/lib/xorg/Xorg                          132MiB |
|    0   N/A  N/A      4567      G   cinnamon                                     57MiB |
|    0   N/A  N/A     21650      G   ...ble-features=BlockInsecureDownloads      167MiB |
+---------------------------------------------------------------------------------------+

--- wavemon -

┌─Interface───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│wlp60s0 (IEEE 802.11), phy 0, reg: n/a, SSID: =CENSORED                                                                                                                                                          │
├─Levels──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                                                                                                                                                 │
│link quality: 73%  (51/70)                                                                                                                                                                                       │
│========================================================================================================================================================                                                         │
│                                                                                                                                                                                                                 │
│                                                                                                                                                                                                                 │
│signal level: -59 dBm (1.26 nW)                                                                                                                                                                                  │
│===============================================================================================                                                                                                                  │
│                                                                                                                                                                                                                 │
├─Statistics──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│RX: 9e+06 (3.26 GiB), drop: 10846 (0.1%)                                                                                                                                                                         │
│TX: 2e+07 (1.74 GiB), retries: 9k (0.0%)                                                                                                                                                                         │
├─Info────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│mode: Managed, connected to: =====CENSORED====, time: 6:51h, inactive: 6.0s                                                                                                                                      │
│freq: 5745 MHz, ctr1: 5755 MHz, channel: 149 (width: 40 MHz)                                                                                                                                                     │
│rx rate: 360.0 Mbit/s VHT-MCS 8 40MHz short GI VHT-NSS 2, tx rate: 400.0 Mbit/s VHT-MCS 9 40MHz short GI VHT-NSS 2                                                                                               │
│beacons: 238339, avg sig: -57 dBm, interval: 0.1s, DTIM: 2                                                                                                                                                       │
│power mgt: off,  tx-power: 22 dBm (158.49 mW)                                                                                                                                                                    │
│retry: short limit 7,  rts/cts: off,  frag: off                                                                                                                                                                  │
├─Network─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│wlp60s0 (UP RUNNING BROADCAST MULTICAST)                                                                                                                                                                         │
│mac: =====CENSORED====, qlen: 1000                                                                                                                                                                               │
│ip: ==CENSORED==/24                                                                                                                                                                                              │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Is it possible, in Linux, to allow an user to change the priority of a process owned by root? More specifically an IRQ process? For an embedded real-time application I want to raise the priority of the GPIO IRQs because that led to better results. However these are owned by root. It would be nice if raising the priorities of the root processes would be possible without using root privileges.

According to what I've read so far, "when the kernel receives an interrupt, all the registered handlers are invoked." I understand that the registered handlers for each IRQ can be viewed via /proc/interrupts, and I also understand that the registered handlers come from the drivers that have invoked request_irq passing in a callback roughly of the form: irqreturn_t (*handler)(int, void *) Based on what I know, each of these interrupt handler callbacks associated with the particular IRQ should be invoked, and it is up to the handler to determine whether the interrupt should indeed be handled by it. If the handler should not handle the particular interrupt it must return the kernel macro IRQ_NONE. What I am having trouble understanding is, how each driver is expected to determine whether it should handle the interrupt or not. I suppose they can keep track internally if they're supposed to be expecting an interrupt. If so, I don't know how they'd be able to deal with the situation in which multiple drivers behind the same IRQ are expecting an interrupt. The reason I'm trying to understand these details is because I'm messing with the kexec mechanism to re-execute the kernel in the middle of system operation while playing with the reset pins and various registers on a PCIe bridge as well as a downstream PCI device. And in doing so, after a reboot I'm either getting kernel panics, or other drivers complaining that they're receiving interrupts even though no operation was taking place. How the handler decided that the interrupt should be handled by it is the mystery. Edit: In case it's relevant, the CPU architecture in question is x86.

Linux version info: uname -a Linux debian 5.10.0-22-amd64 #1 SMP Debian 5.10.178-3 (2023-04-22) x86_64 GNU/Linux The message always shown on my booting : sudo dmesg | grep "irq 7" [ 2.649135] irq 7: nobody cared (try booting with the "irqpoll" option) irq 7 info on my os: cat /proc/interrupts | head -n 5 CPU0 CPU1 CPU2 CPU3 0: 35 0 0 0 IO-APIC 2-edge timer 1: 0 4226 0 0 IO-APIC 1-edge i8042 7: 0 0 100000 0 IO-APIC 7-fasteoi pinctrl_amd 8: 0 0 1 0 IO-APIC 8-edge rtc0 My cpu info: lscpu | grep "Model name" Model name: AMD Athlon 3000G with Radeon Vega Graphics It can't solve the issue even to have written irqpoll in the grub.cfg: sudo grep 'irqpoll' /boot/grub/grub.cfg linux /boot/vmlinuz-5.10.0-22-amd64 root=UUID=55aaf5c1-acd2-4bb2-9536-f2900c96d6b2 ro single iommu=pt irqpoll

When installing a Linux OS (e.g. Mint) from a live cd or USB stick, editing /etc/default/grub does not solve the problem. You can add or edit a line like GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=nomsi" in /etc/default/grub, then running update-grub (which internally uses grub-mkconfig) will transfer that to /boot/grub/grub.cfg . But that becomes effective only after a reboot, and when you reboot a Linux live session (i.e. without having installed anything to hard disk yet), the live stick will reinstate its own Grub settings again, so you have the interrupt looping problem yet again. How to solve this?

I am running Debian bullseye on a (2nd hand) bare metal server, which crashes occasionally (happened 3 times with the course of 8 days now) and I can't seem to figure out why. I haven't found ways to reproduce it either, because the cause _seems_ to come from outside the system. On three occasions, the following happens: * The system is (practically) idle * There is a NETDEV WATCHDOG: enp0s31f6 (e1000e): transmit queue 0 timed out error with a stack trace in de kernel log, no preceding messages (the gap with the message prior is usually a few hours). * The message e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx keeps repeating every 10-20 seconds. * At this point, the network is down, I can no longer access it, so I issued a hardware reset on all occasions to get it back up and running again. Now, I did try the very first time to see if I could reset the network (through the console), (I didn't try to remove/reinsert the driver module though, not sure if that would have helped), but all'n all it didn't seem to be very fruitful endeavor so I decided to reboot and hope for the best. Can anyone help me with some kind of approach on how to debug the situation if it arises again, and maybe some pointers in how to reproduce the problem, and a way to get it running again without a hardware reset? # Log files (this is just the first time, the logs are equal or, at least, very similar for all 3 occasions)

Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.662109] ------------[ cut here ]------------
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.662249] NETDEV WATCHDOG: enp0s31f6 (e1000e): transmit queue 0 timed out
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.662401] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:467 dev_watchdog+0x260/0x270
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.662554] Modules linked in: dm_mod xt_nat vhost_net vhost vhost_iotlb tap tun xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_
ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common intel_pmc_core_pltdrv intel_pmc_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel evdev kvm irqbypass rapl intel_cstate intel_uncore wdat_wdt intel_pch_thermal
 watchdog ee1004 serio_raw ie31200_edac acpi_pad button drm fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid0 multip
ath linear raid1 md_mod crc32_pclmul crc32c_intel ahci xhci_pci ghash_clmulni_intel xhci_hcd libahci nvme e1000e libata aesni_intel usbcore libaes crypto_simd scsi_mod nvme_core ptp psmouse pps_core cryptd glue_helper t10_pi i2c_i801 crc_t10dif
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.663385]  crct10dif_generic i2c_smbus crct10dif_pclmul crct10dif_common wmi usb_common video
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.664310] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.10.0-21-amd64 #1 Debian 5.10.162-1
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.664461] Hardware name: FUJITSU /D3417-B2, BIOS V5.0.0.12 R1.27.0.SR.1 for D3417-B2x               06/10/2020
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.664630] RIP: 0010:dev_watchdog+0x260/0x270
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.664747] Code: eb a9 48 8b 1c 24 c6 05 c7 16 0d 01 01 48 89 df e8 b5 73 fa ff 44 89 e9 48 89 de 48 c7 c7 08 b8 b6 91 48 89 c2 e8 da a0 14 00  0b eb 86 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 41
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.664968] RSP: 0018:ffffbb7e40128eb0 EFLAGS: 00010282
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665088] RAX: 0000000000000000 RBX: ffff920c20740000 RCX: 000000000000083f
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665234] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665381] RBP: ffff920c207403dc R08: 0000000000000000 R09: ffffbb7e40128cd0
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665532] R10: ffffbb7e40128cc8 R11: ffffffff920cb6a8 R12: ffff920b4143c080
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665681] R13: 0000000000000000 R14: ffff920c20740480 R15: 0000000000000001
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665832] FS:  0000000000000000(0000) GS:ffff921a2e440000(0000) knlGS:0000000000000000
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.665985] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666101] CR2: 000000c0002f9000 CR3: 0000000c9480a001 CR4: 00000000003726e0
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666537] Call Trace:
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666646]  
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666754]  ? pfifo_fast_enqueue+0x150/0x150
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666868]  call_timer_fn+0x27/0x100
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.666988]  __run_timers.part.0+0x1d9/0x250
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667106]  ? ktime_get+0x35/0xa0
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667223]  ? lapic_next_deadline+0x28/0x40
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667340]  ? clockevents_program_event+0x8a/0xf0
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667462]  run_timer_softirq+0x26/0x50
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667536]  __do_softirq+0xc2/0x279
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667610]  asm_call_irq_on_stack+0xf/0x20
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667684]  
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667755]  do_softirq_own_stack+0x37/0x50
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667830]  irq_exit_rcu+0x92/0xc0
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667904]  sysvec_apic_timer_interrupt+0x36/0x80
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.667980]  asm_sysvec_apic_timer_interrupt+0x12/0x20
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668057] RIP: 0010:cpuidle_enter_state+0xc7/0x350
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668133] Code: 8b 3d dd 71 f4 6e e8 b8 9a 9f ff 49 89 c5 0f 1f 44 00 00 31 ff e8 29 a6 9f ff 45 84 ff 0f 85 fe 00 00 00 fb 66 0f 1f 44 00 00  85 f6 0f 88 0a 01 00 00 49 63 c6 4c 2b 2c 24 48 8d 14 40 48 8d
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668256] RSP: 0018:ffffbb7e400c3ea8 EFLAGS: 00000246
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668334] RAX: ffff921a2e473c40 RBX: 0000000000000006 RCX: 000000000000001f
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668425] RDX: 0000000000000000 RSI: 0000000021c15a3d RDI: 0000000000000000
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668517] RBP: ffff921a2e47e800 R08: 00007429fb821b6a R09: 0000000000000001
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668608] R10: 0000000000000000 R11: 0000000000002b55 R12: ffffffff921aea80
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668700] R13: 00007429fb821b6a R14: 0000000000000006 R15: 0000000000000000
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668792]  ? cpuidle_enter_state+0xb7/0x350
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668867]  cpuidle_enter+0x29/0x40
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.668941]  do_idle+0x1f3/0x2b0
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.669015]  cpu_startup_entry+0x19/0x20
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.669089]  secondary_startup_64_no_verify+0xb0/0xbb
Feb 21 06:34:54 Debian-1106-bullseye-amd64-base kernel: [127723.669166] ---[ end trace 4e1f5ac6215c3384 ]---

# Hardware info

# lspci -vvvv -s 0000:00:1f.6
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM (rev 31)
	Subsystem: Fujitsu Technology Solutions Ethernet Connection (2) I219-LM
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- 
Installed-Size: 318 MB
Depends: kmod, linux-base (>= 4.3~), initramfs-tools (>= 0.120+deb8u2) | linux-initramfs-tool
Recommends: firmware-linux-free, apparmor
Suggests: linux-doc-5.10, debian-kernel-handbook, grub-pc | grub-efi-amd64 | extlinux
Conflicts: linux-image-5.10.0-21-amd64-unsigned
Breaks: fwupdate (<< 12-7), initramfs-tools (<< 0.120+deb8u2), wireless-regdb (<< 2019.06.03-1~), xserver-xorg-input-vmmouse (<< 1:13.0.99)
Replaces: linux-image-5.10.0-21-amd64-unsigned
Homepage: https://www.kernel.org/ 
Download-Size: 55.5 MB
APT-Manual-Installed: no
APT-Sources: http://security.debian.org/debian-security  bullseye-security/main amd64 Packages
Description: Linux 5.10 for 64-bit PCs (signed)
 The Linux kernel 5.10 and modules for use on PCs with AMD64, Intel 64 or
 VIA Nano processors.
 .
 The kernel image and modules are signed for use with Secure Boot.

I am trying to set irq affinity on linux by changing the value in smp_affinity. When I echo the new value into the file, I don't get any error but when I read it back, the value remains unchanged. I don't have irqbalance enabled, so I am not sure what else could be preventing me from changing it. For example: > cat /proc/irq/51/smp_affinity f > echo 1 > /proc/irq/51/smp_affinity > cat /proc/irq/51/smp_affinity f

Today I've read this opinion however I don't understand the topic of interrupts at all, so it would be nice if knowledgeable people chimed in and explained the rationale behind using this daemon in the past and whether it's advisable or not to use it nowadays.

I don't have knowledge about writing Linux kernel modules / drivers. Let's take a basic example. I have an input device of my own which is connected to a microcontroller on one of the interrupt pins. I know that when I press a button on my device it causes a hardware interrupt to occur in the microntroller. As a bare metal developer, I would put my interrupt handler code at an address where the interrupt vector for that particular interrupt would jump to when it occurs. Now if we have a Linux Kernel running on the microcontroller and I wish to write a Kernel driver for my input device, how would I know exactly where to register my interrupt handlers via the Kernel? Would I still need to know everything about the hardware, addresses etc. ? How do I know which interrupt line in the kernel is associated with the exact pin which I connect my input device to?

I try to figure out the values of interrupt jitter in Linux, especially in the worst cases. Two testbeds are considered, one is Raspberry Pi 4B, the other is a high-end PC with an intel i9 CPU and ASUS motherboard.

+-+ +-+ +-+     Raspberry Pi 4B
   | | | | | |   -------------+
 --+ +-+ +-+ +-   1. GPIO     |              +--------------+
Signal Generator              |              |              |
                              +----> +-----+ |    Linux     |
                                     | CPU | | ------------ |
                              +----> +-----+ |    Kernel    |
       |  |                   |              |              |
     +-+--+-+     2. PCIe MSI |              +--------------+
   --+ FPGA +--   ------------+
     +-+--+-+      high-end PC (i9-9900K)
       |  |

For the Raspberry Pi, a signal generator, which outputs fixed-frequency square waves, is used as the interrupt source. As for the high-end PC, a Xilinx FPGA board with PCIe is used as the interrupt source, and the interrupt type is MSI. Both the signal generator and the Xilinx FPGA board can generate interrupts with little time jitters (spinlock); nwritten = gpio_fifo_write(devinfo->fifo, ×tamp, 1); spin_unlock(&devinfo->spinlock); if (nwritten != 1) { printk(KERN_ERR "GPIOTS: ISR fifo overflow\n"); } // wake up the waiting thread which then read timestamp from the fifo wake_up(&devinfo->waitqueue); return IRQ_HANDLED; } The following is the irq_handler function of the driver code in the high-end PC // IRQ handler static irqreturn_t fPCI_event_isr (int irq, void *cookie) { struct timespec64 tsc_now = {.tsc_high = 0; .tsc_low = 0}; struct fPCI_dev *drv_data = (struct fPCI_dev *) cookie; int nwritten; int fifo_size; // get TSC timestamp get_tsc_cpuid((tsc_now.tsc_high), (tsc_now.tsc_low)); spin_lock(&(drv_data->spin_lock)); nwritten = fpci_fifo_write(drv_data->fifo, &tsc_now, 1); spin_unlock(&(drv_data->spin_lock)); if (nwritten != 1) printk(KERN_INFO, "fPCI_event_isr: fifo overflow!\n"); fifo_size = fpci_fifo_size(drv_data->fifo); if (fifo_size > FIFO_SIZE/2) wake_up(&(drv_data->waitqueue)); return IRQ_HANDLED; } In the code of IRQ handlers, first, the current time is stored in a FIFO. Then, a user-space thread reads time data from the FIFO, and counts the jitters of the interrupt. The results are showed in following two figures. My Question are: 1 Why the PCIe MSI interrupts have larger latency jitters than the GPIO interrupt's ? 2 Is it normal that the morden PCIe MSI interrupt have about 100us latency variation under the worst condition (Is there anything I missed)? How can I minimize such variations in latency (jitter).

I installed a brand new xr17v358 serial card, which claims 25Mbps maximum. (For some reason this was the only card I could find that didn't top out at 1950s speeds. I'll settle for 1970s speeds in this application). How fast should I expect it to run in Linux? I successfully ran a 2.7Mbps loopback test on one port. Yes, I used Linux's silly "custom baud" code and double-checked the speed with an oscilloscope. I can also run two ports simultaneously at 1.0Mbps each. However, when I try having 2 ports at 2.0Mbps simultaneously, I get: serial8250: too much work for irq103 And then all serial ports stop working altogether. Is this a bug/misconfiguration of the xr17v358 driver, or a bug/misconfiguration of Linux itself? Others have researched the source of the above error message [and found](https://marc.info/?l=linux-arm-kernel&m=111334035324612) that this is either a hardware bug or the data is coming in too fast for the CPU. Since this computer has a fully working Gigabit Ethernet card on it (not to mention a 5GHz Skylake processor and 400Gb/s of memory bandwith) can I assume that the CPU will not have any problems with a 25Mbps trickle of serial data?

Whenever i start my pc. I see this error "Disabling IRQ #9". I only have manjaro on my pc (no dual boot) and wanted to know the cause behind this problem. Sudo systemctl returns no error but when i run journalctl i get this output knowing that i completely wiped my hard disk and reset my bios settings to the defaults before installing manjaro and i got the error the first time i started the fresh install. Output of "sudo journalctl -p 3 -xb"

-- Logs begin at Sun 2020-11-15 12:28:58 CET, end at Sun 2020-11-15 12:45:20 CET. --
Nov 15 12:31:30 aminbh kernel: x86/cpu: VMX (outside TXT) disabled by BIOS
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PCI0.XHC.RHUB.TPLD], AE_ALREADY_EXISTS (20200528/dswload2-326)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS01], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS02], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS03], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS04], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS05], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS06], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS07], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS08], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.HS09], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PCI0.XHC.RHUB.HS10._UPC], AE_ALREADY_EXISTS (20200528/dswload2-326)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.USR1], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.USR2], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.SS01], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.SS02], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.SS03], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.SS04], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.SS05], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.XHC.RHUB.SS06], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.I2C2.TPD0], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.I2C3.TPL1], AE_NOT_FOUND (20200528/dswload2-162)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PCI0.PEG0.PEGP._ON], AE_ALREADY_EXISTS (20200528/dswload2-326)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:30 aminbh kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PCI0.PEG0.PEGP._OFF], AE_ALREADY_EXISTS (20200528/dswload2-326)
Nov 15 12:31:30 aminbh kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20200528/psobject-220)
Nov 15 12:31:37 aminbh kernel: irq 9: nobody cared (try booting with the "irqpoll" option)
Nov 15 12:31:37 aminbh kernel: handlers:
Nov 15 12:31:37 aminbh kernel: [] acpi_irq
Nov 15 12:31:37 aminbh kernel: Disabling IRQ #9
Nov 15 12:31:44 aminbh wpa_supplicant: nl80211: kernel reports: Attribute failed policy validation
Nov 15 12:31:44 aminbh wpa_supplicant: Failed to create interface p2p-dev-wlp0s20f3: -22 (Invalid argument)
Nov 15 12:31:44 aminbh wpa_supplicant: nl80211: Failed to create a P2P Device interface p2p-dev-wlp0s20f3
Nov 15 12:31:46 aminbh kernel: nouveau 0000:01:00.0: gr: intr 00000040
Nov 15 12:32:52 aminbh bluetoothd: Failed to set mode: Blocked through rfkill (0x12)

I have a ARM64 board running Ubuntu 18. The board runs a Python script that does the following: - acquire data from a USB camera and a USB microphone - process the data and send the result to an external controller via serial line (ttyS4), with a certain frequency (~10 Hz). According to dmesg, ttyS4 uses the interrupt mode as it fails to request DMA - receive data from the same serial line - stream and receive some information by sending and receiving UDP packets to/from a specific IP address The acquisition, processing and transmission run in parallel on different processes. I want to make sure that there are no conflicts, in particular between the USB acquisition and the transmission on ttyS4. However I have no experience at this level and I don't know what tools should I use to monitor IRQs, DMA etc. and to fix my script accordingly. What are the best practices to follow in this case?

I can not kill irq/${nnn}-nvidia by kill -9 or pkill -f -9. Does anyone how to kill or stop those process?  (I am using Ubuntu 16.04, if that is relevant.)