Are there any opensource/free Unix like RTOS that can run a virtual machine like QNX Neutrino RTOS (http://www.qnx.com/products/evaluation/eval-target.html) ?

I have an application with several CPU intensive threads set to the SCHED_RR scheduling policy. I recently starting running this on a system with a hybrid core Intel CPU, and noticed that these threads are always getting scheduled on E cores, which is creating a performance bottleneck. After doing some experiments, it seems that all threads with a real time scheduling policy end up getting scheduled on E cores unless they are explicitly set to a P core by setting the affinity. With SCHED_NORMAL threads, I see the behavior I would expect, with threads getting dynamically scheduled across P and E cores and CPU intensive tasks generally ending up on P cores. I can reproduce this behavior with the stress-ng utility. With this command, I see the CPU stress threads are scheduled on P cores: stress-ng --cpu 4 --timeout 60s However, if I switch this same test to a real time scheduling policy (RR or FIFO), the threads are always scheduled on E cores and perform worse: stress-ng --sched rr --cpu 4 --timeout 60s What could cause this behavior? Could I be missing something in my kernel configuration? Or is it expected that the scheduler prefers E cores for real time threads? As a workaround I could just force threads on to P cores, but I'd prefer to use dynamic scheduling if possible to take advantage of both core types. Kernel version: 5.14.0-427.76.1.el9_4.x86_64+rt CPU: Intel Core i9-12900

I am using a PC to control a robot at 1000Hz. My machine has Ubuntu 22.04 LTS with Linux 6.7 patched with PREEMPT_RT. The machine communicates with the robot via UDP trough a point-to-point Ethernet connection (full duplex, 1000Mbps). The Ethernet cable is CAT6 (as per the robot manual) and my machine's NIC is an Intel I219-V running the e1000e driver . The control loop is a synchronous read-execute-write-sleep loop, which means that each iteration starts with a blocking read from the UDP socket and ends with a write operation on the socket. The control thread is run with an high priority of 98. As the operating frequency is 1000Hz, each iteration should start 1ms after the start of the previous one. However, I am exeperiencing latency issues while running this application, as sometimes the read operation lasts 3ms to 4ms. I captured a packet trace by using Wireshark during the control loop execution. In the following trace, the IP of the local machine is 192.168.1.2 whereas the IP of the robot is 192.168.1.1. This is the result I got: As you can see after packet #11322 is received at 10.935, the next packet #11325 is received at 10.939 (after 4ms) together with packets #11328, #11330, #11332. I also used LTTng to trace both the Linux kernel and the user-space application, and it seems that when the problem occurs, no hardware interrupt is fired by the NIC during that time delay. Therefore I suspect a problem with the network driver or the NIC firmware. In this documentation page for the e1000e driver I found out that the InterruptThrottleRate parameter can be used to configure IRQ coalescing , however setting it to 0 did not solve the issue. I wonder wether I should try with a different NIC, but I don't have a criteria to select a proper one as I still haven't figured out the exact cause of the issue.

I have a question regarding "Realtime" support in Linux/Ubuntu: After some reasearch onthe topic, it seems that "realtime" support in Linux is twofold: On the one hand, there is the [**PREEMPT_RT**](https://en.wikipedia.org/wiki/PREEMPT_RT) patch, which brings *"real-time computing"* capabilities to the Linux kernel; on the other hand, there are the so-called **"realtime" [scheduling strategies](https://man7.org/linux/man-pages/man7/sched.7.html)** , such as SCHED_FIFO, SCHED_RR and SCHED_DEADLINE, which have been available for a long time. But how does it all come together? What happens if I use one of the *"realtime" scheduling strategies* (e.g. SCHED_DEADLINE) on a "vanilla" kernel, i.e. one that does **not** have PREEMPT_RT enabled? One would assume that this is **not** even possible. But it seems that it ***is*** possible, according to my test on a "vanilla" (non-realtime) kernel! What does it mean? Do the "realtime" scheduling strategies still work with the "non-realtime" kernel, just not "as good"? Or does it mean that the *"realtime" scheduling strategies* silently are the same as SHED_OTHER when running on a "non-realtime" kernel? Furthermore, assuming that I'm using a "realtime" kernel, i.e. one that *does* have PREEMPT_RT enabled: Will there be ***any*** difference at all, compared to a "vanilla" (non-realtime) kernel, if I do **not** explicitely start any processes with one of the "realtime" scheduling strategies? (Bonus question: How does the "niceness" play together with "realtime" scheduling? I have been looking for information, but all sources explain "niceness" *only* in the context of SHED_OTHER)

I have a real time user level software stack running on ubuntu RT kernel. Linux ran416 5.15.0-1061-realtime #69-Ubuntu SMP PREEMPT_RT Mon Apr 15 18:56:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux I also configured CPU isolation and use set core affinity APIs to make some critical threads run on isolated CPU. I ensured none other applications / threads are running on those isolated kernel (be it set core affinity or taskset) So these eliminate any user application / threads from accessing my isolated core. I ensured I am not doing any I/O, network operation or locking of any sort in the critical threads. Still I see that kernel threads are sometime accessing the isolated cpu core in question. In such situation when my thread do not yield to kernel thread it causes system freeze. And in other cases it causes my critical thread to miss events that have catastrophic consequences on my stack. I tried playing with /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us as suggested by ubuntu support, but it does not seem to help or sometime create further instability. Ubuntu support also indicated that "CPU pinning does not isolate kernel thread. Only other user-space applications are guaranteed not to run on the isolated cores." So my question is there any other flavor of linux that can give me this guarantee ? If not, what is the ideal approach to work with this limitation, given the fact that I want few threads of my user space application to be absolutely have no contention for CPU

From my understanding a real-time process means the process will get the CPU as soon as it needs it. The kernel is responsible for this task with its internal scheduling mechanism. On the other hand, Process affinity means the process will get a dedicated CPU to itself and it won't be managed by kernel scheduler. So, if I understand correctly, it's even better than being a real-time process. It's like programming for an Arduino but only with a much faster CPU. Also, to my understanding for process affinity we need at least a dual core CPU, and we must leave at least one CPU for the OS itself. So, with the help of process affinity a program that was written for an Arduino can be ported to a Linux board like Raspberry Pi Zero 2 W (which has a quad-core CPU) and would have all the benefits of a microcontroller only with a faster CPU. Is my understanding correct?

I’m working with an STM32MP157D-DK1, trying to use the hwlat tracer for the board's arm Cortex-A7 CPU, to check for typical hardware latency. The following attempt was made to use the hwlat tracer on the board:

# mount -t debugfs debugfs /sys/kernel/debug
# mount -t tracefs nodev /sys/kernel/tracing
# cd /sys/kernel/tracing
# cat hwlat_detector/window
1000000
# cat hwlat_detector/width
900000
# cat tracing_thresh
0
# cat tracing_max_latency
0
# cat tracing_cpumask
3
# cat hwlat_detector/mode
none [round-robin] per-cpu
# cat tracing_on
0
# cat current_tracer
nop
# echo hwlat > current_tracer

Assigning # echo hwlat > current_tracer would set tracing_thresh to 10, so:

# echo 0 > tracing_thresh
# cat tracing_thresh
0
# echo 1 > tracing_on
# sleep 1800
# cat trace


# tracer: hwlat
#
# entries-in-buffer/entries-written: 46/46   #P:2
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
           -159      d....   240.855853: #1     inner/outer(us):    0/2     ts:946699046.769930071 count:2
           -159      d....   241.871956: #2     inner/outer(us):    0/1     ts:946699047.786034280 count:2
           -159      d....   242.891945: #3     inner/outer(us):    0/1     ts:946699048.806022489 count:2
           -159      d....   243.911906: #4     inner/outer(us):    0/1     ts:946699049.825984865 count:2
           -159      d....   246.971967: #5     inner/outer(us):    0/1     ts:946699052.886044616 count:2
           -159      d....   252.071924: #6     inner/outer(us):    0/1     ts:946699057.986000660 count:2
           -159      d....   257.171915: #7     inner/outer(us):    0/1     ts:946699063.085992329 count:1
           -159      d....   272.471929: #8     inner/outer(us):    0/1     ts:946699078.386005920 count:2
           -159      d....   273.491923: #9     inner/outer(us):    0/1     ts:946699079.406001504 count:2
           -159      d....   277.571906: #10    inner/outer(us):    0/1     ts:946699083.485984589 count:2
           -159      d....   278.591918: #11    inner/outer(us):    0/1     ts:946699084.505996048 count:2
           -159      d....   318.371905: #12    inner/outer(us):    0/1     ts:946699124.285982817 count:2
           -159      d....   346.931903: #13    inner/outer(us):    0/1     ts:946699152.845980372 count:2
           -159      d....   393.851943: #14    inner/outer(us):    0/1     ts:946699199.766020728 count:2
           -159      d....   420.371906: #15    inner/outer(us):    0/1     ts:946699226.285984865 count:1
           -159      d....   487.691904: #16    inner/outer(us):    0/1     ts:946699293.605983314 count:2
           -159      d....   614.171939: #17    inner/outer(us):    0/1     ts:946699420.086017333 count:2
           -159      d....   646.811933: #18    inner/outer(us):    0/1     ts:946699452.726012015 count:1
           -159      d....   650.891904: #19    inner/outer(us):    0/1     ts:946699456.805984600 count:2
           -159      d....   705.971922: #20    inner/outer(us):    0/1     ts:946699511.886000877 count:2
           -159      d....   708.011907: #21    inner/outer(us):    0/1     ts:946699513.925987669 count:1
           -159      d....   763.091907: #22    inner/outer(us):    0/1     ts:946699569.005986862 count:2
           -159      d....   824.291904: #23    inner/outer(us):    0/1     ts:946699630.205985225 count:2
           -159      d....   893.651899: #24    inner/outer(us):    0/1     ts:946699699.565979716 count:2
           -159      d....   895.691904: #25    inner/outer(us):    0/1     ts:946699701.605985717 count:2
           -159      d....   912.011931: #26    inner/outer(us):    0/1     ts:946699717.926011266 count:2
           -159      d....   913.031903: #27    inner/outer(us):    0/1     ts:946699718.945983725 count:2
           -159      d....   930.371903: #28    inner/outer(us):    0/1     ts:946699736.285984275 count:2
           -159      d....   934.451904: #29    inner/outer(us):    0/1     ts:946699740.365986111 count:2
           -159      d....  1034.411903: #30    inner/outer(us):    0/1     ts:946699840.325984200 count:2
           -159      d....  1049.711894: #31    inner/outer(us):    0/1     ts:946699855.625976207 count:1
           -159      d....  1211.891897: #32    inner/outer(us):    0/1     ts:946700017.805979076 count:2
           -159      d....  1222.091906: #33    inner/outer(us):    0/1     ts:946700028.005987831 count:2
           -159      d....  1336.331903: #34    inner/outer(us):    0/1     ts:946700142.245984969 count:2
           -159      d....  1338.371900: #35    inner/outer(us):    0/1     ts:946700144.285983803 count:2
           -159      d....  1356.731931: #36    inner/outer(us):    0/1     ts:946700162.646014437 count:2
           -159      d....  1477.091900: #37    inner/outer(us):    0/1     ts:946700283.005983161 count:2
           -159      d....  1514.831892: #38    inner/outer(us):    0/1     ts:946700320.745975929 count:2
           -159      d....  1530.131903: #39    inner/outer(us):    0/1     ts:946700336.045985853 count:2
           -159      d....  1533.191895: #40    inner/outer(us):    0/1     ts:946700339.105978979 count:1
           -159      d....  1534.211901: #41    inner/outer(us):    0/1     ts:946700340.125985563 count:2
           -159      d....  1571.951896: #42    inner/outer(us):    0/1     ts:946700377.865979289 count:2
           -159      d....  1639.271892: #43    inner/outer(us):    0/1     ts:946700445.185976197 count:2
           -159      d....  1664.771895: #44    inner/outer(us):    0/1     ts:946700470.685979917 count:2
           -159      d....  1679.051899: #45    inner/outer(us):    0/1     ts:946700484.965983799 count:1
           -159      d....  1768.811900: #46    inner/outer(us):    0/1     ts:946700574.725985300 count:2

After a half an hour sleep, this is what is logged to file trace. I'm trying to understand this result. Trying to take assistance in Steven Rosetedt's walkthrough of the hwlat tracer output . In another 20 minutes test I ran with similar conditions, there also appeared a line where numerous encounters of a 3 millisecond latency occurred, rather than the frequent 0, 1 or 2 microsec. It looks like this (see #16's count):

# tracer: hwlat
#
# entries-in-buffer/entries-written: 20/20   #P:2
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
           -169      d....   328.696323: #1     inner/outer(us):    0/2     ts:946685131.612004155 count:2
           -169      d....   329.711962: #2     inner/outer(us):    0/1     ts:946685132.627644489 count:2
           -169      d....   332.771947: #3     inner/outer(us):    0/1     ts:946685135.687627240 count:2
           -169      d....   333.791924: #4     inner/outer(us):    0/1     ts:946685136.707605282 count:2
           -169      d....   334.811973: #5     inner/outer(us):    0/1     ts:946685137.727653991 count:2
           -169      d....   335.831926: #6     inner/outer(us):    0/1     ts:946685138.747608742 count:2
           -169      d....   336.851921: #7     inner/outer(us):    0/1     ts:946685139.767602242 count:2
           -169      d....   345.011927: #8     inner/outer(us):    0/1     ts:946685147.927608371 count:2
           -169      d....   365.411950: #9     inner/outer(us):    0/1     ts:946685168.327631214 count:2
           -169      d....   455.171927: #10    inner/outer(us):    0/1     ts:946685258.087609424 count:2
           -169      d....   471.491916: #11    inner/outer(us):    0/1     ts:946685274.407599140 count:2
           -169      d....   616.331919: #12    inner/outer(us):    0/1     ts:946685419.247601959 count:2
           -169      d....   805.031908: #13    inner/outer(us):    0/1     ts:946685607.947590632 count:2
           -169      d....  1285.451914: #14    inner/outer(us):    0/1     ts:946686088.367599861 count:2
           -169      d....  1324.211913: #15    inner/outer(us):    0/1     ts:946686127.127599213 count:2
           -169      d....  1540.451925: #16    inner/outer(us):    1/3     ts:946686344.024041733 count:457
           -169      d....  1541.471970: #17    inner/outer(us):    0/2     ts:946686344.387656150 count:2
           -169      d....  1546.571943: #18    inner/outer(us):    0/1     ts:946686349.487628111 count:2
           -169      d....  1547.591934: #19    inner/outer(us):    0/2     ts:946686350.507622153 count:2
           -169      d....  1548.611929: #20    inner/outer(us):    0/1     ts:946686351.527615528 count:2

Since then I have made more half an hour/whole hour tests, their results are similar to those of the first log file shown. Please note that according to the hardware latency detector documentation : > The tracer hwlat_detector is a special purpose tracer that is used to > detect large system latencies induced by the behavior of certain > underlying hardware or firmware, independent of Linux itself. The code > was developed originally to detect SMIs (System Management Interrupts) > on x86 systems, **however there is nothing x86 specific about this > patchset.** I'm assuming that the reasons that x86 processors need SMIs (temperature management, ...) are inevitably present on arm cores too. How should these results be interpreted? \ Hardware latency from the Cortex-A7 doesn't exceed 3 microseconds?\ Is there something wrong with what I’m doing?

I am writing a real-time linux application where I need to prevent any page faults from occuring after the initial startup of my application. My initial thought was just to call mlockall(MCL_CURRENT | MCL_FUTURE);. Calling this function returns no error code, but if I inspect my process's pages, it looks like there are still many pages that have the Locked: size at 0 (which I assume means those pages can still cause a page-fault).

$ cat /proc//smaps |  grep -B 21 -A 2 "Locked:                0 kB"  | grep -B 1 "^Size:" | grep -v "Size" | grep -v "^\-\-"
7effd0021000-7effd4000000 ---p 00000000 00:00 0 
7effd4021000-7effd8000000 ---p 00000000 00:00 0 
7effd80c6000-7effdc000000 ---p 00000000 00:00 0 
7effddf02000-7effddfa0000 rw-s 00000000 00:05 368                        /dev/some_char_device
7effddfa0000-7effde1a0000 rw-s f0000000 00:05 368                        /dev/some_char_device
7effde1c1000-7effde1c2000 ---p 00000000 00:00 0 
7effde1c6000-7effde1ca000 rw-s f7c00000 00:05 368                        /dev/some_char_device
7effde1ca000-7effde1cb000 ---p 00000000 00:00 0 
7effe221b000-7effe221c000 ---p 00000000 00:00 0 
7effe2220000-7effe2223000 rw-s 00000000 00:05 90                         /dev/another_char_device
7effe22df000-7effe22e0000 ---p 00013000 08:02 2234654                    //shared_library1.so
7effe22fd000-7effe22fe000 ---p 0000c000 08:02 2231701                    //shared_library2.so
7effe23fc000-7effe23fd000 ---p 0001c000 08:02 2234652                    //shared_library3.so
7effe2e15000-7effe2e16000 ---p 00215000 08:02 1957                       /usr/lib/x86_64-linux-gnu/libc.so.6
7effe2e40000-7effe2e41000 ---p 00011000 08:02 2234649                    //shared_library4.so
7effe2f14000-7effe2f15000 ---p 00046000 08:02 2232115                    //shared_library5.so
7effe321a000-7effe321b000 ---p 0021a000 08:02 855                        /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7effe3258000-7effe3259000 ---p 0001f000 08:02 2234643                    //shared_library6.so
7effe327d000-7effe327e000 ---p 00021000 08:02 2234641                    //shared_library7.so
7effe328a000-7effe328b000 ---p 00009000 08:02 2232116                    //shared_library8.so
7effe348e000-7effe348f000 ---p 00102000 08:02 91759                      //shared_library9.so
7effe34c6000-7effe34c8000 r--p 00000000 08:02 175                        /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7effe34f2000-7effe34fd000 r--p 0002c000 08:02 175                        /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffc1d1b0000-7ffc1d1b4000 r--p 00000000 00:00 0                          [vvar]
7ffc1d1b4000-7ffc1d1b6000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

# Some attempts and some questions... ## Attempt: Move the location of mlockall in the initialization function My main function has a bunch of dlopen calls. Previously the mlockall call was *after* the calls to dlopen. Moving the call to mlockall *before* the dlopen calls seems to lock in the memory of the shared libraries that are loaded after it. However, it does not lock in the memory of shared libraries loaded *before* the call to mlockall (those shared libraries are linked at compile-time and specified in the executable). **Why doesn't MCL_CURRENT lock in already-loaded libraries?**

$ cat /proc//smaps |  grep -B 21 -A 2 "Locked:                0 kB"  | grep -B 1 "^Size:" | grep -v "Size" | grep -v "^\-\-"
7fef0c021000-7fef10000000 ---p 00000000 00:00 0 
7fef10021000-7fef14000000 ---p 00000000 00:00 0 
7fef140c6000-7fef18000000 ---p 00000000 00:00 0 
7fef1875d000-7fef187fb000 rw-s 00000000 00:05 368                        /dev/some_char_device
7fef187fb000-7fef189fb000 rw-s f0000000 00:05 368                        /dev/some_char_device
7fef18a0a000-7fef18a0b000 ---p 00000000 00:00 0 
7fef1ca2e000-7fef1ca2f000 ---p 00000000 00:00 0 
7fef1ca33000-7fef1ca37000 rw-s f7c00000 00:05 368                        /dev/some_char_device
7fef1ca37000-7fef1ca38000 ---p 00000000 00:00 0 
7fef1ca3c000-7fef1ca3f000 rw-s 00000000 00:05 90                         /dev/another_char_device
7fef1d615000-7fef1d616000 ---p 00215000 08:02 1957                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fef1da1a000-7fef1da1b000 ---p 0021a000 08:02 855                        /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7fef1dcea000-7fef1dceb000 ---p 00102000 08:02 91760                      //shared_library9.so
7fef1dd22000-7fef1dd24000 r--p 00000000 08:02 175                        /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fef1dd4e000-7fef1dd59000 r--p 0002c000 08:02 175                        /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffece1c4000-7ffece1c8000 r--p 00000000 00:00 0                          [vvar]
7ffece1c8000-7ffece1ca000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

## Attempt: Use madvise to prefault pages I tried calling this prefault function (inspired from here ), but madvise seems to return a -1 or EPERM code on the pages with permissions ---p. If I understand correctly, those pages are mapped with MAP_PRIVATE and are supposed to be allocated in physical memory only when written to (as per the Copy-On-Write pattern). However, madvise does not seem to trigger the allocation, and just returns an error instead. **How can I pre-fault pages mapped with MAP_PRIVATE?**

void prefault()
{
    const pid_t pid = getpid();
    
    FILE *fp;
    char path[PATH_MAX];
    char buf;

    (void)snprintf(path, sizeof(path), "/proc/%" PRIdMAX "/maps", (intmax_t)pid);

    fp = fopen(path, "r");

    volatile uint8_t val;

    while (fgets(buf, sizeof(buf), fp)) {
        void *start, *end, *offset;
        int major, minor, n, ret;
        uint64_t inode;
        char prot;

        n = sscanf(buf, "%p-%p %4s %p %x:%x %" PRIu64 " %s\n",
            &start, &end, prot, &offset, &major, &minor,
            &inode, path);

   
        if (n = end) { continue; /* invalid addresse range */ }

        ret = madvise(start, (size_t)((uint8_t *)end - (uint8_t *)start), MADV_POPULATE_WRITE);
    }

    (void)fclose(fp);
}

I have been developing an application that would benefit from faster, more reliable, and ideally synchronized, communication. This led me to look at whether we could implement the App using the open-source provided by OPC Foundation and open62541 Foundation. It appears we can, however, there appear to be some specific hardware requirements (bridges, etc.) that are not available by default when communicating across the internet. Is that correct? Or does as standard fast domestic fibre connection do what we need? Also, I seems like the internet already provides Precision Time Protocol PTP communications for audio visual applications under 802.1Qav. This may in fact be a good starting point for us, so can someone explain if would be feasible to use OPC UA / open62541 with 802.1Qav. And if your wondering - I am Systems Engineer not Networking engineer!

I am trying to isolate CPUs on an AMD EPYC 8534PN (64C/128T). Unfortunately, I see the RCU stalls and the server crashes over and over. And I am not really sure what I am doing wrong. The crashes only happen when the server is under load. Usually, I am running build jobs and some tasks with RT priorities there (integration tests). But from my understanding, by using rcu_nocbs=8-63,72-127 irqaffinity=0-7,64-71 rcu_nocb_poll in grub, setting IRQBALANCE_BANNED_CPULIST=8-63,72-127, and moving the remaining rco's over to the houskeeping group with tuna -t rcu* -c 0-7,64-71 -m should avoid this, shouldnt it? I expect no rcu to be executed on the isolated cores anymore. The errors I am seeing looking like the following: [RCU stalls](https://i.sstatic.net/BKd4B.jpg) This is a part of the dmesg output:

Apr 22 00:08:37 hp-epyc kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Apr 22 00:08:37 hp-epyc kernel: rcu:         Tasks blocked on level-1 rcu_node (CPUs 48-63): P497/2:b..l
Apr 22 00:08:37 hp-epyc kernel:         (detected by 53, t=2940082 jiffies, g=1051549, q=30752212)
Apr 22 00:08:37 hp-epyc kernel: task:ksoftirqd/53    state:R  running task     stack:    0 pid:  497 ppid:     2 flags:0x00004000
Apr 22 00:08:37 hp-epyc kernel: Call Trace:
Apr 22 00:08:37 hp-epyc kernel:  
Apr 22 00:08:37 hp-epyc kernel:  __schedule+0x238/0x5e0
Apr 22 00:08:37 hp-epyc kernel:  ? asm_sysvec_reschedule_ipi+0x1b/0x20
Apr 22 00:08:37 hp-epyc kernel:  preempt_schedule+0x60/0x80
Apr 22 00:08:37 hp-epyc kernel:  preempt_schedule_thunk+0x16/0x18
Apr 22 00:08:37 hp-epyc kernel:  rt_mutex_slowunlock+0x280/0x2f0
Apr 22 00:08:37 hp-epyc kernel:  ? try_to_wake_up+0x307/0x670
Apr 22 00:08:37 hp-epyc kernel:  rt_spin_unlock+0x3e/0x50
Apr 22 00:08:37 hp-epyc kernel:  __hrtimer_run_queues+0x3a0/0x3c0
Apr 22 00:08:37 hp-epyc kernel:  hrtimer_run_softirq+0xa6/0x110
Apr 22 00:08:37 hp-epyc kernel:  __do_softirq+0xc9/0x2cc
Apr 22 00:08:37 hp-epyc kernel:  run_ksoftirqd+0x30/0x80
Apr 22 00:08:37 hp-epyc kernel:  smpboot_thread_fn+0x1d3/0x2d0
Apr 22 00:08:37 hp-epyc kernel:  kthread+0x191/0x1b0
Apr 22 00:08:37 hp-epyc kernel:  ? smpboot_register_percpu_thread+0xe0/0xe0
Apr 22 00:08:37 hp-epyc kernel:  ? set_kthread_struct+0x50/0x50
Apr 22 00:08:37 hp-epyc kernel:  ret_from_fork+0x22/0x30
Apr 22 00:08:37 hp-epyc kernel:

[Complete output](https://pastebin.com/dBtyZ13B) Based on the [lscpu -p output](https://pastebin.com/WKGgUWq1) I came up with the following isolation scheme:

housekeeping & IRQs: 0-7,64-71
isolation: 8-63,72-127

Systemd is configured as following:

sudo systemctl set-property user.slice AllowedCPUs=0-7,64-71
sudo systemctl set-property system.slice AllowedCPUs=0-7,64-71
sudo systemctl set-property init.scope AllowedCPUs=0-7,64-71

The IRQs are banned from the isolated cores in /etc/default/irqbalance:

IRQBALANCE_BANNED_CPULIST=8-63,72-127
IRQBALANCE_ONESHOT=0

And I have moved the rcuo's via tuna -t rcu* -c 0-7,64-71 -m to the housekeeping group. Grub is configured as following:

BOOT_IMAGE=/vmlinuz-5.15.0-1032-realtime root=/dev/mapper/ubuntu--vg-ubuntu--lv ro quiet splash selinux=0 enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 nosoftlockup audit=0 mce=off tsc=nowatchdog skew_tick=1 default_hugepagesz=1GB hugepagesz=1G hugepages=12 iommu=pt amd_iommu=on nohz_full=8-63,72-127 rcu_nocbs=8-63,72-127 irqaffinity=0-7,64-71 nohz=on rcu_nocb_poll numa_balancing=disable transparent_hugepage=never nosoftlockup rcu_nocb_poll processor.max_cstate=0 kthread_cpus=0-7,64-71

OS: Ubuntu Server 22.04.4 LTS \ Kernel: 5.15.0-1032-realtime \ CPU: AMD EPYC 8534PN 64-Core Processor I really don't understand whats going wrong here. I appreciate any kind of help! Thank you! Edit: I was able to stabilize the server. I had some help by this article (unfortunately, subscription only). These are the steps I have taken: The first step was to disable irqbalanced. I found there is a bug that silently fails to enforce the IRQBALANCE_BANNED_CPULIST parameter. I tried to use IRQBALANCE_BANNED_CPU, but this parameter was already deprecated in the version we are using. So I disabled irqbalanced completely. In the article it says "In the past, the ksoftirqd/N threads handled the timer softirq tasks, along with the work for the other softirqs.". I couldn't find any ksoftirqd processes, instead I only found the softirqs which have also appeared in the kernel logs. This is why I have moved the ksoftirqd threads to the highest priority to avoid them starving on the CPU. x=8 y=63 for i in $(seq $x $y) ; do tuna -t "ksoftirqd/$i" -p fifo:99; done a=72 b=127 for i in $(seq $a $b) ; do echo "$i"; tuna -t "ksoftirqd/$i" -p fifo:99; done I have also applied the rcutree.kthread_prio=99 kernel parameter to the kernel command line to set the priority of the RCU rcuc/N and rcub/N threads to 99. To give a complete picture, I am running RT workloads with priorities up to 50 on the server through Kubernetes. I also run unit and build tests in parallel to the RT workloads. Can someone confirm the above assumption that ksoftirqd still handels timers and work ob Ubuntu is still correct? I couldnt find much about it researching. Another question I have regarding this topic is, why are not all rcu workloads moved properly to the housekeeping group? I don't want to use isolcpus, which I guess would help with this. Is there another way to move the interrupts completely so the isolated cores are free for RT workloads? Thanks in advance!

I'm trying to install a real-time kernel on EndeavourOS without any success. I tried different method... the first one, downloading it from yay repository as suggested by the Arch Linux wiki but compiling never ends; then, I tried to download first the PKGBUILD and then compiling it later but the result was the same; finally, I've found a webpage on the Linux Foundation's site about PREEMPT_RT patch and I followed the instruction... first, download the kernel (in my case, 6.0[EndeavourOS already mount a 6.0.2 version, but real-time patch was not upgraded yet to this... that's why I downloaded the '6.0' one and the related rt-patch '6.0' to match them perfectly]). In a first attempt to this last method, I tried mixing the 6.0.3 version of the linux kernel with the 6.0 rt-patch. Maybe I have to remove something of the old attempts? And how I have to do it? Arch (and linux in general) is a new thing for me (I know a little better fedora but arch is more performant for my purpose). Terminal returns this: Reversed (or previously applied) patch detected! Assume -R? [n] and this is returned for different stuff but it's long to show here. Any suggestion? Thanks in advance.

What is the correct way to sleep to us resolution in a thread? I'm trying to write a device driver and I need to keep a pin high for a certain number of us after a GPIO interrupt. My plan is to create a kernel module for GPIO interrupt which will set the pin high and spawn a kernel thread to bring it low after a certain time. Will this work?

I'm using a Raspberry Pi board and I would like to build a real time system. During the kernel configuration, I found the **CONFIG_CPU_FREQ** option, which (from the help section) > allows you to change the clock speed of the CPU on the fly. This is a > nice method to save power, because the lower the CPU clock speed, the > less power the CPU consumes. That's a nice approach to consume less power but in real time applications this could lead to unpredictable behavior and high latencies. Xenomai developers suggest to disable **CONFIG_CPU_IDLE** because (from here ) > it allows the CPU to enter deep sleep states, increasing the time it > takes to get out of these sleep states, hence the latency of an idle > system. Also, on some CPU, entering these deep sleep states causes the > timers used by Xenomai to stop functioning I was wondering if disabling these two options could damage the cpu since the board does not come with a heat dissipation system? I imagine that, without these options, the cpu will be more reactive but it sounds like a CPU-Seppuku to me.

I want to use network emulation ([netem](https://man7.org/linux/man-pages/man8/tc-netem.8.html)) on a [PREEMPT-RT kernel](https://archive.kernel.org/oldwiki/rt.wiki.kernel.org/) to emulate latency and jitter down to 0.5 ms +- 10 %. Thus, I initially thought that I have to adapt the internal kernel clock rate to at least 2000 Hz as this means I can add time-deterministic delays of 0.5 ms. However, it seems that with lower kernel clock rates, it should work fine too (measured with experiments). After some research, I thought it could be due to a tickless setting or dynamic tick in the kernel config, but basically I am now rather confused how kernel clock rate works and why it is important. So is the kernel clock rate adaption needed at all and do I have a comprehension problem how the clock rates actually work? Thanks for your help :)

I'm programming an embedded real-time Linux device with a 4 core ARM CPU. There is a requirement for a periodic computation at 10 kHz that should not jitter too much and should never be lost. My POSIX thread can read a HW-provided 10 kHz toggle register in a busy loop and perform the computation whenever the bit is toggled. In order to keep the SCHED_RR scheduler from interrupting the pthread I set the CPU affinity to core 2 exclusively for this thread, and the priority to 99 (maximum). A handful of other threads have CPU affinity set to cores other than 2 and priority 50. But there's still a systemd running and a bunch of other processes. Is this enough to keep the thread from ever being interrupted by the scheduler? If not, is there a way to achieve this?

is there a sleep command that works with real date time and ignores suspend? echo a;sleep 3600;echo b i work 10mins, suspend the notebook for 2h. When I wake it up from suspend, "b" will only print after 50mins instead of imediately. an alternative would be to

nA=date +%s;nB=$((nA+3600))
while((date +%s

But i wonder if there is a command for that? If sleep --usedatetimecheck 3s 3600 (the suspendable sleep would be optional like 3s) existed it would be great. Btw, a trick i use to detect when the pc wakes up from suspend is sleep 3s, compare datetime of before sleep and if it is more than 9s it guesses it wokeup (could also check for high load average and increase the 9s test). i use ubuntu. Ps.: someone could argue that several things would kickin imediately. But that is why it should be optional. Another thing i just thought is if we could send a signal to update all sleep commands like pkill -SIGUSR1 sleep?