**How can I best debug the following issue from here on?** I have a - python application - running in a **podman container** which has - plenty disk I/O due to a **numpy memmap** of two huge files, one input file which I read in chunks, process the chunk and write the result in the other mem mapped file. I noticed that running the application leads to - **increasing total RAM usage**, eventually leading to - OOM situations on my system. I tried finding the cause but - couldn't find references in my python code which would leak memory. According to sudo htop - the application itself **does not consume the memory** when accounting for resident memory size only, which it defines as text sections + data sections + stack usage. - Mem 11.0G / 14.6G - 18.7 MEM% (of ~16G RAM) later: - Mem 14.3 / 14.6G - 0.1 MEM% (of ~16G RAM) According to ps auwx: |USER|PID|%CPU|%MEM|VSZ|RSS|TTY|STAT|START|TIME|COMMAND |--|--|--|--|--|--|--|--|--|--|--|--|--|--|--| |user|9874|44.9|0.0|14651602572|8092|pts/0|DNsl+|10:40|275:13|/root/.local/share/virtualenvs/app-4PlAip0Q/bin/python main.py

# /proc/meminfo
MemTotal:       15258424 kB
MemFree:          143284 kB
MemAvailable:          0 kB
Buffers:             304 kB
Cached:            51496 kB
SwapCached:         1380 kB
Active:              812 kB
Inactive:           5800 kB
Active(anon):          0 kB
Inactive(anon):     1492 kB
Active(file):        812 kB
Inactive(file):     4308 kB
Unevictable:       46068 kB
Mlocked:               0 kB
SwapTotal:      67108860 kB
SwapFree:       66860540 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:               144 kB
Writeback:           124 kB
AnonPages:          1276 kB
Mapped:             2916 kB
Shmem:             46020 kB
KReclaimable:      40196 kB
Slab:            1482400 kB
SReclaimable:      40196 kB
SUnreclaim:      1442204 kB
KernelStack:        3984 kB
PageTables:     13424812 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    74737048 kB
Committed_AS:    1396560 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       54808 kB
VmallocChunk:          0 kB
Percpu:             2496 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       1
HugePages_Free:        1
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:            2048 kB
DirectMap4k:      464816 kB
DirectMap2M:    14129152 kB
DirectMap1G:     2097152 kB

**Screenshots**

I have been working on a Altera DE1-SoC Development Board for 8 months. The system I was working on includes a Cyclone V FPGA chip, particularly the 5CSEMA5F31C6N. It was running an embedded Linux operating system on chip. All was well and development was on-going. Two weeks ago, a new custom board was put together by a hardware engineer in the company. The design and components were mostly similar to that of the Development Board. All HPS related pins are wired the same way, with one main difference being that the default console port was UART1 instead. That issue has since been resolved and I now am able to receive the U-boot and Kernel messages through UART1. But the system did not completely boot. I have pinpointed this to multiple reasons. Firstly, I had an init.d script that would export the GPIO LEDs and create a sysfs file. Exporting the gpio pin works, however, changing the direction, or changing the value, or reading from it, causes the system to freeze. I disabled that function on the init.d script and rebooted the device. This time the boot failed on another init.d script line. This line was going to change the value of a register in the lightweight bridge. The command was devmem 0xFF200XXX 32 1, with XXX being the specific register. I tried using devmem on all bridges but all attempts would freeze Linux. I tried using devmem on the UART register of HPS, on the SDCard register of HPS (referenced [here](https://www.intel.com/content/www/us/en/programmable/hps/cyclone-v/hps.html)) , and it does not freeze. I can verify that the bridge is enabled by reading the state sysfs file of each bridge: [fpga_bridge state returns enabled](https://i.sstatic.net/pqyha.png) I can also verify that the bridges are linked to the driver from this dmesg output: [dmesg output](https://i.sstatic.net/EQrDY.png) I have enabled all three bridges in the hps configuration using Quartus Platform Designer. I also have the following lines in my u-boot.scr:

fatload mmc 0:1 $fpgadata soc_system.rbf;
fpga load 0 $fpgadata $filesize;
setenv fdtimage soc_system.dtb;
run bridge_enable_handoff;
run mmcload;
run mmcboot;

I have also attempted enabling the bridges through the U-boot command line following [these instructions](https://www.rocketboards.org/foswiki/Documentation/GSRD131ProgrammingFPGA) . However, I am unable to write anything into $l3regs: [writing into l3regs](https://i.sstatic.net/s1pvY.png) I am building the OS using Buildroot 2016.05 with a 4.4 Linux Kernel. To create the .rbf, .dts, .dtb, preloader-mkpimage.bin, and u-boot files, I am using SoC EDS 18.1 [Build 625]. I have run out of things to try. I would consider the issue solved if I am able to toggle an LED on and off from the Linux OS, using sysfs files. Assuming that the hardware is correct, what else could be the cause and how do I fix it?

I'm curious because today the only way I know how to give two different processes the same shared-memory is through a memory-mapped file, in other words, both processes open the same memory-mapped file and write/read to/from it. That has penalties / drawbacks as the operating system needs to swap between disk and memory. Apologies in advance if that is a silly question, but is there such a thing as a pure shared memory between processes, not backed by a file. If yes, how would the processes get a hold of it if not using a memory-mapped file or /dev/shm file?

I've been tasked at worked to explore memory mapping to see if we can utilize it. I'm trying to wrap my head around the concept and how to write the code. I've been trying out the code in the following video and blog post, but I'll go over my steps here as well so it's easy to recreate. I'm working in C++ (gcc version 11.2.0) on a Linux system https://www.youtube.com/watch?v=m7E9piHcfr4 https://bertvandenbroucke.netlify.app/2019/12/08/memory-mapping-files/ Here's my code write_mmap.cpp

#include 
#include 
#include 
#include 

int main() {
  // reading
  int file_read = open("test.dat", O_RDONLY, 0);
  // writing
  int file_write = open("test.dat", O_CREAT | O_RDWR,
                      S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP);
  posix_fallocate(file_write, 0, 4096);

  char *buffer = reinterpret_cast (mmap(NULL, 4096, PROT_WRITE, MAP_SHARED, file_write, 0));

  for (int i = 0; i < 4096; i++) {
    // Change each upper-case A to lowercase in the file
    if (buffer[i] == 'A') {
      buffer[i] = 'a';
    }
  }
}

And test.dat looks like this

Name,marker1,marker2,marker3,marker4
barc1,AA,AB,BB,--
barc2,AB,AA,BB,--

After running the commands

g++ write_mmap.cpp
./a.out

I see that test.dat has been changed to

Name,marker1,marker2,marker3,marker4
barc1,aa,aB,BB,--
barc2,aB,aa,BB,--

Which is what I was going for. I'm just unsure about a lot of things here Firstly, Is test.dat a memory-mapped file? I see it in the filesystem when I ls in my directory, but is it in virtual memory somewhere? If so, is there some way that I can figure out where in the virtual memory it's stored? Why is it that it was originally a text file when I wrote it, but it becomes a binary file after the script is done with it - when I open the file with nano, I see

Name,marker1,marker2,marker3,marker4
barc1,aa,aB,BB,--
barc2,aB,aa,BB,--
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^

I'd also like to get some things cleared up about the memory mapping 1) This file is in RAM right now, right? Will any program (regardless of language used) be able to read from those portions of memory? 2) Is there some way to set "bookmarks" in this file. So that if I want the computer to start reading the file from a certain portion, I can tell it to read from address A to address B?

Is there any difference between the data transfer paths using read()/write() and using mmap() on a file? What does "kernel" mean in https://stackoverflow.com/a/41419353 ? > mmap doesn't require a copy of the file data from **kernel** to user-space. Is it correct that read() copies data from file to a buffer in kernel space , then from the kernel buffer to a buffer in user space? When calling mmap() on a file, does it map virtual memory addresses directly to the file, without anything (e.g. a buffer in kernel space) in between? When accessing virtual memory addresses returned by mmap(), what is the path for data transfer? Is there any buffer in kernel space involved in the middle behind the scene? Does mmap() have an advantage over read()/write(), as far as data transfer path is concerned?

I know that

of a process shows how the memory mapping is done. For example, the first lines of pmap output shows the memory mapping to the text segment of the process' executable. Assume I have a really big executable and my code size is large. For a small text segment(small executable) the pmamp shows

- 401000 - 4096 B    - 4 KiB

of text segment mapped. (1) If the text segment is really large how does it map? (2) Is it map dynamically based on the pages required as part of the code execution progress?

I have a SoC-FPGA(DE0-nano-soc) which contains an ARM-Cortex-A9 cpu with a Cyclone V FPGA on a single chip. the CPU has access to 1gb of DDR3 memory but the FPGA can also access this memory so they can share data. Now I have configured linux to only see 900M of the total RAM using the boot parameter mem=900M. This way I have some RAM left linux doesn't touch that I can use to communicate with the FPGA. I want to write to that part of the RAM using /dev/mem. I get a permission denied error when I memory map the appropriate address, I am guessing this is because my kernel has CONFIG_STRICT_DEVMEM set. Now I don't really want to unset this since I really only need to write to adresses which linux doesn't use. Is there a way to tell the linux kernel that it is fine to write to that specific part but the normal memory is still of limits?

In Operating System Concepts, memory mapped files and memory mapped I/O are two different things. See below about memory mapped I/O. To use memory mapped files, we have mmap(). To use memory mapped I/O, what functions can we use? Is memory mapped I/O only used internally by OS, not exposed to and used by programmers on top of Linux? Thanks. > In the case of I/O, as mentioned in Section 1.2.1, each I/O controller includes registers to hold commands and the data being transferred. Usually, special I/O instructions allow data transfers between these registers and system memory. To allow more convenient access to I/O devices, many computer architectures provide **memory-mapped I/O**. In this case, ranges of memory addresses are set aside and are mapped to the device registers. Reads and writes to these memory addresses cause the data to be transferred to and from the device registers. This method is appropriate for devices that have fast response times, such as video controllers. In the IBM PC, each location on the screen is mapped to a memory location. Displaying text on the screen is almost as easy as writing the text into the appropriate memory-mapped locations. ... > **How can the processor give commands and data to a controller to accomplish an I/O transfer?** The short answer is that the controller has one or more registers for data and control signals. The processor communicates with the controller by reading and writing bit patterns in these registers. One way in which this communication can occur is **through the use of special I/O instructions** that specify the transfer of a byte or word to an I/O port address. The I/O instruction triggers bus lines to select the proper device and to move bits into or out of a device register. Alternatively, the device controller can support **memory-mapped I/O**. In this case, the device-control registers are mapped into the address space of the processor. The CPU executes I/O requests using the standard data-transfer instructions to read and write the device-control registers at their mapped locations in physical memory.

Are sharing a memory-mapped file and sharing a memory region implemented based on each other? The following two quotes seem to say so, and seem a chicken-egg problem to me. Operating System Concepts introduces sharing a memory-mapped file in the following. Do the multiple processes share the same file by sharing the same physical memory region holding the content of the file? > **Multiple processes may be allowed to map the same file concurrently**, to allow sharing of data. Writes by any of the processes modify the data in virtual memory and can be seen by all others that map the same section of the file. Given our earlier discussions of virtual memory, it should be clear how the sharing of memory-mapped sections of memory is implemented: **the virtual memory map of each sharing process points to the same page of physical memory—the page that holds a copy of the disk block.** This memory sharing is illustrated in Figure 9.22. > It also introduces shared memory in the following. - Do multiple processes share a memory region by sharing a memory-mapped file? - Does a "memory-mapped file" reside on disk or main memory? I think it is on the disk, but "The memory-mapped file serves as the region of shared memory between the communicating processes" seems to mean that it resides in main memory. > Quite often, **shared memory is in fact implemented by memory mapping files**. Under this scenario, processes can communicate using shared memory by having the communicating processes memory-map the same file into their virtual address spaces. The memory-mapped file serves as the region of shared memory between the communicating processes (Figure 9.23). > Thanks.