_Originally posted [here](https://stackoverflow.com/questions/79391172/clarifications-on-perf-events-data-collection) on stackoverflow_ I have never used the perf command before (but I need it), hence I have been reading the (really useful) [PerfWiki](https://perfwiki.github.io/main/) . The section devoted to [Event-based sampling overview](https://perfwiki.github.io/main/tutorial/#sampling-with-perf-record) contains a number of statements that are not completely clear to me. As they are quite essential to precisely understand how data collection is carried out, here I am asking for your help. In the following I will quote paragraphs from that section and explain my duobts immediately after. > Perf_events is based on event-based sampling. The period is expressed as the number of occurrences of an event, not the number of timer ticks. A sample is recorded when the sampling counter overflows, i.e., wraps from 2^64 back to 0. No PMU implements 64-bit hardware counters, but perf_events emulates such counters in software. Q1. Consider a CPU working at a fixed frequency (no frequency scaling): what is the precise definition of a timer tick? Is it true that the _number_ of ticks in a second equals the _value_ of the frequency (e.g. 1_000_000_000 ticks/second for a CPU working at 1 GHz)? Q2. perf does not use timer ticks, instead, it counts the number of times an event occurs and only "stops" the CPU to gather the relevant data once ever period times; e.g. if period=1 each occurence of an event is registered, if period=2 it only registers half of the total number of occurrences, and so on... is it right? When period > 1 does perf automatically scale the final values and provide data as if all the events were registered? Q3. The above section says that "A sample is recorded when the sampling counter overflows, i.e., wraps from 2^64 back to 0" which seems to contradict the measurement being taken once every period occurrences of an event... what am I missing? More generally, why does perf wait for a counter to overflow before gathering the information? Also, what happens when more than one event is being monitored? > The way perf_events emulates 64-bit counter is limited to expressing sampling periods using the number of bits in the actual hardware counters. If this is smaller than 64, the kernel **silently** truncates the period in this case. Therefore, it is best if the period is always smaller than 2^31 if running on 32-bit systems. Q4. I cannot truly understand the meaning of this paragraph (maybe I am missing some underlying knowledge). If the actual hardware counter has `N On counter overflow, the kernel records information, i.e., a sample, about the execution of the program. What gets recorded depends on the type of measurement. This is all specified by the user and the tool. But the key information that is common in all samples is the instruction pointer, i.e. where was the program when it was interrupted. Ok, I got this. > Interrupt-based sampling introduces skids on modern processors. That means that the instruction pointer stored in each sample designates the place where the program was interrupted to process the PMU interrupt, not the place where the counter actually overflows, i.e., where it was at the end of the sampling period. In some case, the distance between those two points may be several dozen instructions or more if there were taken branches. When the program cannot make forward progress, those two locations are indeed identical. **For this reason, care must be taken when interpreting profiles.** Q5. I am aware I know way too little about how a CPU actually works to understand this section, but could you confirm that this paragraph is warning about the following potential chain of events (pun not intended): 1. A sample must be taken 2. The current value in the istruction pointer register is taken 3. A few more instructions are executed by the CPU 4. The sample is taken and the gathered data is saved using and associated with the instruction pointer taken at step (2) If this is correct, could you give me a brief explanation (or point me to some external resource) about what may cause step (3)? More importantly, how can I monitor how many times skids have occurred? Is there anything on my side that I can do to mitigate this? > By default, perf record uses the cycles event as the sampling event. > > [...] > > The perf_events interface allows two modes to express the sampling period: > > * the number of occurrences of the event (period) > * the average rate of samples/sec (frequency) > > The perf tool defaults to the average rate. It is set to 1000Hz, or 1000 samples/sec. That means that the kernel is dynamically adjusting the sampling period to achieve the target average rate. Q6. Does perf use the cycles event as a reference to compute the sampling period even if it is not among the set of events being monitored? What happens when multiple events are monitored? Does each event have its own period, or is there one event that counts for all? When a frequency is used to determine when samples must be taken, the event used as reference for sampling should be irrelevant, right?

The Ubuntu version I am using is **Ubuntu 18.04.6 LTS**, and the kernel version is **5.4.0-148 generic**. My processor is **12th Gen Intel (R) Core (TM) i7-12700**. I would like to know the number of TLB entries in my CPU for different page sizes (1G, 2MB, 4KB), as well as the number of dTLB misses during program execution.

-1

command told me they are 0： L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax): instruction # entries = 0x0 (0) instruction associativity = 0x0 (0) data # entries = 0x0 (0) data associativity = 0x0 (0) L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx): instruction # entries = 0x0 (0) instruction associativity = 0x0 (0) data # entries = 0x0 (0) data associativity = 0x0 (0) L1 data cache information (0x80000005/ecx): line size (bytes) = 0x0 (0) lines per tag = 0x0 (0) associativity = 0x0 (0) size (KB) = 0x0 (0) L1 instruction cache information (0x80000005/edx): line size (bytes) = 0x0 (0) lines per tag = 0x0 (0) associativity = 0x0 (0) size (KB) = 0x0 (0) L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax): instruction # entries = 0x0 (0) instruction associativity = L2 off (0) data # entries = 0x0 (0) data associativity = L2 off (0) L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx): instruction # entries = 0x0 (0) instruction associativity = L2 off (0) data # entries = 0x0 (0) data associativity = L2 off (0) L2 unified cache information (0x80000006/ecx): line size (bytes) = 0x40 (64) lines per tag = 0x0 (0) associativity = 0x7 (7) size (KB) = 0x500 (1280) L3 cache information (0x80000006/edx): line size (bytes) = 0x0 (0) lines per tag = 0x0 (0) associativity = L2 off (0) size (in 512KB units) = 0x0 (0)

stat -e dTLB-loads,dTLB-load-misses,iTLB-load-misses

shows not supported UPDATE： $ cpuid -1 -l 2 CPU: 0xff: cache data is in CPUID 4 0xfe: unknown 0xf0: 64 byte prefetching $ cpuid -1 -l 0x18 CPU: 0x00000018 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 $ perf -v perf version 5.4.231 $ perf list List of pre-defined events (to be used in -e): branch-instructions OR branches [Hardware event] branch-misses [Hardware event] bus-cycles [Hardware event] cache-misses [Hardware event] cache-references [Hardware event] cpu-cycles OR cycles [Hardware event] instructions [Hardware event] ref-cycles [Hardware event] alignment-faults [Software event] bpf-output [Software event] context-switches OR cs [Software event] cpu-clock [Software event] cpu-migrations OR migrations [Software event] dummy [Software event] emulation-faults [Software event] major-faults [Software event] minor-faults [Software event] page-faults OR faults [Software event] task-clock [Software event] I'm completely confused about what's wrong with my machine.

Do I need to have to run perf userspace tool as system administrator (root), or can I run it (or at least some subcommands) as an ordinary user?

perf list shows a bunch of what it calls metrics. The list starts off with List of pre-defined events (to be used in -e): Metrics: BAClear_Cost [Average Branch Address Clear Cost (fraction of cycles)] C2_Pkg_Residency [C2 residency percent per package] C3_Core_Residency [C3 residency percent per core] C3_Pkg_Residency [C3 residency percent per package] I'd like to experiment with these but I can't get perf stat to use them. Since the first line says "(to be used in -e)", I tried the following, but it produces an error 103> perf stat -e IPL sleep 10 event syntax error: 'IPL' \___ parser error Then I found some examples on the internet using -M, but this doesn't work either. 103 > perf stat -M IPL sleep 10 Cannot find metric or group `IPL' Usage: perf stat [] [] -M, --metrics monitor specified metrics or metric groups (separated by ,) Can someone explain how to get perf to use these metrics? Thanks.

I am trying to get the L3 cache miss rate when I try to use perf. The following is the command I run: perf stat -e instructions,LLC-load-misses,LLC-loads ./program Then I get the following output: 667,922,092,353 instructions LLC-load-misses LLC-loads 47.619915412 seconds time elapsed 42.245969000 seconds user 5.371741000 seconds sys I have already run the following command to install perf: apt-get install linux-tools-common linux-tools-generic linux-tools-uname -r The following are my system specs: OS: Ubuntu 22.04.1 \ Kernel: 5.15.0-58-generic \ CPU: AMD Ryzen 9 5900x It also does not work if I run as root

I ran perf stat -e itlb_misses.miss_causes_a_walk,itlb_misses.walk_completed,itlb_misses.walk_pending grep Hecuba /storage/nvme6/complete_shakespeare_4096_times and got Performance counter stats for 'grep Hecuba /storage/nvme6/complete_shakespeare_4096_times': 27,732 itlb_misses.miss_causes_a_walk 13,492 itlb_misses.walk_completed 1,048,087 itlb_misses.walk_pending 0.252038252 seconds time elapsed So obviously these three are all different. The actual workload I'm running perf against is not important in that whatever workload I run relative values for the counters are similar. I know why a miss would cause a walk, but the above seems to imply that not all walks complete. Why is this? Also why are the pending walks so much greater than the caused walks? And finally if I want to understand the itlb miss rate, which of the above counters should I use?

When reading the Performance Counters (PMC) on CPUS, the kernel.perf_event_paranoid must be 1 (check after syscall) I tested it on - Archlinux Linux host1 5.17.9-arch1-1 #1 SMP PREEMPT Wed, 18 May 2022 17:30:11 +0000 x86_64 GNU/Linux - Ubuntu Linux host2 5.15.0-30-generic #31-Ubuntu SMP Thu May 5 10:00:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Now, by default, the parameter is 2 (Arch), 4 (Ubuntu). I run the program and it should not be able to open the perf counter. On Arch, it can still do it, on Ubuntu fails (as expected). After tinkering and pointers from Kamil on SU SE , I found that for Ubuntu there exists a patch which requires the setting to be 4 to actually have an effect. - Is there anything similar on ArchLinux? - Can this behaviour be reproduced on other Distros, too? This script could help to reproduce

#!/bin/bash

cat >pmc.c 
#include 
#include 
#include 
#include 
#include 
#include 

static struct perf_event_attr attr;
static int fdperf = -1;
static struct perf_event_mmap_page *buf = 0;

long long cpucycles_amd64rdpmc(void) {
  long long result;
  unsigned int seq;
  long long index;
  long long offset;

  if (fdperf == -1) {
    attr.type = PERF_TYPE_HARDWARE;
    attr.config = PERF_COUNT_HW_CPU_CYCLES;
    attr.exclude_kernel = 1;
    fdperf = syscall(__NR_perf_event_open, &attr, 0, -1, -1, 0);
    if (fdperf == -1){
        fprintf(stderr, "\03331m--> could not open perf counter. Check paranoid setting\033[0m\n");
        exit(1);
    }
    buf = mmap(NULL, sysconf(_SC_PAGESIZE), PROT_READ, MAP_SHARED, fdperf, 0);
  }

  do {
    seq = buf->lock;
    asm volatile("" ::: "memory");
    index = buf->index;
    offset = buf->offset;
    asm volatile("rdpmc;shlq $32,%%rdx;orq %%rdx,%%rax"
                 : "=a"(result)
                 : "c"(index - 1)
                 : "%rdx");
    asm volatile("" ::: "memory");
  } while (buf->lock != seq);

  result += offset;
  result &= 0xffffffffffff;
  return result;
}

int main() {

  long long c = cpucycles_amd64rdpmc();
  printf("counter: %llx\n", c);
  return 0;
}
EOF

param_name=kernel.perf_event_paranoid
read_pmc() {
  echo -n "--> reading, "
  sysctl "${param_name}"
}

set_pmc() {
  n=$1
  echo -e "--> setting to ${n}"
  sudo sysctl -w "${param_name}=${n}"

  read_pmc
  echo "should be ${n}"
}

run() {
  echo "--> running"
  ./pmc
}

#compile
gcc pmc.c -o pmc

read_pmc
run
echo -e "\n--> if ${param_name} as >1, that should have printed the error message\n\n"

set_pmc 1

run
echo -e "\n--> that should have worked and printed the counter.\n"

#re-set to 2
set_pmc 2
run
echo -e "\033[31m--> that should NOT have worked but it printed the counter\033[0m.\n"

rm pmc.c pmc

(rdpmc code is based on cpucycles/amd64rdpmc.c from [SUPERCOP ) Related: - setting the kernel parameter - My question on how I read the PMC

I'm struggling getting perf_events to give me stack traces with symbols, despite reading many tutorials on the subject and doing (I think) all the necessary things. It's possible that my local install of perf (details on that below) is somehow botched? Anyway, here's what I did: main.cpp is a simple C++ program that calls a few functions defined in the same file, allocates some memory and frees it, and prints a few things out. compilation command: gcc -std=c++11 -lstdc++ main.cpp -Og -fno-omit-frame-pointer -fno-inline -o arr_test profile command: perf record -a -g -- ./arr_test && perf report --stdio I do get the following warnings about kernel symbols, but I don't think this should matter given that I only care about symbols in my application for now: [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.052 MB perf.data (~2285 samples) ] [kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806 not found, continuing without symbols [kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806 not found, continuing without symbols Warning: Kernel address maps (/proc/{kallsyms,modules}) were restricted. Check /proc/sys/kernel/kptr_restrict before running 'perf record'. As no suitable kallsyms nor vmlinux was found, kernel samples can't be resolved. Samples in kernel modules can't be resolved as well. Here's a snippet of the output: # Overhead Command Shared Object # ........ ........ ................. # 83.27% arr_test arr_test | |--34.12%-- 0x400908 | 0x7fe72b381ec5 | |--10.48%-- 0x400903 | 0x7fe72b381ec5 | |--10.08%-- 0x4008b8 | 0x7fe72b381ec5 | |--9.22%-- 0x4008e5 | 0x7fe72b381ec5 | |--9.05%-- 0x4008da | 0x7fe72b381ec5 | |--8.49%-- 0x4008f0 | 0x7fe72b381ec5 | |--6.87%-- 0x4008d5 | 0x7fe72b381ec5 | |--6.23%-- 0x4008c2 | 0x7fe72b381ec5 | |--4.76%-- 0x4008fd | 0x7fe72b381ec5 --0.70%-- [...] 8.02% arr_test [kernel.kallsyms] | |--4.87%-- 0xffffffff81140b64 | 0xffffffff81146646 | 0xffffffff81182751 | 0xffffffff811829eb | 0xffffffff8173317d | 0x7fe72bab86a7 | 0x7fe72baa7e00 file info (shows "not stripped"): $ file arr_test arr_test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, not stripped Details on my perf install (do any of these warnings prevent me from seeing symbols in stacks?) Auto-detecting system features: ... backtrace: [ on ] ... dwarf: [ OFF ] ... fortify-source: [ on ] ... glibc: [ on ] ... gtk2: [ on ] ... gtk2-infobar: [ on ] ... libaudit: [ OFF ] ... libbfd: [ OFF ] ... libelf: [ OFF ] ... libelf-getphdrnum: [ OFF ] ... libelf-mmap: [ OFF ] ... libnuma: [ on ] ... libperl: [ on ] ... libpython: [ on ] ... libpython-version: [ on ] ... libslang: [ on ] ... libunwind: [ OFF ] ... on-exit: [ on ] ... stackprotector: [ on ] ... stackprotector-all: [ on ] ... timerfd: [ on ] config/Makefile:264: No libelf found, disables 'probe' tool, please install elfutils-libelf-devel/libelf-dev config/Makefile:329: No libunwind found, disabling post unwind support. Please install libunwind-dev[el] >= 1.1 config/Makefile:354: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev How can I find my symbols in perf?

I am trying to use linux curt (https://github.com/open-power-sdk/curt) tool for monitoring CPU utilization values of threads. On running this command ''' perf record -e '{raw_syscalls:*,sched:sched_switch,sched:sched_migrate_task,sched:sched_process_exec,sched:sched_process_fork,sched:sched_process_exit,sched:sched_stat_runtime,sched:sched_stat_wait,sched:sched_stat_sleep,sched:sched_stat_blocked,sched:sched_stat_iowait,powerpc:hcall_entry,powerpc:hcall_exit}' -a *command --args* ''' I am getting following error (my ubuntu distribution is 20.04). even if skip this step. I am getting errors while running ''' perf script -s ./curt.py''' command. please guide me on how to solve this issue?

I'm writing a video game, and have used perf(1) and flame graphs to get some decent "whole program performance" information. However, I also want to investigate specific worst-case periods, such as certain frames that take too long. Is there a good way to do this kind of measurement with perf? Equivalently, you could imagine this in a web server context, where you want to investigate specific slow responses that were sent, not just overall throughput. I had the following notion, but I'm wondering if I can do better: dump out the full list of events with perf record and perf script, and also dump out start/end timestamps from my game for slow frames. Then, use some script to correlate the timestamps and only look at the perf events relevant to that time period. Seems workable, but a little brittle based on timestamps. Maybe there is some way to add calls inside my code that say "start recording perf events now", and "stop recording them now", like creating a perf.dat file per-frame ? Or maybe I can output some special events like "frame begin, frame end", which would get included in the full event log that is dumped from my program run? This way everything is in the perf.dat file, and I don't have to correlate across two different output files. Or any other ideas?

I saw in my syslog kernel.perf_event_max_sample_rate get changed. I was wondering if I could write a quick script to log this variable every few minutes. Currently it is: sysctl -a | grep kernel.perf_event_max_sample_rate In the man page sysctl says > sysctl - configure kernel parameters at runtime Does that mean that my script would get the parameter as it was set when the kernel starts? Would it pick up changes?

I have a problem in using linux perf on a newly bought laptop: there is no available hardware cache effect in my perf list!!! Well, this is really all important information that I wish to sample!! Here is my perf list: List of pre-defined events (to be used in -e): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] cache-references [Hardware event] cache-misses [Hardware event] branch-instructions OR branches [Hardware event] branch-misses [Hardware event] bus-cycles [Hardware event] ref-cycles [Hardware event] cpu-clock [Software event] task-clock [Software event] page-faults OR faults [Software event] context-switches OR cs [Software event] cpu-migrations OR migrations [Software event] minor-faults [Software event] major-faults [Software event] alignment-faults [Software event] emulation-faults [Software event] dummy [Software event] branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] branch-misses OR cpu/branch-misses/ [Kernel PMU event] bus-cycles OR cpu/bus-cycles/ [Kernel PMU event] cache-misses OR cpu/cache-misses/ [Kernel PMU event] cache-references OR cpu/cache-references/ [Kernel PMU event] cpu-cycles OR cpu/cpu-cycles/ [Kernel PMU event] instructions OR cpu/instructions/ [Kernel PMU event] power/energy-cores/ [Kernel PMU event] power/energy-gpu/ [Kernel PMU event] power/energy-pkg/ [Kernel PMU event] power/energy-ram/ [Kernel PMU event] ref-cycles OR cpu/ref-cycles/ [Kernel PMU event] rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] (see 'man perf-list' on how to encode it) mem:[:access] [Hardware breakpoint] [ Tracepoints not available: Permission denied ] while this is the perf list I used to see: . What I used to do is: sudo perf stat -e L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses -a --append -o perf.txt [some command to run a file] but this does not work on my new machine. How can I collect data I want in this case? I am using Ubuntu 14.04, with kernel . Perf version . ------ **update** I installed the papi-tools library, and papi_native_avail gives me Available native events and hardware information. PAPI Version : 5.3.0.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Core(TM) M-5Y71 CPU @ 1.20GHz (61) CPU Revision : 4.000000 CPUID Info : Family: 6 Model: 61 Stepping: 4 CPU Max Megahertz : 2900 CPU Min Megahertz : 500 Hdw Threads per core : 1 Cores per Socket : 2 Sockets : 2 NUMA Nodes : 1 CPUs per Node : 4 Total CPUs : 4 Running in a VM : no Number Hardware Counters : 0 Max Multiplex Counters : 64

I am trying to use Linux's perf_events framework to investigate an issue with an application on one of our servers. Based on my reading about the perf tool, collecting stacks is relatively straightforward. I want to know if it's possible to simply use the wall-time as a kind of event to collect stacks every N seconds. The current command I've been using is:

perf record -e cycles -T -o /samples.data -F 1 --call-graph dwarf -T -p

From my current understanding, this command will sample every 1 second (-F 1) and grab the stacks (-g) from the process (-p ) until the perf command is terminated with a signal. But based on the data I collect, it seems like there is more than one set of samples per second? So, I think I am misinterpreting or misunderstanding something about the way perf collects samples. Also, what if I wanted to record stacks every 2, 5, or 10 seconds to reduce the amount of data collected? Is there a way to achieve that with perf?

I want to retrieve performance counter counts at a high frequency (i.e. 100-200Hz) using the perf tool (similar in functionality to https://github.com/RRZE-HPC/likwid/wiki/likwid-perfctr#the-timeline-mode but at a higher frequency). Is there a way to do this? If so, what flags do I need to use when recording with perf record and reporting results with perf report? So far, I've tried the following to retrieve the r6d70 performance counter at 5ms intervals during sleep 5 execution: sudo perf record -F200 -e r6d70 -a sleep 5. However, when I use perf report to view the outputted data, I see the following, which isn't really what I want:

Samples: 109  of event 'r6d70', Event count (approx.): 68432
Overhead  Command     Shared Object      Symbol
  33.77%  swapper     [kernel.kallsyms]  [k] update_blocked_averages
  10.30%  node        [kernel.kallsyms]  [k] update_blocked_averages
   9.07%  containerd  [kernel.kallsyms]  [k] update_load_avg
   8.98%  containerd  [kernel.kallsyms]  [k] __switch_to
   8.56%  node        node               [.] Builtins_LdaNamedPropertyHandler
   5.90%  swapper     [kernel.kallsyms]  [k] __sched_text_start
   5.88%  swapper     [kernel.kallsyms]  [k] cpufreq_this_cpu_can_update
   5.81%  nautilus    [kernel.kallsyms]  [k] update_blocked_averages
   4.56%  node        node               [.] v8::platform::tracing::TracingController
   3.82%  swapper     [kernel.kallsyms]  [k] arch_irq_work_raise
   3.20%  containerd  [kernel.kallsyms]  [k] select_task_rq_fair
   0.03%  swapper     [kernel.kallsyms]  [k] acpi_idle_do_entry

When I am using "perf record -F 99 -g -p -- sleep 300" during average load, I am getting the perf.data file without any problem. But when the load is high, it is crashing. The only way to get an ok result is to reduce the frequency to around 30, which is terrible. Ideally, the rate should be high to get better results(about 997 samples/sec). Is there any solution to this. I think that the issue might be related to the Linux kernel space. If so, what would I have to do to solve this issue?

I am trying to understand the --cpu option of perf-stat. In the man pages it says: > Count only on the list of CPUs provided. Multiple CPUs can be provided > as a comma-separated list with no space: 0,1. Ranges of CPUs are > specified with -: 0-2. In per-thread mode, this option is ignored. The > -a option is still necessary to activate system-wide monitoring. Default is to count on all CPUs. Can we control with this option on which CPU the program will be executed, ie. if I put --cpu 0, will the program be executed only on CPU 0? If not, what is the value of this option (since the results are probably going to be different each time we run perf stat with this option)?

I recently installed Perf tool On my platform and wanted to use it for profiling my platform. I started profiling a standalone application. Below is the command I used: perf start ./helloworld Performance counter stats for './helloworld': 4.555957 task-clock # 0.000 CPUs utilized 1 context-switches # 0.219 K/sec 0 cpu-migrations # 0.000 K/sec 124 page-faults # 0.027 M/sec cycles stalled-cycles-frontend stalled-cycles-backend instructions branches branch-misses 60.005519331 seconds time elapsed Now I am not sure how should I interpretate this output.is it as expected? Also, what should I make of `` field here, is there anything I need to enable to support this field before running the command?

I use perf quite often to profile my applications. Recently I got a few new machine with ~750G RAM each. I am trying to profile a process that uses about 400G of ram on one of them. This works fine on all the new machines but one. They were all installed the same way, running ubuntu. I tried both perf as installed through apt-get and compiled my own from git . When running ./perf top -p 14182 I some time manage to get results (after a long wait, and very rarely) but most other times I am getting: The sys_perf_event_open() syscall returned with 3 (No such process) for event (cycles:pp). /bin/dmesg may provide additional information. No CONFIG_PERF_EVENTS=y kernel support configured? For the same pid(!). I get similar problems when trying to record, for instance: time ./perf record -F 111 -a -g -p 14182 -- sleep 3 Warning: PID/TID switch overriding SYSTEMsleep: Terminated Command exited with non-zero status 255 2.66user 91.58system 1:36.68elapsed 97%CPU (0avgtext+0avgdata 4896maxresident)k 0inputs+5248outputs (0major+4847minor)pagefaults 0swaps This command takes way longer than 3 seconds, I cannot ctrl+c it, and killing it requires -9. It emits some data into perf.data but when I try to view it, i get file has no samples. Everything works just fine on other (smaller) processes on this machine, and as I mentioned earlier this works great on the rest of the machines, for processes weighting about 200G. When I run the previous command on the other machines, it also uses much less cpu (33% rather than 97%). I am not sure where else to look, and google has not been very helpful for me. Any ideas or directions? **Edit** Seeing how system was high, I ran this with strace -c, this is what I get: Another machines with a process weighting ~200G: _ strace -c perf record -F 111 -a -g -p 27879 -- sleep 3 Warning: PID/TID switch overriding SYSTEM[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.684 MB perf.data (~29876 samples) ] % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 91.43 2.548664 109 23481 read 2.47 0.068712 11 6457 15 stat 2.42 0.067392 9 7303 write 0.68 0.018960 12 1646 getdents 0.68 0.018841 22 847 1 mmap This machine with a process weighting ~400G: _ strace -c perf record -F 111 -a -g -p 14182 -- sleep 3 /tmpvaknins9@snlp-brevis-3 Warning: PID/TID switch overriding SYSTEMsleep: Terminated sleep: Terminated % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.70 131.243199 472 278178 read 0.16 0.207757 8 25895 write 0.04 0.048962 8 6398 5 stat 0.02 0.031247 14 2217 mmap 0.02 0.026385 9 3091 close It seems lots of time is being spent on reading /proc/tid/maps|status.. Perhaps my process is simply too large? Any clues how I could still get it to record this process? Thanks!

Searching for *what* one can monitor with perf_events on Linux, I cannot find what Kernel PMU event are? Namely, with perf version 3.13.11-ckt39 the perf list shows events like: branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] Overall there are: Tracepoint event Software event Hardware event Hardware cache event Raw hardware event descriptor Hardware breakpoint Kernel PMU event and I would like to understand what they are, where they come from. I have some kind of explanation for all, but Kernel PMU event item. From [perf wiki tutorial](https://perf.wiki.kernel.org/index.php/Tutorial#Events) and [Brendan Gregg's page](http://www.brendangregg.com/perf.html#Events) I get that: * Tracepoints are the clearest -- these are macros on the kernel source, which make a probe point for monitoring, they were introduced with ftrace project and now are used by everybody * Software are kernel's low level counters and some internal data-structures (hence, they are different from tracepoints) * Hardware event are some very basic CPU events, found on *all* architectures and somehow easily accessed by kernel * Hardware cache event are nicknames to Raw hardware event descriptor -- it works as follows as I got it, Raw hardware event descriptor are more (micro?)architecture-specific events than Hardware event, the events come from Processor Monitoring Unit (PMU) or other specific features of a given processor, thus they are available only on some micro-architectures (let's say "architecture" means "x86_64" and all the rest of the implementation details are "micro-architecture"); and they are accessible for instrumentation via these strange descriptors rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] (see 'man perf-list' on how to encode it) -- these descriptors, which events they point to and so on is to be found in processor's manuals ([PMU events in perf wiki](https://perf.wiki.kernel.org/index.php/Tutorial#Hardware_events)) ; but then, when people know that there is some useful event on a given processor they give it a nickname and plug it into linux as Hardware cache event for ease of access -- correct me if I'm wrong (strangely all Hardware cache event are about something-loads or something-misses -- very like the actual processor's cache..) * now, the Hardware breakpoint mem:[:access] [Hardware breakpoint] is a hardware feature, which is probably common to most modern architectures, and works as a breakpoint in a debugger? (probably it is googlable anyway) * finally, Kernel PMU event I don't manage to google on; it also doesn't show up in [the listing of Events in Brendan's perf page](http://www.brendangregg.com/perf.html#Events) , so it's new? Maybe it's just nicknames to hardware events specifically from PMU? (For ease of access it got a separate section in the list of events in addition to the nickname.) In fact, maybe Hardware cache events are nicknames to hardware events from CPU's cache and Kernel PMU event are nicknames to PMU events? (Why not call it Hardware PMU event then?..) It could be just new naming scheme -- the nicknames to hardware events got sectionized? And these events refer to things like cpu/mem-stores/, plus since some linux version events got descriptions in /sys/devices/ and: # find /sys/ -type d -name events /sys/devices/cpu/events /sys/devices/uncore_cbox_0/events /sys/devices/uncore_cbox_1/events /sys/kernel/debug/tracing/events -- debug/tracing is for ftrace and tracepoints, other directories match exactly what perf list shows as Kernel PMU event. Could someone point me to a good explanation/documentation of what Kernel PMU events or /sys/..events/ systems are? Also, is /sys/..events/ some new effort to systemize hardware events or something alike? (Then, Kernel PMU is like "the Performance Monitoring Unit of Kernel".) # PS To give better context, not-privileged run of perf list (tracepoints are not shown, but all 1374 of them are there) with full listings of Kernel PMU events and Hardware cache events and others skipped: $ perf list List of pre-defined events (to be used in -e): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] ... cpu-clock [Software event] task-clock [Software event] ... L1-dcache-load-misses [Hardware cache event] L1-dcache-store-misses [Hardware cache event] L1-dcache-prefetch-misses [Hardware cache event] L1-icache-load-misses [Hardware cache event] LLC-loads [Hardware cache event] LLC-stores [Hardware cache event] LLC-prefetches [Hardware cache event] dTLB-load-misses [Hardware cache event] dTLB-store-misses [Hardware cache event] iTLB-loads [Hardware cache event] iTLB-load-misses [Hardware cache event] branch-loads [Hardware cache event] branch-load-misses [Hardware cache event] branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] branch-misses OR cpu/branch-misses/ [Kernel PMU event] bus-cycles OR cpu/bus-cycles/ [Kernel PMU event] cache-misses OR cpu/cache-misses/ [Kernel PMU event] cache-references OR cpu/cache-references/ [Kernel PMU event] cpu-cycles OR cpu/cpu-cycles/ [Kernel PMU event] instructions OR cpu/instructions/ [Kernel PMU event] mem-loads OR cpu/mem-loads/ [Kernel PMU event] mem-stores OR cpu/mem-stores/ [Kernel PMU event] ref-cycles OR cpu/ref-cycles/ [Kernel PMU event] stalled-cycles-frontend OR cpu/stalled-cycles-frontend/ [Kernel PMU event] uncore_cbox_0/clockticks/ [Kernel PMU event] uncore_cbox_1/clockticks/ [Kernel PMU event] rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] (see 'man perf-list' on how to encode it) mem:[:access] [Hardware breakpoint] [ Tracepoints not available: Permission denied ]

I want to check how much internet use Firefox with a shell script. Here I found good examples to see performances . #!/bin/sh free -m | awk 'NR==2{printf "Memory Usage: %s/%sMB (%.2f%%)\n", $3,$2,$3*100/$2 }' df -h | awk '$NF=="/"{printf "Disk Usage: %d/%dGB (%s)\n", $3,$2,$5}' top -bn1 | grep load | awk '{printf "CPU Load: %.2f\n", $(NF-2)}' might show if it uses the internet or to see a specified application if it uses Intenet?