What are Kernel PMU event-s in perf_events list?

Searching for *what* one can monitor with perf_events on Linux, I cannot find what Kernel PMU event are? Namely, with perf version 3.13.11-ckt39 the perf list shows events like: branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] Overall there are: Tracepoint event Software event Hardware event Hardware cache event Raw hardware event descriptor Hardware breakpoint Kernel PMU event and I would like to understand what they are, where they come from. I have some kind of explanation for all, but Kernel PMU event item. From [perf wiki tutorial](https://perf.wiki.kernel.org/index.php/Tutorial#Events) and [Brendan Gregg's page](http://www.brendangregg.com/perf.html#Events) I get that: * Tracepoints are the clearest -- these are macros on the kernel source, which make a probe point for monitoring, they were introduced with ftrace project and now are used by everybody * Software are kernel's low level counters and some internal data-structures (hence, they are different from tracepoints) * Hardware event are some very basic CPU events, found on *all* architectures and somehow easily accessed by kernel * Hardware cache event are nicknames to Raw hardware event descriptor -- it works as follows as I got it, Raw hardware event descriptor are more (micro?)architecture-specific events than Hardware event, the events come from Processor Monitoring Unit (PMU) or other specific features of a given processor, thus they are available only on some micro-architectures (let's say "architecture" means "x86_64" and all the rest of the implementation details are "micro-architecture"); and they are accessible for instrumentation via these strange descriptors rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] (see 'man perf-list' on how to encode it) -- these descriptors, which events they point to and so on is to be found in processor's manuals ([PMU events in perf wiki](https://perf.wiki.kernel.org/index.php/Tutorial#Hardware_events)) ; but then, when people know that there is some useful event on a given processor they give it a nickname and plug it into linux as Hardware cache event for ease of access -- correct me if I'm wrong (strangely all Hardware cache event are about something-loads or something-misses -- very like the actual processor's cache..) * now, the Hardware breakpoint mem:[:access] [Hardware breakpoint] is a hardware feature, which is probably common to most modern architectures, and works as a breakpoint in a debugger? (probably it is googlable anyway) * finally, Kernel PMU event I don't manage to google on; it also doesn't show up in [the listing of Events in Brendan's perf page](http://www.brendangregg.com/perf.html#Events) , so it's new? Maybe it's just nicknames to hardware events specifically from PMU? (For ease of access it got a separate section in the list of events in addition to the nickname.) In fact, maybe Hardware cache events are nicknames to hardware events from CPU's cache and Kernel PMU event are nicknames to PMU events? (Why not call it Hardware PMU event then?..) It could be just new naming scheme -- the nicknames to hardware events got sectionized? And these events refer to things like cpu/mem-stores/, plus since some linux version events got descriptions in /sys/devices/ and: # find /sys/ -type d -name events /sys/devices/cpu/events /sys/devices/uncore_cbox_0/events /sys/devices/uncore_cbox_1/events /sys/kernel/debug/tracing/events -- debug/tracing is for ftrace and tracepoints, other directories match exactly what perf list shows as Kernel PMU event. Could someone point me to a good explanation/documentation of what Kernel PMU events or /sys/..events/ systems are? Also, is /sys/..events/ some new effort to systemize hardware events or something alike? (Then, Kernel PMU is like "the Performance Monitoring Unit of Kernel".) # PS To give better context, not-privileged run of perf list (tracepoints are not shown, but all 1374 of them are there) with full listings of Kernel PMU events and Hardware cache events and others skipped: $ perf list List of pre-defined events (to be used in -e): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] ... cpu-clock [Software event] task-clock [Software event] ... L1-dcache-load-misses [Hardware cache event] L1-dcache-store-misses [Hardware cache event] L1-dcache-prefetch-misses [Hardware cache event] L1-icache-load-misses [Hardware cache event] LLC-loads [Hardware cache event] LLC-stores [Hardware cache event] LLC-prefetches [Hardware cache event] dTLB-load-misses [Hardware cache event] dTLB-store-misses [Hardware cache event] iTLB-loads [Hardware cache event] iTLB-load-misses [Hardware cache event] branch-loads [Hardware cache event] branch-load-misses [Hardware cache event] branch-instructions OR cpu/branch-instructions/ [Kernel PMU event] branch-misses OR cpu/branch-misses/ [Kernel PMU event] bus-cycles OR cpu/bus-cycles/ [Kernel PMU event] cache-misses OR cpu/cache-misses/ [Kernel PMU event] cache-references OR cpu/cache-references/ [Kernel PMU event] cpu-cycles OR cpu/cpu-cycles/ [Kernel PMU event] instructions OR cpu/instructions/ [Kernel PMU event] mem-loads OR cpu/mem-loads/ [Kernel PMU event] mem-stores OR cpu/mem-stores/ [Kernel PMU event] ref-cycles OR cpu/ref-cycles/ [Kernel PMU event] stalled-cycles-frontend OR cpu/stalled-cycles-frontend/ [Kernel PMU event] uncore_cbox_0/clockticks/ [Kernel PMU event] uncore_cbox_1/clockticks/ [Kernel PMU event] rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] (see 'man perf-list' on how to encode it) mem:[:access] [Hardware breakpoint] [ Tracepoints not available: Permission denied ]