Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
6
votes
1
answers
2347
views
What are "self" and "master" options for in "bridge vlan add"?
I'm reading "man bridge" and it says something like: bridge vlan { add | del } dev DEV vid VID [ pvid ] [ untagged ] [ self ] [ master ] What are "self" and "master" options for? man says: self the vlan is configured on the specified physical device. Required if the device is the bridge device. mast...
I'm reading "man bridge" and it says something like:
bridge vlan { add | del } dev DEV vid VID [ pvid ] [ untagged ] [ self ] [ master ]
What are "self" and "master" options for?
man says:
self the vlan is configured on the specified physical device. Required if the device is the bridge device.
master the vlan is configured on the software bridge (default).
Can anyone elaborate?
As I understand "self" is for virtual interfaces like "br0.10" etc to let the bridge (br0) know that recived frames can be for it. Right?
2c9
(61 rep)
Nov 28, 2018, 11:53 AM
• Last activity: Jul 24, 2025, 01:03 PM
0
votes
2
answers
557
views
systemd-networkd bridge does not work with VLAN
I am trying to configure one bridge per VLAN (without VLAN filtering) with ```systemd-networkd``` for easy configuration of libvirt VMs on Debian 12. This does work as expected without VLAN and it does work with VLAN with some manual help (`ip link set master ...`). The problem symptom is that ```sy...
I am trying to configure one bridge per VLAN (without VLAN filtering) with
-networkd
for easy configuration of libvirt VMs on Debian 12.
This does work as expected without VLAN and it does work with VLAN with some manual help (ip link set master ...
). The problem symptom is that -networkd
does create the bridge and VLAN interfaces but does not connect them:
2: onbunten: mtu 1500 qdisc mq master brlan state UP group default qlen 1000
link/ether 04:92:26:b7:a9:9f brd ff:ff:ff:ff:ff:ff
4: brlan: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether c6:50:f5:f2:22:8c brd ff:ff:ff:ff:ff:ff
inet 192.168.2.222/24 brd 192.168.2.255 scope global brlan
valid_lft forever preferred_lft forever
5: brvlan2: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 46:ba:f7:a8:3d:d7 brd ff:ff:ff:ff:ff:ff
6: brvlan3: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 26:96:9b:cc:69:af brd ff:ff:ff:ff:ff:ff
7: onbunten.4@onbunten: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 04:92:26:b7:a9:9f brd ff:ff:ff:ff:ff:ff
8: onbunten.2@onbunten: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 04:92:26:b7:a9:9f brd ff:ff:ff:ff:ff:ff
9: onbunten3@onbunten: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 04:92:26:b7:a9:9f brd ff:ff:ff:ff:ff:ff
The .2
/
pattern difference is intentional; for testing whether that made any difference.
I have found descriptions on the Internet which seem to state that what I did is the solution. I also found the statement
> This configuration is deprecated and no longer supported
on this site (the answer ); unfortunately without a clear explanation what exactly is supposed to be deprecated (I did not find any such hints in the Systemd man pages).
I have been running systemd-networkd
in debug mode (SYSTEMD_LOG_LEVEL=debug
) but among the 500 lines of logging I do not notice any which are related to the VLAN and bridge interfaces simultaneously. No error messages.
# the config files
## this is what does work
==> 31-brlan.netdev 32-brvlan2.netdev 33-brvlan3.netdev 41-onbunten-vlans.network 61-brlan-onbunten.network 71-brlan.network 62-brvlan2-onbunten2.network 63-brvlan3-onbunten3.network 72-brvlan2.network 73-brvlan3.network <==
[Match]
Name=brvlan3
[Network]
DHCP=no
LinkLocalAddressing=no
Gateway=192.168.116.1
# NTP=
ConfigureWithoutCarrier=yes
IgnoreCarrierLoss=yes
[Address]
Address=192.168.116.222/24
DuplicateAddressDetection=ipv4
Scope=global
Hauke Laging
(93688 rep)
Dec 26, 2023, 07:19 PM
• Last activity: Jun 23, 2025, 07:38 PM
2
votes
1
answers
60
views
How to mark 802.1Q ethernet frame with PCP bits according to encapsulated IP header IP Precedence bits
I would like the IP header IP Precedence bits to be copied into 802.1Q PCP bits for outgoing traffic sourced from the host in question. Specifically for iperf3 and ping utilities. I have failed to set PCP bits for pings. OS Fedora release 38, "Server Edition", NetworkManager, eno2 ethernet eno2 eno2...
I would like the IP header IP Precedence bits to be copied into 802.1Q PCP bits for outgoing traffic sourced from the host in question. Specifically for iperf3 and ping utilities.
I have failed to set PCP bits for pings.
OS Fedora release 38, "Server Edition", NetworkManager,
eno2 ethernet eno2
eno2.814 vlan eno2.814
ip -d link show eno2
3: eno2: mtu 1600 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether ac:16:2d:72:3f:fd brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 60 maxmtu 9000 addrgenmode none numtxqueues 5 numrxqueues 5 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 parentbus pci parentdev 0000:03:00.1
altname enp3s0f1
ip -d link show eno2.814
10: eno2.814@eno2: mtu 1600 qdisc pfifo state UP mode DEFAULT group default qlen 1000
link/ether ac:16:2d:72:3f:fd brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 0 maxmtu 65535
vlan protocol 802.1Q id 814
ingress-qos-map { 1:1 2:2 3:3 4:4 5:5 6:6 7:7 }
egress-qos-map { 1:1 2:2 3:3 4:4 5:5 6:6 7:7 } addrgenmode none numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536
cat /proc/net/vlan/eno2.814
eno2.814 VID: 814 REORDER_HDR: 0 dev->priv_flags: 81021
total frames received 294
total bytes received 21846
Broadcast/Multicast Rcvd 0
total frames transmitted 271
total bytes transmitted 23846
Device: eno2
INGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
Ping command to send 8 requests:
for pcp in 0x00 0x20 0x40 0x60 0x80 0xA0 0xC0 0xE0; do ping 192.168.22.3 -w2 -c1 -Q $pcp ; done
Sent packets are captured on outgoing interface with "tshark -i eno2 -f 'icmp and dst host 192.168.22.3' -V".
grep for L2 and L3 CoS fields in headres shows intended DSCP values there but '000' PCP "Priority" values:
000. .... .... .... = Priority: Best Effort (default) (0)
0000 00.. = Differentiated Services Codepoint: Default (0)
000. .... .... .... = Priority: Best Effort (default) (0)
0010 00.. = Differentiated Services Codepoint: Class Selector 1 (8)
000. .... .... .... = Priority: Best Effort (default) (0)
0100 00.. = Differentiated Services Codepoint: Class Selector 2 (16)
000. .... .... .... = Priority: Best Effort (default) (0)
0110 00.. = Differentiated Services Codepoint: Class Selector 3 (24)
000. .... .... .... = Priority: Best Effort (default) (0)
1000 00.. = Differentiated Services Codepoint: Class Selector 4 (32)
000. .... .... .... = Priority: Best Effort (default) (0)
1010 00.. = Differentiated Services Codepoint: Class Selector 5 (40)
000. .... .... .... = Priority: Best Effort (default) (0)
1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
000. .... .... .... = Priority: Best Effort (default) (0)
1110 00.. = Differentiated Services Codepoint: Class Selector 7 (56)
What I've tried that haven't helped:
swithing off reorder_hdr
ip link set eno2.814 type vlan reorder_hdr off
Setting vlan egress-qos-map to map kernel values(wich IMHO should be already set equal to IP precedence values of the ping utility) to PCP:
ip link set eno2.814 type vlan egress-qos-map 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
Setting outgoing interface qdisc. I've created eno2.814 on eno2 with nmtui and no qdisc was set by default. So I've thought it could be the problem and tried to set the queues and qdisc(s) manually
ip link set eno2 numtxqueues 8 numrxqueues 8
tc qdisc add dev eno2.814 root handle 1: mq -- RTNETLINK answers: Operation not supported
tc qdisc add dev eno2.814 root handle 1: mqprio -- Error: Specified qdisc kind is unknown.
tc qdisc add dev eno2.814 root handle 1: multiq -- Error: Specified qdisc kind is unknown.
tc qdisc delete dev eno2.814 root
tc qdisc add dev eno2.814 root handle 1: pfifo_fast
sudo systemctl restart NetworkManager does not seem to help either.
What I don't get:
I assume that ping -Q set kernel SO_PRIORITY for a packet. Does it?
Can vlan and parent qdiscs difference have any influence?
Why "/proc/net/vlan/eno2.814" EGRESS priority mappings shows mapping 0:0 but "ip -d link show eno2.814 egress-qos-map" does not?
Do I need to get into hw queus presented to kernel or I need just one hw or some default queues if I just want packet marking, not specific queue handling?
What is wrong with my config?
off-on
(61 rep)
Jun 11, 2025, 11:54 AM
• Last activity: Jun 11, 2025, 08:39 PM
5
votes
2
answers
2960
views
Backup and restore of Centos network interfaces
I have a server running Centos 7 which needs to be rebooted to upgrade some software. Some of the physical NICs have around 5-10 VLAN interfaces each. They're subject to change on a weekly/monthly basis so storing the details in `/etc/sysconfig/network-scripts` to persist across reboots isn't practi...
I have a server running Centos 7 which needs to be rebooted to upgrade some software.
Some of the physical NICs have around 5-10 VLAN interfaces each. They're subject to change on a weekly/monthly basis so storing the details in
/etc/sysconfig/network-scripts
to persist across reboots isn't practical.
Is there an simple way to take a snapshot of the current networking stack and restore after the reboot? Similar to the way you can save/restore iptables rules?
I've found several references to the system-config-network-cmd
but I'm wary of using this tool in the event it overwrites the static configs for the physical interfaces we do have in /etc/sysconfig/network-scripts
Thanks!
popcornuk
(97 rep)
Mar 28, 2018, 01:44 PM
• Last activity: May 18, 2025, 04:06 AM
4
votes
1
answers
3413
views
Bridge interfaces with Vlan on Linux based access point
I have a Linux based wireless access point. It has **eth0** interface for ethernet, **ath0** for wireless and **br-wan** as a bridge between the two. bridge name bridge id STP enabled interfaces br-wan 8000.001567000041 no eth0 ath0 **Q:** How can I add vlan tagging on the wireless side so that all...
I have a Linux based wireless access point. It has **eth0** interface for ethernet, **ath0** for wireless and **br-wan** as a bridge between the two.
bridge name bridge id STP enabled interfaces
br-wan 8000.001567000041 no eth0
ath0
**Q:** How can I add vlan tagging on the wireless side so that all the outgoing traffic will be tagged, and only incoming tagged traffic will be accepted?
I'v tried creating a new vlan using **vconfig**
vconfig add ath0 15
ip link set ath0.15 up
I then used a computer with a traffic generator software to send frames through the Ethernet port of the access point. Using **tcpdump** -e I can see them pass through both ath0 and ath0.15, however no tagging is added whatsoever.
DoctorBurp
(41 rep)
Sep 3, 2013, 01:48 PM
• Last activity: Apr 18, 2025, 07:04 PM
1
votes
0
answers
67
views
Confusing about vlan on bridge and filtering
I have created on openwrt router an interface tagged with vlan id 1 On one server i have some lxc containers and I want some of them using this vlan (id 1), I have created the bridge ip link add name br1 type bridge ip link set br1 type bridge vlan_filtering 1 ip link set enp7s0.1 up ip link set br1...
I have created on openwrt router an interface tagged with vlan id 1
On one server i have some lxc containers and I want some of them using this vlan (id 1), I have created the bridge
ip link add name br1 type bridge
ip link set br1 type bridge vlan_filtering 1
ip link set enp7s0.1 up
ip link set br1 up
ip addr add 192.168.179.2 dev br1
add the vlan tag
bridge vlan del dev enp7s0.1 vid 1
bridge vlan add dev enp7s0.1 vid 1
bridge vlan del dev br1 vid 1 self
bridge vlan add dev br1 vid 1 self
bridge vlan add dev veth1001_17Yt vid 1
This is the result
bridge vlan sh
port vlan-id
enp7s0 1 PVID Egress Untagged
wlan0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
veth1001_LoZK 1 PVID Egress Untagged
veth1001_4Kss 1 PVID Egress Untagged
veth1001_ZF7k 1 PVID Egress Untagged
veth1001_FuzI 1 PVID Egress Untagged
veth1001_1C5W 1 PVID Egress Untagged
veth1001_W2FM 1 PVID Egress Untagged
veth1001_YsU5 1 PVID Egress Untagged
veth1001_823Q 1 PVID Egress Untagged
veth1001_Uf5S 1 PVID Egress Untagged
veth1001_R5zN 1 PVID Egress Untagged
veth1001_9gyW 1 PVID Egress Untagged
lxcbr0 1 PVID Egress Untagged
enp7s0.1 1
br1 1
veth1001_17Yt 1
From pc using the bridge I try to ping ip of lxc container (which use vlan tag 1)
ping 192.168.179.3
PING 192.168.179.3 (192.168.179.3) 56(84) bytes of data.
It doesn't work!
But if I disable vlan filtering...
ip link set dev br1 type bridge vlan_filtering 0
works
ping 192.168.179.3
PING 192.168.179.3 (192.168.179.3) 56(84) bytes of data.
64 bytes from 192.168.179.3: icmp_seq=1 ttl=63 time=8.75 ms
64 bytes from 192.168.179.3: icmp_seq=2 ttl=63 time=6.88 ms
but seems vlan tag are not used
tcpdump -i br1 -e vlan
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
What I miss?
elbarna
(13690 rep)
Feb 2, 2025, 02:05 AM
1
votes
1
answers
84
views
Linux server VLAN tags using SNMP
I have a number of Linux servers than I need to monitor. Each have configured a number of VLAN interfaces. Is there any way to read VLAN tags for interfaces for a Linux server using SNMP? Anything I need to install? Which MIB applies? I have been unable to find any info on this online.
I have a number of Linux servers than I need to monitor. Each have configured a number of VLAN interfaces. Is there any way to read VLAN tags for interfaces for a Linux server using SNMP? Anything I need to install? Which MIB applies?
I have been unable to find any info on this online.
Astronaut
(11 rep)
Jan 28, 2025, 12:41 PM
• Last activity: Jan 30, 2025, 07:30 AM
1
votes
1
answers
890
views
Does libvirt support vlan aware bridges?
I am using libvirt/kvm in Debian 10 with a vlan aware bridge device, and I am stuck trying to configure the network interfaces for my VMs. Does libvirt support vlans? When I do a search the only references I find have to do with OVS.
I am using libvirt/kvm in Debian 10 with a vlan aware bridge device, and I am stuck trying to configure the network interfaces for my VMs. Does libvirt support vlans? When I do a search the only references I find have to do with OVS.
Chris_F
(247 rep)
Sep 4, 2020, 07:43 PM
• Last activity: Jan 23, 2025, 08:58 AM
0
votes
0
answers
214
views
Configure hostapd with sae_password and vlanid
According [to the hostapd documentation](https://w1.fi/cgit/hostap/plain/hostapd/hostapd.conf), when you use WPA3, the hostapd AP can bind to a vlan interface based on the received passphrase. > an optional VLAN ID specification can be used to bind the station > to the specified VLAN whenever the sp...
According [to the hostapd documentation](https://w1.fi/cgit/hostap/plain/hostapd/hostapd.conf) , when you use WPA3, the hostapd AP can bind to a vlan interface based on the received passphrase.
> an optional VLAN ID specification can be used to bind the station
> to the specified VLAN whenever the specific SAE password entry is used.
> [...]
>
> #sae_password=example secret|vlanid=3|id=pw identifier
I have a minimal working hostapd configuration for WPA3 but it does not work if I try to bind to a vlan :
interface=wlan0
ssid2="Test-44"
country_code=
# Advertises the country_code and the set of allowed
# channels and transmit power levels based on the regulatory limits.
ieee80211d=1
logger_syslog=-1
logger_syslog_level=0
logger_stdout=-1
logger_stdout_level=1
driver=nl80211
ieee80211n=1
ieee80211ac=1
# Support for 802.11a(c|x)
hw_mode=g
# Required for 802.11n/802.11ac/802.11ax
# https://www.wi-fi.org/discover-wi-fi/wi-fi-certified-wmm-programs
wmm_enabled=1
# Enable WPA. Needed for WPA3
wpa=2
# Set of accepted key management algorithms (SAE = WPA3 / WPA-PSK = WPA2)
wpa_key_mgmt=SAE
rsn_pairwise=CCMP
# ieee80211w: Enable management frame protection (MFP)
ieee80211w=2
sae_password=password|vlanid=100
root@router ~# ip l
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wan0: mtu 1500 qdisc mq state UP qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: lan: mtu 1500 qdisc mq state DOWN qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
4: wlan0: mtu 1500 qdisc noqueue state UP qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
9: vlan100@wlan0: mtu 1500 qdisc noqueue state UP qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
Hostapd seems to correctly see the VLAN interface :
RTM_NEWLINK: ifi_index=4 ifname=wlan0 operstate=2 linkmode=0 ifi_family=0 ifi_flags=0x1002 ()
nl80211: Ignore interface down event since interface wlan0 is up
RTM_NEWLINK: ifi_index=9 ifname=vlan100 operstate=2 linkmode=0 ifi_family=0 ifi_flags=0x1002 ()
RTM_NEWLINK: ifi_index=4 ifname=wlan0 operstate=2 linkmode=0 ifi_family=0 ifi_flags=0x1003 ([UP])
RTM_NEWLINK: ifi_index=9 ifname=vlan100 operstate=3 linkmode=0 ifi_family=0 ifi_flags=0x1003 ([UP])
But when I try to connect to the hotspot with a device, hostapd refuses with this debug log:
SAE: Assign STA xx:xx:xx:xx:xx:xx to VLAN ID 100
Invalid VLAN ID 100 in sae_password
Any ideas?
Rand0mMan
(101 rep)
Oct 31, 2024, 01:01 AM
• Last activity: Oct 31, 2024, 01:15 AM
0
votes
0
answers
106
views
VLAN over VxLAN tunnel problem
I m trying to extend a layer 2 network (Vlans) over a layer 3 network using vxlan tunnels ... i set up a lab where i have two VMs where, - I created a vxlan tunnel between the 2 main interfaces of the VMs - I created 2 vlan sub-interfaces under the second interface for each machine - I linked each v...
I m trying to extend a layer 2 network (Vlans) over a layer 3 network using vxlan tunnels ... i set up a lab where i have two VMs where,
- I created a vxlan tunnel between the 2 main interfaces of the VMs
- I created 2 vlan sub-interfaces under the second interface for each machine
- I linked each vlan sub interface with the vxlan sub interface in separate bridges for each machine
- I assigned an ip to every bridge (10.1.100.1/24 , 10.1.100.2/24 and 10.1.100.3/24 , 10.1.100.4/24)
===> now when itry to ping from one bridge to another in (same vlan tag) it doesn t work
[root@Asguard ~]# ping 192.168.100.1 -I 192.168.100.3
PING 192.168.100.1 (192.168.100.1) from 192.168.100.3 : 56(84) bytes of data.
From 192.168.100.3 icmp_seq=10 Destination Host Unreachable
ping: sendmsg: No route to host
From 192.168.100.3 icmp_seq=11 Destination Host Unreachable
From 192.168.100.3 icmp_seq=12 Destination Host Unreachable
From 192.168.100.3 icmp_seq=14 Destination Host Unreachable
From 192.168.100.3 icmp_seq=15 Destination Host Unreachable
From 192.168.100.3 icmp_seq=16 Destination Host Unreachable
From 192.168.100.3 icmp_seq=17 Destination Host Unreachable
this is the script i run in each VM
VM1 :
#!/bin/bash
# Bridge and interface setup
ip link add br10 type bridge
ip link add br20 type bridge
ip link set br10 up
ip link set br20 up
# VLAN 10 on bridge br10
ip link add link enp0s9 name enp0s9.10 type vlan id 10
ip link set enp0s9.10 master br10
ip link set enp0s9.10 up
# VLAN 20 on bridge br20
ip link add link enp0s9 name enp0s9.20 type vlan id 20
ip link set enp0s9.20 master br20
ip link set enp0s9.20 up
# VXLAN on both bridges
ip link set vxlan1000 master br10
ip link set vxlan1000 up
#ip link add vxlan1000_2 type vxlan id 1000 dev enp0s3 remote 10.1.25.235 dstport 4789
ip link set vxlan1000 master br20
ip link set vxlan1000 up
ip addr add 192.168.100.1/24 dev br10
ip addr add 192.168.100.2/24 dev br20
VM2
#!/bin/bash
# Bridge and interface setup
ip link add br11 type bridge
ip link add br22 type bridge
ip link set br11 up
ip link set br22 up
# VLAN 10 on bridge br10
ip link add link enp0s8 name enp0s8.10 type vlan id 10
ip link set enp0s8.10 master br11
ip link set enp0s8.10 up
# VLAN 20 on bridge br20
ip link add link enp0s8 name enp0s8.20 type vlan id 20
ip link set enp0s8.20 master br22
ip link set enp0s8.20 up
# VXLAN on both bridges
ip link add vxlan1001 type vxlan id 1000 dev enp0s3 remote 10.1.25.31 dstport 4789
ip link set vxlan1001 master br11
ip link set vxlan1001 up
#ip link add vxlan1000_2 type vxlan id 1000 dev enp0s3 remote 10.1.25.235 dstport 4789
ip link set vxlan1001 master br22
ip link set vxlan1001 up
ip addr add 192.168.100.3/24 dev br11
ip addr add 192.168.100.4/24 dev br22
!!!
i want to know how linux handle the tagging and encapsulation to make them work together to make the vxlan extention
GStaim
(1 rep)
Oct 25, 2024, 09:57 AM
• Last activity: Oct 25, 2024, 10:56 AM
0
votes
1
answers
132
views
Access port for tagged VLAN just doesn't work for vlan aware bridge
I am on my whits end. I have a config which identically works on a different machine (different board, CPU, network adapter though): auto eno1 iface eno1 inet manual auto vmbr0 iface vmbr0 inet manual bridge-ports eno1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 1-4096 bridge-pvid 1...
I am on my whits end. I have a config which identically works on a different machine (different board, CPU, network adapter though):
auto eno1
iface eno1 inet manual
auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1-4096
bridge-pvid 1
auto vmbr0.1
iface vmbr0.1 inet static
address 10.227.1.21/24
gateway 10.227.1.254
iface vmbr0.4 inet static
address 10.227.4.67/24
Network interface eno1 is connected to a managed switch with trunk access port, VLAN4 tagged and VLAN1 untagged. In above configuration, **vmbr0.1 works and vmbr0.4 does not work**: I cannot reach any other host on the same VALN4. If I run tcpdump on vmbr0.4, I can only see ARP requests being sent out but nothing received.
Now, on this, I could expect all VLANs so something seems wrong:
# bridge vlan show
port vlan-id
eno1 1 PVID Egress Untagged
vmbr0 1
4
The exact identical config works on another PC with the same managed switch port (but using enp3s0 and enp4s0 as bridge-ports instead).
VLAN per se are working: If I remove the VLAN aware bridge but instead configure eno1.1 and eno1.4, everything works as expected too!
In case it matters, it's a I219-LM network adapter in a HP Elitedesk 800 G mini PC. It uses e1000e driver.
What the heck can possibly be wrong here?
divB
(218 rep)
Sep 27, 2024, 08:38 PM
• Last activity: Sep 27, 2024, 10:20 PM
0
votes
1
answers
726
views
how to create a persistent VLAN-aware bridge on RHEL based system (i.e. with NetworkManager)
On RHEL based systems network is supposed to be managed by NetworkManager. I would like to build a persistent configuration of a VLAN-aware bridge that survives reboots. Building on top of the excellent [answer by A.B.][1], I have created a network configuration for a virtual machine running Alma Li...
On RHEL based systems network is supposed to be managed by NetworkManager.
I would like to build a persistent configuration of a VLAN-aware bridge that survives reboots.
Building on top of the excellent answer by A.B. , I have created a network configuration for a virtual machine running Alma Linux 9.4 that receives VLAN 1 tagged traffic and is reachable from management VLAN.
But how to make this persistent?
Can it be done using standard network management tool, NetworkManager?
# cat mkbr2.sh
ip link add name bridge0 type bridge vlan_filtering 1 vlan_default_pvid 0
ip link set dev ens192 master bridge0
ip link set bridge0 up
bridge vlan add vid 1 dev bridge0 pvid untagged self
bridge vlan add vid 2-4094 dev ens192
bridge vlan add vid 1 dev ens192 pvid
ip addr add 10.200.200.106/24 dev bridge0
ip route add default via 10.200.200.10
# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 00:0c:29:44:89:b3 brd ff:ff:ff:ff:ff:ff
altname enp11s0
6: bridge0: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:0c:29:44:89:b3 brd ff:ff:ff:ff:ff:ff
inet 10.200.200.106/24 scope global bridge0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe44:89b3/64 scope link
valid_lft forever preferred_lft forever
# bridge -compressvlans vlan show
port vlan-id
ens192 1 PVID
2-4094
bridge0 1 PVID Egress Untagged
shpokas
(163 rep)
Sep 17, 2024, 02:23 PM
• Last activity: Sep 17, 2024, 10:44 PM
0
votes
2
answers
249
views
No internet access with IPv6 prefix delegation to vlan with systemd-networkd
I have a router that gets a /48 prefix from my ISP, and had prefix delegation enabled. I run a (debain)server on the router's main network. I've set up a vlan for my IoT devices on my server, for ipv4 I have set up a NAT to connect the device on the vlan to the internet. I've managed to configure sy...
I have a router that gets a /48 prefix from my ISP, and had prefix delegation enabled.
I run a (debain)server on the router's main network.
I've set up a vlan for my IoT devices on my server, for ipv4 I have set up a NAT to connect the device on the vlan to the internet.
I've managed to configure systemd-networkd to request a prefix from the router and apply it on the
iot
vlan interface. Systemd-networkd also serves as router advertisement for the clients on the vlan. When connecting a device to the iot-vlan it receives a ipv6 address within the prefix.
So far so good.
However, the addresses within the prefix are not routed to the internet.
I seem to be missing the part where the server 'knows' that traffic from the prefix :C001:
needs to be forwarded to the router (over eth0).
Pinging google.com from the main interface of my server:
# ping6 2a00:1450:400e:80f::200e -I eth0
PING 2a00:1450:400e:80f::200e(2a00:1450:400e:80f::200e) from 2a02:a46e:53ea:0:54ce:59ff:fec6:b62d eth0: 56 data bytes
64 bytes from 2a00:1450:400e:80f::200e: icmp_seq=1 ttl=119 time=7.13 ms
Pinging from the iot-vlan interface:
# ping6 2a00:1450:400e:80f::200e -I iot
PING 2a00:1450:400e:80f::200e(2a00:1450:400e:80f::200e) from 2a02:a46e:53ea:c001:54ce:59ff:fec6:b62d iot: 56 data bytes
From 2a02:a46e:53ea:c001:54ce:59ff:fec6:b62d icmp_seq=1 Destination unreachable: Address unreachable
edit: Pinging from the *address* of the iot interface works (tnx @u1686_grawity)
$ ping6 google.com -I "2a02:a46e:53ea:c000:54ce:59ff:fec6:b62d"
PING google.com(ams17s12-in-x0e.1e100.net (2a00:1450:400e:810::200e)) from 2a02:a46e:53ea:c000:54ce:59ff:fec6:b62d : 56 data bytes
64 bytes from ams17s12-in-x0e.1e100.net (2a00:1450:400e:810::200e): icmp_seq=1 ttl=60 time=7.64 ms
Pinging from a device in the iot-vlan does not work.
$ ping6 google.com
PING google.com (2a00:1450:400e:803::200e) 56 data bytes
--- google.com ping statistics ---
236 packets transmitted, 0 received, 100% packet loss, time 240617ms
ip address of device on iot-vlan:
$ ip -6 addr show dev wlp1s0
3: wlp1s0: mtu 1500 qdisc noqueue state UP group default qlen 1000
inet6 2a02:a46e:53ea:c001:dc81:6cfc:2115:899/64 scope global temporary dynamic
valid_lft 3576sec preferred_lft 1776sec
inet6 2a02:a46e:53ea:c001:4f32:ea73:f336:3907/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3576sec preferred_lft 1776sec
inet6 fe80::aff0:9c03:3ce:7601/64 scope link noprefixroute
valid_lft forever preferred_lft forever
route of device on iot-vlan
$ ip -6 route
2a02:a46e:53ea:c001::/64 dev wlp1s0 proto ra metric 600 pref medium
fe80::/64 dev veth0ad8518 proto kernel metric 256 pref medium
fe80::/64 dev docker0 proto kernel metric 256 pref medium
fe80::/64 dev veth998c5e2 proto kernel metric 256 pref medium
fe80::/64 dev wlp1s0 proto kernel metric 1024 pref medium
default via fe80::54ce:59ff:fec6:b62d dev wlp1s0 proto ra metric 20600 pref medium
The systemd .network file for the vlan-interface:
[Match]
Name=iot
Type=vlan
[Network]
DHCPPrefixDelegation=yes
IPv6DuplicateAddressDetection=1
IPv6SendRA=yes
LinkLocalAddressing=ipv6
[Address]
Address=192.168.100.1/24
[DHCPPrefixDelegation]
UplinkInterface=:auto
SubnetId=1
Announce=yes
#[Route]
#Gateway=::
#Table=local
#This does not seem to do anything
Edit:
tcpdump shows my ping request entering the server in interface iot
:
#tcpdump -eni iot ip6
15:49:12.290030 c0:a5:e8:46:79:31 > 56:ce:59:c6:b6:2d, ethertype IPv6 (0x86dd), length 118: 2a02:a46e:53ea:c001:dc81:6cfc:2115:899 > 2a00:1450:400e:80f::200e: ICMP6, echo request, id 30134, seq 1, length 64
but not being forwarded to eth0
(only the same packet WITH vlan tag):
#tcpdump -eni eth0 ip6
15:50:55.083174 c0:a5:e8:46:79:31 > 56:ce:59:c6:b6:2d, ethertype 802.1Q (0x8100), length 122: vlan 2, p 0, ethertype IPv6 (0x86dd), 2a02:a46e:53ea:c001:dc81:6cfc:2115:899 > 2a00:1450:400e:80f::200e: ICMP6, echo request, id 30653, seq 1, length 64
Pelle
(401 rep)
Aug 16, 2024, 09:49 AM
• Last activity: Aug 19, 2024, 11:32 AM
1
votes
1
answers
513
views
How do I forward VLAN-tagged Ethernet frames through a Linux bridge between two physical Ethernet interfaces?
## Background I am currently working on a device that uses a Beaglebone Black as the base board, and has a 3rd-party MAC/PHY attached to the breakout pins. ## Problem The desire is to forward all Ethernet packets received on one interface to the other interface, in both directions. Configuring a bri...
## Background
I am currently working on a device that uses a Beaglebone Black as the base board, and has a 3rd-party MAC/PHY attached to the breakout pins.
## Problem
The desire is to forward all Ethernet packets received on one interface to the other interface, in both directions. Configuring a bridge between the two interfaces is simple enough:
ip link add name br0 type bridge
ip link set dev br0 up
ip link set eth0 up
ip link set eth1 up
ip link set dev eth0 master br0
ip link set dev eth1 master br0
This works great until I start transmitting VLAN-tagged frames. If I tag the frame with VLAN ID 0 or 1, the frames are forwarded. Unfortunately, the VLAN tag is dropped before being transmitted on the other interface. If I tag with any other VLAN ID, the frame is not forwarded.
## Solution Attempt 1
The first thing I tried was to use ebtables:
ebtables -A FORWARD -i br0 --vlan-id 2 -j ACCEPT
This didn't have any effect. Even if it did, it wouldn't scale very well -- I'd have to add an entry for all VLAN IDs I wish to use.
## Solution Attempt 2
The second thing I tried was bridging sub-interfaces:
ip link add name br0.2 type bridge
ip link set dev br0.2 up
vconfig add eth0 1
vconfig add eth1 2
ip link set dev eth0.2 up
ip link set dev eth1.2 up
ip link set dev eth0.2 master br0.2
ip link set dev eth1.2 master br0.2
This works partially, but suffers from the same scaling problem as solution 1. I say it works partially, because the VLAN tag data is partially lost. Using tcpdump --x -i ethX
, I observed the following when transmitting VLAN-tagged frames to the Beaglebone's onboard Ethernet interface (eth0):
1. When the packet arrives on eth0
, the entire VLAN tag is present.
2. When the packet arrives on eth0.2
, the VLAN tag has been stripped.
3. When the packet arrives on br0.2
, the VLAN tag is still not present.
4. When the packet arrives on eth1.2
, the VLAN tag is still not present.
5. When the packet arrives on eth1
, a VLAN tag with the correct ID is present, but the priority has been lost.
**Is there any way to preserve the entire VLAN tag as it passes through this chain of interfaces?**
## Solution Attempt 3
**VLAN-aware bridge**
ip link add name br0 type bridge vlan_filtering 1
ip link set dev br0 up
ip link set eth0 up
ip link set eth1 up
ip link set dev eth0 master br0
ip link set dev eth1 master br0
bridge vlan add vid 2-4094 dev br0 self
bridge vlan add vid 2-4094 dev eth0 master
bridge vlan add vid 2-4094 dev eth1 master
This would be a potentially ideal solution, since it doesn't suffer from the scaling problem of solution 2. However, the driver for the TI Ethernet switch fails to initialize when VLAN forwarding is enabled. The TI CPSW driver reports an error saying it is unable to initialize VLAN forwarding on the interface.
user25729276
(19 rep)
Jun 25, 2024, 11:29 PM
• Last activity: Jun 26, 2024, 08:29 PM
1
votes
1
answers
186
views
Is it possible to configure Ethertype DSA VLAN tagging on an Ethernet port that has a real PHY connection to the outside world?
I am working on an embedded system that will connect a processor running Linux to an Ethernet switch via a "conduit" Ethernet link (e.g. eth0) using the port-based Ethertype DSA (Marvell) frame tagging protocol, so that the Linux kernel will present the userspace with separate Ethernet links for eac...
I am working on an embedded system that will connect a processor running Linux to an Ethernet switch via a "conduit" Ethernet link (e.g. eth0) using the port-based Ethertype DSA (Marvell) frame tagging protocol, so that the Linux kernel will present the userspace with separate Ethernet links for each of the switch's user ports (e.g. lan1, lan2, lan3).
As I understand it this is usually done via a fixed phy-less conduit (e.g. xMII) and an MDIO link between the processor and the switch so that the processor can configure and control the physical user ports.
I want to do some software development ahead of receiving the hardware, so I want to "fake" the MDIO link, or in some other way configure the kernel, such that it uses port-based EDSA frame tagging over an ordinary (PHY-equipped) Ethernet port, resulting in a frame layout like this:
7 6 5 4 3 2 1 0
. . . . . . . . .
0 +---+---+---+---+---+---+---+---+
| Ether Destination Address |
+6 +---+---+---+---+---+---+---+---+
| Ether Source Address |
+6 +---+---+---+---+---+---+---+---+ --
| Prog. DSA Ether Type [15:8] | |
+1 +---+---+---+---+---+---+---+---+ |
| Prog. DSA Ether Type [7:0] | | EDSA tag
+1 +---+---+---+---+---+---+---+---+ |
| Reserved (0x00 0x00) | |
+2 +---+---+---+---+---+---+---+---+ | --
| Mode |b29| Switch Device | | |
+1 +---+---+---+---+---+---+---+---+ | |
| Switch Port |b18|b17|b16| | |
+1 +---+---+---+---+---+---+---+---+ | | DSA tag
| PRI [2:0] |b12| VID [11:8] | | |
+1 +---+---+---+---+---+---+---+---+ | |
| VID [7:0] | | |
+1 +---+---+---+---+---+---+---+---+ -- --
| Ether Length/Type |
+2 +---+---+---+---+---+---+---+---+
. . . . . . . . .
...where
Switch Port
corresponds to the switch's user port number.
The documentation of kernel modules dsa_loop
and dsa_loop_bdinfo
imply that these might be suitable, and the source code seems to link .netdev = "eth0"
to some user ports ("lan1".."lan4"
) but even though I can modprobe dsa_loop
into my kernel I don't see any virtual port devices created by the kernel (I am using ifconfig -a
to show net devices).
~# lsmod
Module Size Used by
dsa_loop 16384 0
dsa_core 118784 1 dsa_loop
When doing the modprobe dsa_loop
I see a call to dsa_loop_init
but no call to dsa_loop_drv_probe
.
So I guess I have several questions:
- Are the dsa_loop
and dsa_loop_init
modules intended to allow EDSA tagging to be performed on a "normal" Ethernet link?
- How are these modules intended to be used?
- Are there any other ways of configuring Linux to do this without access to the switch MDIO?
srl100
(13 rep)
Apr 2, 2024, 10:43 AM
• Last activity: May 26, 2024, 10:03 AM
0
votes
0
answers
68
views
What's the difference between setting up bridge VLAN with `bridge` command and /etc/network/interfaces
On Debian, I noticed I can set up bridge VLANs in /etc/network/interfaces with bridge-tools installed: auto br0 iface br0 inet static bridge_ports enp2s0 enp3s0 enp4s0 enp5s0 enp7s0 address 192.168.0.1 netmask 255.255.255.0 auto br0.2 iface br0.2 inet static address 192.168.1.1 netmask 255.255.255.0...
On Debian, I noticed I can set up bridge VLANs in /etc/network/interfaces with bridge-tools installed:
auto br0
iface br0 inet static
bridge_ports enp2s0 enp3s0 enp4s0 enp5s0 enp7s0
address 192.168.0.1
netmask 255.255.255.0
auto br0.2
iface br0.2 inet static
address 192.168.1.1
netmask 255.255.255.0
vlan_raw_device br0
I just learned the other way to do this is via bridge command. What's the difference between them? Since the first way of setting up VLAN is not mentioned a lot, are there problems with it?
CodeDuck
(1 rep)
May 21, 2024, 10:03 PM
1
votes
1
answers
250
views
Marking packets in iptables based on output interface
I have an unusual setup on my server. We have three outgoing ethernet ports, all connected to a single bridge interface that we split into two VLANs: ```none ip link add veth type bridge ip link set veth address 01:23:45:67:89:0A ip link set dev eth1 master veth ip link set dev eth2 master veth ip l...
I have an unusual setup on my server. We have three outgoing ethernet ports, all connected to a single bridge interface that we split into two VLANs:
ip link add veth type bridge
ip link set veth address 01:23:45:67:89:0A
ip link set dev eth1 master veth
ip link set dev eth2 master veth
ip link set dev eth3 master veth
[...]
ip link add veth name veth.10 type vlan id 10
ip link add veth name veth.20 type vlan id 20
ip link add veth.20-local link veth.20 type macvlan mode bridge
[...]
docker network create --driver=macvlan --subnet=192.168.XXX.0/24 --opt com.docker.network.driver.mpu=1500 --gateway 192.168.XXX.1 --opt parent=veth.20-local dockerbr20
I have a docker image inside my server connected to the veth.20
address, that is only allowed to communicate over veth.20
. There are routing and forwarding rules in the rest of the network that allow the docker image to communicate to a few select destinations outside of that VLAN.
I would like to add an iptables
rule covering outgoing packets that leave my server out of the veth.20
interface, regardless of their destination. (Some packets must stay within the veth.20
interface; some can be routed over to other VLANS.)
The following rules have been attempted, and for whatever reason, do not appear to mark packets leaving over veth.20
from the docker container:
-shell
iptables -A POSTROUTING -t mangle -o veth.20 -j MARK --set-mark 3
iptables -A PREROUTING -t mangle -i veth.20 -j MARK --set-mark 3
iptables -A POSTROUTING -t nat -o veth.20 -j MARK --set-mark 3
iptables -A PREROUTING -t nat -i veth.20 -j MARK --set-mark 3
iptables -A OUTPUT -t mangle -o veth.20 -j MARK --set-mark 3
iptables -A INPUT -t mangle -i veth.20 -j MARK --set-mark 3
iptables -A FORWARD -i veth.20 -j MARK --set-mark 3
iptables -A FORWARD -o veth.20 -j MARK --set-mark 3
That is, iptables -L -n -v -t mangle
and iptables -L -n -v -t nat
do not show any of these rules being applied to outgoing packets from veth.20
to a host on another VLAN.
I have confirmed, through ifconfig
, that all the packets from the docker image are leaving the server over veth.20
; the
-shell
iptables -A OUTPUT -t mangle -o veth.20 -j MARK --set-mark 3
rule applies when I send packets to external machines on VLAN 20, from the server or the docker image; but when I send packets out over the veth.20
interface from docker that are routed to an external VLAN 10 or VLAN 30 address (not pictured), no marks are applied.
I feel like this should be a simple problem, but nothing I've tried has been able to mark based on the interface that the packet uses to leave the box. What am I missing?
Garrett
(13 rep)
Mar 12, 2024, 07:59 PM
• Last activity: Apr 11, 2024, 10:01 AM
1
votes
1
answers
5719
views
How to create a linux vlan aware bridge with ingress packets untagged and egress packets tagged
I am trying to implement an obscure workaround to a specific problem tied to a switch misconfiguration I observe in a real case scenario. Please just assume the following : - My system is plugged to a trunk link with a cisco switch ; - I can send tagged packets in a specific VLAN (not native vlan),...
I am trying to implement an obscure workaround to a specific problem tied to a switch misconfiguration I observe in a real case scenario.
Please just assume the following :
- My system is plugged to a trunk link with a cisco switch ;
- I can send tagged packets in a specific VLAN (not native vlan), it will be properly routed to the correct VLAN ;
- I receive responses as untagged traffic, even though it is on a trunk and from not-native VLAN ;
- I cannot modify the switch's configuration.
What I am trying to perform is a way to use a single network interface on Debian, where I can send and receive traffic correctly to insert my traffic in the desired VLAN, even though the packets must be tagged going out and will be caught back untagged.
To give an example, It is possible to ping a remote system with an ICMP echo request. Using my simple eth0 interface will have no effect, no ICMP echo reply will be seen back. However, if I tag my ICMP echo request in say VLAN 10, I will receive an untagged ICMP echo reply ! It works also with DHCP for example (the remote network do has a DHCP server). If I send a tagged DHCP discover packet, I will receive an untagged DHCP offer with the corresponding transaction ID.
I know this is not normal behavior and it can end up in having packets being wrongly routed or assumed to be in a VLAN falsely. It is due to switches being misconfigured (I think it has something to do with native vlan mismatch somewhere).
If I use a vlan subinterface (ip link add link type vlan ...), I will send tagged traffic but responses will never be routed back from the main interface to the sub interface because it expects traffic to be tagged back.
I then discovered vlan aware bridges, which look like a proper solution to my problem, though I dont succeed in doing this with the bridge vlan add commands (pvid, vid, untagged or not, do I need to use a vlan sub-interface anyway ?).
I only need to do this for a single VLAN at a time. It will not be possible to determine to which VLAN a packet is sent because of the tags missing. However, I can infer this VLAN tag based on IP address, STP and CDP traffic. I do not want to do anything automated, I dont want to guess the vlan, I just want to be able to force untagged traffic to be considered being in a chosen, arbitrary vlan, one at a time.
Thank you !
Almandin
(13 rep)
Feb 16, 2024, 02:45 PM
• Last activity: Feb 20, 2024, 07:50 AM
1
votes
0
answers
115
views
Ignore vlan tagged packets on primary interface
I have an Ubuntu VM on ESXi with a single network interface which is a port group tagged as VLAN 4095 so it has access to all tagged VLANs. The native VLAN is 37. I've created several vlan interfaces using Netplan like this: ```yaml network: ethernets: ens160: dhcp4: true version: 2 vlans: vlan98: i...
I have an Ubuntu VM on ESXi with a single network interface which is a port group tagged as VLAN 4095 so it has access to all tagged VLANs. The native VLAN is 37.
I've created several vlan interfaces using Netplan like this:
network:
ethernets:
ens160:
dhcp4: true
version: 2
vlans:
vlan98:
id: 98
link: ens160
dhcp4: true
optional: true
link-local: []
I have a service receiving multicast traffic on all interfaces, including the primary ens160
, and I want the traffic to remain separate for each interface. The problem I'm facing is: ens160
receives *all* packets, including tagged ones. Is there a way to make this primary interface *only* see untagged packets?
# ip -d link show
2: ens160: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:07:84:53 brd ff:ff:ff:ff:ff:ff promiscuity 13 minmtu 60 maxmtu 9000 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
25: vlan98@ens160: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:07:84:53 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 0 maxmtu 65535
vlan protocol 802.1Q id 98 addrgenmode none numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
Example showing the packet I would like to exclude:
# tcpdump -ni ens160 -e host 10.37.154.4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
15:55:00.023677 00:60:74:fb:d0:87 > 01:00:5e:00:01:81, ethertype 802.1Q (0x8100), length 90: vlan 98, p 0, ethertype IPv4, 10.37.154.4.319 > 224.0.1.129.319: UDP, length 44
I tried disabling dhcp on ens160 and then adding an additional interface for vlan37 but that brings networking down entirely.
Elliott B
(575 rep)
Dec 2, 2023, 12:03 AM
5
votes
4
answers
7510
views
Private VLAN's under Linux?
Has anyone tried [private VLANs][1] under Linux? Any experiences with them? My real question is does anybody have howtos regarding this? [1]: https://en.wikipedia.org/wiki/Private_VLAN
Has anyone tried private VLANs under Linux? Any experiences with them? My real question is does anybody have howtos regarding this?
LanceBaynes
(41465 rep)
Apr 26, 2011, 09:36 AM
• Last activity: Oct 31, 2023, 06:39 PM
Showing page 1 of 20 total questions