ice.7

.\" Man page generated from reStructuredText.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "ICE" "7" "March 02, 2026" "" "Linux ice readme"
.SH NAME
ice \- ice Linux* Base Driver for the Intel(R) Ethernet 800 Series
.SS Contents
.INDENT 0.0
.IP \(bu 2
\fI\%ice Linux* Base Driver for the Intel(R) Ethernet 800 Series\fP
.INDENT 2.0
.IP \(bu 2
\fI\%Overview\fP
.IP \(bu 2
\fI\%Related Documentation\fP
.IP \(bu 2
\fI\%Identifying Your Adapter\fP
.IP \(bu 2
\fI\%Important Notes\fP
.IP \(bu 2
\fI\%Building and Installation\fP
.IP \(bu 2
\fI\%Command Line Parameters\fP
.IP \(bu 2
\fI\%Additional Features and Configurations\fP
.IP \(bu 2
\fI\%Performance Optimization\fP
.IP \(bu 2
\fI\%Known Issues/Troubleshooting\fP
.IP \(bu 2
\fI\%Support\fP
.IP \(bu 2
\fI\%License\fP
.IP \(bu 2
\fI\%Trademarks\fP
.UNINDENT
.UNINDENT
.SH OVERVIEW
.sp
This driver supports Linux* kernel versions 3.10.0 and newer. However, some
features may require a newer kernel version. The associated Virtual Function
(VF) driver for this driver is iavf. The associated RDMA driver for this driver
is irdma.
.sp
Driver information can be obtained using ethtool, devlink, lspci, and ip.
Instructions on updating ethtool can be found in the section Additional
Configurations later in this document.
.sp
This driver is only supported as a loadable module at this time. Intel is not
supplying patches against the kernel source to allow for static linking of the
drivers.
.sp
For questions related to hardware requirements, refer to the documentation
supplied with your Intel adapter. All hardware requirements listed apply to use
with Linux.
.sp
This driver supports XDP (Express Data Path) on kernel 4.14 and later and
AF_XDP zero\-copy on kernel 4.18 and later. Note that XDP is blocked for frame
sizes larger than 3KB.
.SH RELATED DOCUMENTATION
.sp
See the \(dqIntel(R) Ethernet Adapters and Devices User Guide\(dq for additional
information on features. It is available on the Intel website at
\fI\%https://cdrdv2.intel.com/v1/dl/getContent/705831\fP\&.
.SH IDENTIFYING YOUR ADAPTER
.sp
The driver is compatible with devices based on the following:
.INDENT 0.0
.IP \(bu 2
Intel(R) Ethernet Controller E810\-C
.IP \(bu 2
Intel(R) Ethernet Controller E810\-XXV
.IP \(bu 2
Intel(R) Ethernet Connection E822\-C
.IP \(bu 2
Intel(R) Ethernet Connection E822\-L
.IP \(bu 2
Intel(R) Ethernet Connection E823\-C
.IP \(bu 2
Intel(R) Ethernet Connection E823\-L
.IP \(bu 2
Intel(R) Ethernet Connection E825\-C
.IP \(bu 2
Intel(R) Ethernet Controller E830
.IP \(bu 2
Intel(R) Ethernet Controller E835
.UNINDENT
.sp
For information on how to identify your adapter, and for the latest Intel
network drivers, refer to the Intel Support website at
\fI\%https://www.intel.com/support\fP\&.
.SH IMPORTANT NOTES
.SS Configuring SR\-IOV for improved network security
.sp
In a virtualized environment, on Intel(R) Ethernet Network Adapters that
support SR\-IOV, the virtual function (VF) may be subject to malicious behavior.
Software\-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
802.1Qbb (priority based flow\-control), and others of this type, are not
expected and can throttle traffic between the host and the virtual switch,
reducing performance. To resolve this issue, and to ensure isolation from
unintended traffic streams, configure all SR\-IOV enabled
ports for VLAN tagging from the administrative interface on the PF. This
configuration allows unexpected, and potentially malicious, frames to be
dropped.
.sp
See \fI\%Configuring VLAN Tagging on SR\-IOV Enabled Adapter Ports\fP later in this README for configuration
instructions.
.SS Do not unload port driver if VF with active VM is bound to it
.sp
Do not unload a port\(aqs driver if a Virtual Function (VF) with an active Virtual
Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
Once the VM shuts down, or otherwise releases the VF, the command will complete.
.SS Firmware Recovery Mode
.sp
A device will enter Firmware Recovery mode if it detects a problem that
requires the firmware to be reprogrammed. When a device is in Firmware Recovery
mode it will not pass traffic or allow any configuration; you can only attempt
to recover the device\(aqs firmware. Refer to the \(dqIntel(R) Ethernet Adapters and
Devices User Guide\(dq for details on Firmware Recovery Mode and how to recover
from it.
.SS Important Notes for SR\-IOV, RDMA, and Link Aggregation
.sp
The VF driver will not block teaming/bonding/link aggregation, but this is not
a supported feature. Do not expect failover or load balancing on the VF
interface.
.sp
LAG and RDMA are compatible only in certain conditions. See the \fI\%Link Aggregation and RDMA\fP
section later in this README for more information.
.sp
Likewise, LAG and SR\-IOV are compatible only in certain conditions. See the
\fI\%Link Aggregation and SR\-IOV VFs\fP section for more information.
.sp
Bridging and MACVLAN are also affected by this. If you wish to use bridging or
MACVLAN with RDMA/SR\-IOV, you must set up bridging or MACVLAN before enabling
RDMA or SR\-IOV. If you are using bridging or MACVLAN in conjunction with SR\-IOV
and/or RDMA, and you want to remove the interface from the bridge or MACVLAN,
you must follow these steps:
.INDENT 0.0
.IP 1. 3
Remove RDMA if it is active
.IP 2. 3
Destroy SR\-IOV VFs if they exist
.IP 3. 3
Remove the interface from the bridge or MACVLAN
.IP 4. 3
Reactivate RDMA and recreate SR\-IOV VFs as needed
.UNINDENT
.SH BUILDING AND INSTALLATION
.sp
The ice driver requires the Dynamic Device Personalization (DDP) package file
to enable advanced features (such as dynamic tunneling, Intel(R) Ethernet Flow
Director, RSS, and ADQ, or others). The driver installation process installs
the default DDP package file and creates a soft link \fBice.pkg\fP to the physical
package \fBice\-x.x.x.x.pkg\fP in the firmware root directory (typically
\fB/lib/firmware/\fP or \fB/lib/firmware/updates/\fP). The driver install process
also puts both the driver module and the DDP file in the \fBinitramfs/initrd\fP
image.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When the driver loads, it looks for \fBintel/ice/ddp/ice.pkg\fP in the
firmware root. If this file exists, the driver will download it into the
device. If not, the driver will go into Safe Mode where it will use the
configuration contained in the device\(aqs NVM. This is NOT a supported
configuration and many advanced features will not be functional. See
\fI\%Dynamic Device Personalization\fP later for more information.
.UNINDENT
.UNINDENT
.SS To manually build the driver
.INDENT 0.0
.IP 1. 3
Move the base driver tar file to the directory of your choice.
For example, use \fB/home/username/ice\fP or \fB/usr/local/src/ice\fP\&.
.IP 2. 3
Untar/unzip the archive, where \fB<x.x.x>\fP is the version number for the
driver tar file:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
tar zxf ice\-<x.x.x>.tar.gz
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
Change to the driver src directory, where \fB<x.x.x>\fP is the version number
for the driver tar:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
cd ice\-<x.x.x>/src/
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 4. 3
Compile the driver module:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
make install
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The binary will be installed as:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
/lib/modules/<KERNEL VER>/updates/drivers/net/ethernet/intel/ice/ice.ko
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The install location listed above is the default location. This may differ
for various Linux distributions.
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
To build the driver using the schema for unified ethtool statistics,
use the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
make CFLAGS_EXTRA=\(aq\-DUNIFIED_STATS\(aq install
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
To compile the driver with ADQ (Application Device Queues) flags
set, use the following command, where \fB<nproc>\fP is the number of logical
cores:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
make \-j<nproc> CFLAGS_EXTRA=\(aq\-DADQ_PERF_COUNTERS\(aq install
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
(This will also apply the above \fBmake install\fP command.)
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
You may see warnings from depmod related to unknown RDMA symbols
during the make of the out\-of\-tree base driver. These warnings are normal and
appear because the in\-tree RDMA driver will not work with the out\-of\-tree base
driver. To address the issue, you need to install the latest out\-of\-tree versions
of the base and RDMA drivers.
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
Some Linux distributions require you to manually regenerate initramfs/initrd
after installing the driver to allow the driver to properly load with the firmware
at boot time. Please refer to the distribution documentation for instructions.
.UNINDENT
.UNINDENT
.IP 5. 3
Load the module using the modprobe command.
.sp
To check the version of the driver and then load it:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
modinfo ice
modprobe ice
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Alternately, make sure that any older ice drivers are removed from the
kernel before loading the new module:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
rmmod ice; modprobe ice
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
To enable verbose debug messages in the kernel log, use the
dynamic debug feature (dyndbg). See \fI\%Dynamic Debug\fP later in this
README for more information.
.UNINDENT
.UNINDENT
.IP 6. 3
Assign an IP address to the interface by entering the following,
where \fB<ethX>\fP is the interface name that was shown in dmesg after
modprobe:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ip address add <IP_address>/<netmask bits> dev <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 7. 3
Verify that the interface works. Enter the following, where \fBIP_address\fP
is the IP address for another machine on the same subnet as the interface
that is being tested:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ping <IP_address>
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.SS To build a binary RPM package of this driver
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
RPM functionality has only been tested in Red Hat distributions.
.UNINDENT
.UNINDENT
.INDENT 0.0
.IP 1. 3
Run the following command, where \fB<x.x.x>\fP is the version number for the
driver tar file:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
rpmbuild \-tb ice\-<x.x.x>.tar.gz
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
For the build to work properly, the currently running kernel MUST
match the version and configuration of the installed kernel sources. If
you have just recompiled the kernel, reboot the system before building.
.UNINDENT
.UNINDENT
.IP 2. 3
After building the RPM, the last few lines of the tool output contain the
location of the RPM file that was built. Install the RPM with one of the
following commands, where \fB<RPM>\fP is the location of the RPM file:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
rpm \-Uvh <RPM>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
or:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
dnf/yum localinstall <RPM>
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
If your distribution or kernel does not contain inbox support for auxiliary
bus, you must also install the auxiliary RPM:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
rpm \-Uvh <ice RPM> <auxiliary RPM>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
or:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
dnf/yum localinstall <ice RPM> <auxiliary RPM>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
On some distributions, the auxiliary RPM may fail to install due
to missing kernel\-devel headers. To workaround this issue, specify
\fB\-\-excludepath\fP during installation. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
rpm \-Uvh auxiliary\-1.0.0\-1.x86_64.rpm \-\-excludepath=/lib/modules/3.10.0\-957.el7.x86_64/source/include/linux/auxiliary_bus.h
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
To compile the driver on some kernel/arch combinations, you may need to
install a package with the development version of libelf (e.g. libelf\-dev,
libelf\-devel, elfutils\-libelf\-devel).
.IP \(bu 2
When compiling an out\-of\-tree driver, details will vary by distribution.
However, you will usually need a kernel\-devel RPM or some RPM that provides
the kernel headers at a minimum. The RPM kernel\-devel will usually fill in
the link at \fB/lib/modules/\(aquname \-r\(aq/build\fP\&.
.UNINDENT
.UNINDENT
.UNINDENT
.SH COMMAND LINE PARAMETERS
.sp
The only command line parameter the ice driver supports is the debug parameter
that can control the default logging verbosity of the driver. (Note: dyndbg
also provides dynamic debug information.)
.sp
In general, use ethtool and other OS\-specific commands to configure
user\-changeable parameters after the driver is loaded.
.SH ADDITIONAL FEATURES AND CONFIGURATIONS
.SS ethtool
.sp
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at
\fI\%https://kernel.org/pub/software/network/ethtool/\fP\&.
.SS Viewing Link Messages
.sp
Link messages will not be displayed to the console if the distribution is
restricting system messages. In order to see network driver link messages on
your console, set dmesg to eight by entering the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
dmesg \-n 8
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
This setting is not saved across reboots.
.UNINDENT
.UNINDENT
.SS Dynamic Device Personalization
.sp
Dynamic Device Personalization (DDP) allows you to change the packet processing
pipeline of a device by applying a profile package to the device at runtime.
Profiles can be used to, for example, add support for new protocols, change
existing protocols, or change default settings. DDP profiles can also be rolled
back without rebooting the system.
.sp
The ice driver automatically installs the default DDP package file during
driver installation.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
It\(aqs important to do \fBmake install\fP during initial ice driver
installation so that the driver loads the DDP package automatically.
.UNINDENT
.UNINDENT
.sp
The DDP package loads during device initialization. The driver looks for
\fBintel/ice/ddp/ice.pkg\fP in your firmware root (typically \fB/lib/firmware/\fP
or \fB/lib/firmware/updates/\fP) and checks that it contains a valid DDP package
file.
.sp
If the driver is unable to load the DDP package, the device will enter Safe
Mode. Safe Mode disables advanced and performance features and supports only
basic traffic and minimal functionality, such as updating the NVM or
downloading a new driver or DDP package. Safe Mode only applies to the affected
physical function and does not impact any other PFs. See the \(dqIntel(R) Ethernet
Adapters and Devices User Guide\(dq for more details on DDP and Safe Mode.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
If you encounter issues with the DDP package file, you may need to download
an updated driver or DDP package file. See the log messages for more
information.
.IP \(bu 2
The \fBice.pkg\fP file is a symbolic link to the default DDP package file
installed by the Linux\-firmware software package or the ice out\-of\-tree
driver installation.
.IP \(bu 2
You cannot update the DDP package if any PF drivers are already loaded. To
overwrite a package, unload all PFs and then reload the driver with the
new package.
.IP \(bu 2
Only the first loaded PF per device can download a package for that device.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
You can install specific DDP package files for different physical devices in
the same system. To install a specific DDP package file:
.INDENT 0.0
.IP 1. 3
Download the DDP package file you want for your device.
.IP 2. 3
Rename the file \fBice\-xxxxxxxxxxxxxxxx.pkg\fP, where \fBxxxxxxxxxxxxxxxx\fP is
the unique 64\-bit PCI Express device serial number (in hex) of the device
you want the package downloaded on. The file name must include the complete
serial number (including leading zeros) and be all lowercase. For example,
if the 64\-bit serial number is b887a3ffffca0568, then the file name would be
\fBice\-b887a3ffffca0568.pkg\fP\&.
.UNINDENT
.INDENT 0.0
.INDENT 3.5
To find the serial number from the PCI bus address, you can use the following
command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
lspci \-vv \-s af:00.0 | grep \-i Serial

Capabilities: [150 v1] Device Serial Number b8\-87\-a3\-ff\-ff\-ca\-05\-68
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
You can use the following command to format the serial number without the
dashes:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
lspci \-vv \-s af:00.0 | grep \-i Serial | awk \(aq{print $7}\(aq | sed s/\-//g b887a3ffffca0568
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.INDENT 0.0
.IP 3. 3
Copy the renamed DDP package file to \fB/lib/firmware/updates/intel/ice/ddp/\fP\&.
If the directory does not yet exist, create it before copying the file.
.IP 4. 3
Unload all of the PFs on the device.
.IP 5. 3
Reload the driver with the new package.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
The presence of a device\-specific DDP package file overrides the
loading of the default DDP package file (\fBice.pkg\fP).
.UNINDENT
.UNINDENT
.SS RDMA (Remote Direct Memory Access)
.sp
Remote Direct Memory Access, or RDMA, allows a network device to transfer data
directly to and from application memory on another system, increasing
throughput and lowering latency in certain networking environments.
.sp
The ice driver supports the following RDMA protocols:
.INDENT 0.0
.IP \(bu 2
iWARP (Internet Wide Area RDMA Protocol)
.IP \(bu 2
RoCEv2 (RDMA over Converged Ethernet)
.UNINDENT
.sp
The major difference is that iWARP performs RDMA over TCP, while RoCEv2 uses
UDP.
.sp
RDMA requires auxiliary bus support. Refer to \fI\%Auxiliary Bus\fP in this README
for more information.
.sp
Devices based on the Intel(R) Ethernet 800 Series do not support RDMA when
operating in multiport mode with more than 4 ports.
.sp
For detailed installation and configuration information for RDMA, see the
README file in the irdma driver tarball.
.SS RDMA in the VF
.sp
Devices based on the Intel(R) Ethernet 800 Series support RDMA in a Linux VF,
on supported Windows or Linux hosts.
.sp
The iavf driver supports the following RDMA protocols in the VF:
.INDENT 0.0
.IP \(bu 2
iWARP (Internet Wide Area RDMA Protocol)
.IP \(bu 2
RoCEv2 (RDMA over Converged Ethernet)
.UNINDENT
.sp
Refer to the README inside the irdma driver tarball for details on configuring
RDMA in the VF.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
To support VF RDMA, load the irdma driver on the host before creating
VFs. Otherwise VF RDMA support may not be negotiated between the VF and PF
driver.
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When using Link Aggregation, RDMA in VF is not available \- the
VIRTCHNL capability VIRTCHNL_VF_CAP_RDMA is disabled.
.UNINDENT
.UNINDENT
.SS Auxiliary Bus
.sp
Inter\-Driver Communication (IDC) is the mechanism in which LAN drivers (such as
ice) communicate with peer drivers (such as irdma). Starting in kernel 5.11,
Intel LAN and RDMA drivers use an auxiliary bus mechanism for IDC.
.sp
RDMA functionality requires use of the auxiliary bus.
.sp
If your kernel supports the auxiliary bus, the LAN and RDMA drivers will use
the inbox auxiliary bus for IDC. For kernels lower than 5.11, the base driver
will automatically install an out\-of\-tree auxiliary bus module.
.SS Source Pruning
.sp
E800 Series devices by default prune packets where the source MAC address
matches the destination MAC address.
The ice driver allows you to enable or disable source MAC address pruning of
Ethernet packets. You can use this feature without a bond created or when
active\-backup bonding is configured with ARP monitoring.
.sp
Use the ethtool private flag mac\-source\-pruning to enable or disable source
MAC pruning on received packets:
.INDENT 0.0
.TP
.B # ethtool \-\-set\-priv\-flags <ethX> mac\-source\-pruning on|off
Where:
on \- discards packets with source MAC address matching the source MAC
.INDENT 7.0
.INDENT 3.5
address of the interface (default)
.UNINDENT
.UNINDENT
.INDENT 7.0
.TP
.B off \- receives packets with source MAC address matching the source MAC
address of the interface
.UNINDENT
.UNINDENT
.sp
NOTE: When you set mac\-source\-pruning to on (disabled):
\- The driver will disable spoof check.
\- The network interface will receive all packets with a source MAC address
matching the network interface\(aqs MAC address.
.sp
The ice driver allows to enable or disable source pruning on a specified VF
by setting the private PF flag vf\-source\-pruning and setting correct spoofchk
and trust settings on the specified VF (see below example).
.INDENT 0.0
.TP
.B # ethtool \-\-set\-priv\-flags <ethX> vf\-source\-pruning on|off
Where:
on \- discards packets with source MAC address matching the source MAC
.INDENT 7.0
.INDENT 3.5
address of the interface (default)
.UNINDENT
.UNINDENT
.INDENT 7.0
.TP
.B off \- receives packets with source MAC address matching the source MAC
address of the interface
.UNINDENT
.UNINDENT
.sp
NOTE: Enabling vf\-source\-pruning will not automatically set anti\-spoofing and
trust mode automatically. You must perform these steps yourself. They can be
performed in any order. For example:
# ip link set <ethX> vf <vf id> spoofchk off
# ip link set <ethX> vf <vf id> trust on
# ethtool \-\-set\-priv\-flag <ethX> vf\-source\-pruning off
.sp
For more information, refer to _macvlanantispoofing
and _trusted\-vf in this README.
.SS NVM Express* (NVMe) over TCP and Fabrics
.sp
RDMA provides a high throughput, low latency means to directly access NVM
Express* (NVMe*) drives on a remote server.
.sp
Refer to the following configuration guides for details on supported operating
systems and how to set up and configure your server and client systems:
.INDENT 0.0
.IP \(bu 2
NVM Express over TCP for Intel(R) Ethernet Products Configuration Guide
.IP \(bu 2
NVM Express over Fabrics for Intel(R) Ethernet Products with RDMA
Configuration Guide
.UNINDENT
.sp
Both guides are available on the Intel Technical Library at:
\fI\%https://www.intel.com/content/www/us/en/design/products\-and\-solutions/networking\-and\-io/ethernet\-controller\-e810/technical\-library.html\fP
.SS Link Aggregation
.sp
Link Aggregation (LAG)/bonding allows combining multiple physical interfaces to
provide redundancy and in some configurations increase total throughput.
.SS Link Aggregation and RDMA
.sp
Link aggregation (LAG) and RDMA are compatible only if all the following are
true:
.INDENT 0.0
.IP \(bu 2
You are using an Intel Ethernet 810 Series device with the latest drivers and
NVM installed.
.IP \(bu 2
RDMA technology is set to RoCEv2.
.IP \(bu 2
LAG configuration is either active\-backup or active\-active.
.IP \(bu 2
Bonding is between two ports within the same device.
.IP \(bu 2
The QoS configuration of the two ports matches prior to the bonding of the
devices.
.UNINDENT
.sp
If the above conditions are not met:
.INDENT 0.0
.IP \(bu 2
The PF driver will not enable RDMA.
.IP \(bu 2
RDMA peers will not be able to register with the PF.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
The first interface added to an aggregate (bond) is assigned as the
\(dqprimary\(dq interface for RDMA and LAG functionality. If LAN interfaces are
assigned to the bond and you remove the primary interface from the bond,
RDMA will not function properly over the bonded interface. To address the
issue, remove all interfaces from the bond and add them again. Interfaces
that are not assigned to the bond will operate normally.
.UNINDENT
.UNINDENT
.sp
If the ice driver is configured for active\-backup or active\-active LAG:
.INDENT 0.0
.IP \(bu 2
The ice driver will block any DCB/hardware QoS configuration changes on the
bonded ports.
.IP \(bu 2
Only the primary port is available for the RDMA driver.
.IP \(bu 2
The ice driver will forward RoCEv2 traffic from the secondary port to the
primary port by creating an appropriate switch rule.
.UNINDENT
.sp
If the ice driver is configured for active\-active LAG:
.INDENT 0.0
.IP \(bu 2
The ice driver will allow the RDMA driver to configure QSets for both active
ports.
.IP \(bu 2
A port failure on the active port will trigger a failover mechanism and move
the queue pairs to the currently active port. Once the port has recovered,
the RDMA driver will move RDMA QSets back to the originally allocated port.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When using Link Aggregation, RDMA in VF is not available \- the
VIRTCHNL capability VIRTCHNL_VF_CAP_RDMA is disabled.
.UNINDENT
.UNINDENT
.SS Link Aggregation and SR\-IOV VFs
.sp
LAG and SR\-IOV are compatible only if all the following are true:
.INDENT 0.0
.IP \(bu 2
The bond is in one of the supported modes. Remaining modes are not compatible
.UNINDENT
.INDENT 0.0
.TP
.B with bridges:
.INDENT 7.0
.IP \(bu 2
active\-backup (1)
.IP \(bu 2
balance\-xor (2)
.IP \(bu 2
broadcast (3)
.IP \(bu 2
802.3ad (4)
.UNINDENT
.UNINDENT
.INDENT 0.0
.IP \(bu 2
The bond comprises of exactly two ports from the same NIC MAC chip
.IP \(bu 2
The primary port (first one added to the bond) is in switchdev mode (see
\fI\%Switchdev Mode\fP)
.IP \(bu 2
Both bond ports have identical DCB configurations
.IP \(bu 2
The FW LLDP Agent is disabled on both bond ports (see \fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP)
.IP \(bu 2
.INDENT 2.0
.TP
.B Firmware supports LAN VF LAG capability:
.INDENT 7.0
.IP \(bu 2
For active\-backup mode: version 4.30 or newer
.IP \(bu 2
For other modes: version 4.80 or newer
.UNINDENT
.UNINDENT
.IP \(bu 2
The ice_lag DDP package is loaded
.IP \(bu 2
Firmware supports LAN VF LAG capability (minimal version):
.UNINDENT
.TS
center;
|l|l|l|.
_
T{
Device
T}	T{
active\-backup
T}	T{
Other modes
T}
_
T{
E810
T}	T{
4.30
T}	T{
4.80
T}
_
T{
E83x Series
T}	T{
1.00
T}	T{
1.20
T}
_
.TE
.sp
In modes other than active\-backup (1), splitting traffic is achieved through
queue\-balancing.\ Half of each VFs queues are configured to utilize the
secondary uplink when it is active.\ When a link failure is detected, all queues
are reconfigured to utilize the remaining uplink. Association of VF queues to
uplinks is deterministic, therefore after restoring link after failover, queues
return to the original uplink.
.sp
In order to configure SR\-IOV LAG:
.INDENT 0.0
.IP 1. 3
Enable switchdev mode on the primary port:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev eswitch set <pci/0000:##:##.#> mode switchdev
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Create SR\-IOV VFs:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo 2 > /sys/class/net/$PF1/device/sriov_numvfs
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
Create a bond interface and configure it to one of the supported modes (see
above):
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ip link add bond0 type bond
ip link set bond0 type bond miimon 100 mode active\-backup
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 4. 3
Enslave primary and secondary port to bond:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set $PF1 down
ip link set $PF1 master bond0

ip link set $PF2 down
ip link set $PF2 master bond0
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 5. 3
Add a Linux bridge. Add bond and port representors to it:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ip link add $BR type bridge
ip link set $BOND master $BR
ip link set $VF1_PR master $BR
ip link set $VF2_PR master $BR
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Note: in this example \fB$PF1\fP is the primary port and \fB$PF2\fP is the secondary
port.
.SS Link aggregation with both SR\-IOV and RDMA
.sp
It is possible to use both SR\-IOV VFs and RDMA at the same time. In this
configuration bridge is not supported (tc filters should be configured to enable
VF connectivity \- see \fI\%Switchdev Mode\fP) and promiscuous mode should be
enabled on the secondary uplink:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set promisc on $PF2
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Application Device Queues (ADQ)
.sp
Application Device Queues (ADQ) allow you to dedicate one or more queues to a
specific application. This can reduce latency for the specified application,
and allow Tx traffic to be rate limited per application.
.sp
The ADQ information contained here is specific to the ice driver. For more
details, refer to the E810 ADQ Configuration Guide at
\fI\%https://cdrdv2.intel.com/v1/dl/getContent/609008\fP\&.
.sp
Requirements:
.INDENT 0.0
.IP \(bu 2
Kernel version: Varies by feature. Refer to the E810 ADQ Configuration Guide
for more information on required kernel versions for different ADQ features.
.IP \(bu 2
Operating system: Red Hat* Enterprise Linux* 7.5+ or SUSE* Linux Enterprise
Server* 12+
.IP \(bu 2
The latest ice driver and NVM image (Note: You must compile the ice driver
with the ADQ flag as shown in the \fI\%Building and Installation\fP section.)
.IP \(bu 2
The \fBsch_mqprio\fP, \fBact_mirred\fP, and \fBcls_flower\fP modules must be loaded.
For example:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
modprobe sch_mqprio
modprobe act_mirred
modprobe cls_flower
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
The latest version of iproute2
.sp
We recommend the following installation method:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
cd iproute2
\&./configure
make DESTDIR=/opt/iproute2 install
ln \-s /opt/iproute2/sbin/tc /usr/local/sbin/tc
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
When ADQ is enabled:
.INDENT 0.0
.IP \(bu 2
You cannot change RSS parameters, the number of queues, or the MAC address in
the PF or VF. Delete the ADQ configuration before changing these settings.
.IP \(bu 2
The driver supports subnet masks for IP addresses in the PF and VF. When you
add a subnet mask filter, the driver forwards packets to the ADQ VSI instead of
the main VSI.
.IP \(bu 2
When the PF adds or deletes a port VLAN filter for the VF, it will extend to
all the VSIs within that VF.
.IP \(bu 2
The driver supports ADQ and GTP filters in the PF. Note: You must have a DDP
package that supports GTP; the default OS package does not. Download the
appropriate package from your hardware vendor and load it on your device.
.IP \(bu 2
ADQ allows tc ingress filters that include any destination MAC address.
.IP \(bu 2
You can configure up to 256 queue pairs (256 MSI\-X interrupts) per PF.
.UNINDENT
.sp
See \fI\%Creating Traffic Class Filters\fP in this README for more information on
configuring filters, including examples. See the E810 ADQ Configuration Guide
for detailed instructions.
.sp
ADQ KNOWN ISSUES:
.INDENT 0.0
.IP \(bu 2
The latest RHEL and SLES distros have kernels with back\-ported support for
ADQ. For all other Linux distributions, you must use LTS Linux kernel v4.19.58
or higher to use ADQ. The latest out\-of\-tree driver is required for ADQ on all
operating systems.
.IP \(bu 2
You must clear ADQ configuration in the reverse order of the initial
configuration steps. Issues may result if you do not execute the steps to clear
ADQ configuration in the correct order.
.IP \(bu 2
ADQ configuration is not supported on a bonded or teamed ice interface.
Issuing the ethtool or tc commands to a bonded ice interface will result in
error messages from the ice driver to indicate the operation is not supported.
.IP \(bu 2
If the application stalls, the application\-specific queues may stall for up
to two seconds. Configuring only one application per Traffic Class (TC) channel
may resolve the issue.
.IP \(bu 2
DCB and ADQ cannot coexist. A switch with DCB enabled might remove the ADQ
configuration from the device. To resolve the issue, do not enable DCB on the
switch ports being used for ADQ. You must disable LLDP on the interface and
stop the firmware LLDP agent using the following command:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> fw\-lldp\-agent off
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
MACVLAN offloads and ADQ are mutually exclusive. System instability may occur
if you enable \fBl2\-fwd\-offload\fP and then set up ADQ, or if you set up ADQ and
then enable \fBl2\-fwd\-offload\fP\&.
.IP \(bu 2
Note (unrelated to Intel drivers): The version 5.8.0 Linux kernel introduced
a bug that broke the interrupt affinity setting mechanism, which breaks the
ability to pin interrupts to ADQ hardware queues. Use an earlier or later
version of the Linux kernel.
.IP \(bu 2
A core\-level reset of an ADQ\-configured PF port (rare events usually
triggered by other failures in the device or ice driver) results in loss of ADQ
configuration. To recover, reapply the ADQ configuration to the PF interface.
.IP \(bu 2
Commands such as \fBtc qdisc add\fP and \fBethtool \-L\fP will cause the driver
to close the associated RDMA interface and reopen it. This will disrupt RDMA
traffic for 3\-5 seconds until the RDMA interface is available again for
traffic.
.IP \(bu 2
Commands such as \fBtc qdisc add\fP and \fBethtool \-L\fP will clear other tuning
settings such as interrupt affinity. These tuning settings will need to be
reapplied. When the number of queues are increased using \fBethtool \-L\fP, the
new queues will have the same interrupt moderation settings as queue 0 (i.e.,
Tx queue 0 for new Tx queues and Rx queue 0 for new Rx queues). You can change
this using the ethtool per\-queue coalesce commands.
.IP \(bu 2
TC filters may not get offloaded in hardware if you apply them immediately
after issuing the \fBtc qdisc add\fP command. We recommend you wait 5 seconds
after issuing \fBtc qdisc add\fP before adding TC filters. Dmesg will report
the error if TC filters fail to add properly.
.UNINDENT
.sp
To set up the adapter for ADQ, where \fB<ethX>\fP is the interface in use:
.INDENT 0.0
.IP 1. 3
Reload the ice driver to remove any previous TC configuration:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
rmmod ice
modprobe ice
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Enable hardware TC offload on the interface:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> hw\-tc\-offload on
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
Disable LLDP on the interface, if it isn\(aqt already:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> fw\-lldp\-agent off
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 4. 3
Verify settings:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-k <ethX> | grep \(dqhw\-tc\(dq
ethtool \-\-show\-priv\-flags <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Intel also provides a script to configure ADQ. This script allows you configure
ADQ\-specific parameters such as traffic classes, priority, filters, and ethtool
parameters.
.sp
Refer to the \fBREADME.md\fP file in \fBscripts/adqsetup\fP inside the driver
tarball for more information.
.sp
The script and README are also available as part of the Python Package Index
at \fI\%https://pypi.org/project/adqsetup\fP\&.
.sp
The ice driver supports ADQ acceleration using independent pollers. Independent
pollers are kernel threads invoked by interrupts and are used for busy polling
on behalf of the application.
.sp
You can configure the number of queues per poller and poller timeout per ADQ
traffic class (TC) or queue group using the \fBdevlink dev param\fP interface.
.sp
To set the number of queue pairs per poller, use the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set <pci/D:b:d.f> name tc<x>_qps_per_poller value <num> cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <pci/D:b:d.f>
The PCI address of the device (pci/Domain:bus:device.function).
.TP
.B tc<x>
The traffic class number.
.TP
.B <num>
The number of queues of the corresponding traffic class that
each poller would poll.
.UNINDENT
.sp
To set the timeout for the independent poller, use the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set <pci/D:b:d.f> name tc<x>_poller_timeout value <num> cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <pci/D:b:d.f>
The PCI address of the device (pci/Domain:bus:device.function).
.TP
.B tc<x>
The traffic class number.
.TP
.B <num>
A nonzero integer value in jiffies.
.UNINDENT
.sp
For example:
.INDENT 0.0
.IP \(bu 2
To configure 3 queues of TC1 to be polled by each independent poller:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set pci/0000:3b:00.0 name tc1_qps_per_poller value 3 cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To set the timeout value in jiffies for TC1 when no traffic is flowing:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set pci/0000:3b:00.0 name tc1_poller_timeout value 1000 cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The ice out\-of\-tree driver allows you to configure inline Intel(R) Ethernet Flow
Director (Intel(R) Ethernet FD) filters per traffic class (TC) using the
devlink interface. Inline Intel Ethernet FD allows uniform distribution of
flows among queues in a TC.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
This functionality requires Linux kernel version 5.6 or newer and is
supported only with the out\-of\-tree ice driver.
.IP \(bu 2
You must enable Transmit Packet Steering (XPS) using receive queues for
this feature to work correctly.
.IP \(bu 2
Per\-TC filters set with devlink are not compatible with Intel Ethernet FD
filters set via ethtool.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Use the following to configure inline Intel Ethernet FD filters per TC:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set <pci/D:b:d.f> name tc<x>_inline_fd value <setting> cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <pci/D:b:d.f>
The PCI address of the device (pci/Domain:bus:device.function).
.TP
.B tc<x>
The traffic class number.
.TP
.B <setting>
Set to true to enable inline per\-TC Intel Ethernet FD, or
false to disable it.
.UNINDENT
.sp
For example, to enable inline Intel Ethernet FD for TC1:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set pci/0000:af:00.0 name tc1_inline_fd value true cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To show the current inline Intel Ethernet FD setting:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param show <pci/D:b:d.f> name tc<x>_inline_fd
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example, to show the inline Intel Ethernet FD setting for TC2 for the
specified device:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param show pci/0000:af:00.0 name tc2_inline_fd
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Creating Traffic Classes
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
These instructions are not specific to ADQ configuration. Refer to
the tc and tc\-flower man pages for more information on creating traffic
classes (TCs).
.IP \(bu 2
Disable FW\-LLDP before using tc command to add filter. Refer to the
\fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for more information on disabling
FW\-LLDP.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To create traffic classes on the interface:
.INDENT 0.0
.IP 1. 3
Use the tc command to create traffic classes. You can create a maximum of
16 TCs per interface:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev <ethX> root mqprio num_tc <tcs> map <priorities>
queues <count1@offset1 ...> hw 1 mode channel shaper bw_rlimit
min_rate <min_rate1 ...> max_rate <max_rate1 ...>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 3.0
.TP
.B num_tc <tcs>
The number of TCs to use.
.TP
.B map <priorities>
The map of priorities to TCs. You can map up to
16 priorities to TCs.
.TP
.B queues <count1@offset1 ...>
For each TC, \fB<num queues>\e@<offset>\fP\&. The max total
number of queues for all TCs is the number of cores.
.TP
.B hw 1 mode channel
\fBchannel\fP with \fBhw\fP set to 1 is a new hardware offload
mode in mqprio that makes full use of the mqprio options,
the TCs, the queue configurations, and the QoS parameters.
.TP
.B shaper bw_rlimit
For each TC, sets the minimum and maximum bandwidth
rates. The totals must be equal to or less than the port
speed. This parameter is optional and is required only to
set up the Tx rates.
.TP
.B min_rate <min_rate1>
Sets the minimum bandwidth rate limit for each TC.
.TP
.B max_rate <max_rate1 ...>
Sets the maximum bandwidth rate limit for each TC. You
can set a min and max rate together.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
If you set \fBmax_rate\fP to less than 50Mbps, then \fBmax_rate\fP is
rounded up to 50Mbps and a warning is logged in dmesg.
.IP \(bu 2
See the mqprio man page and the examples below for more information.
.UNINDENT
.UNINDENT
.UNINDENT
.IP 2. 3
Verify the bandwidth limit using network monitoring tools such as \fBifstat\fP
or \fBsar \-n DEV [interval] [number of samples]\fP\&.
.sp
\fBNOTE:\fP
.INDENT 3.0
.INDENT 3.5
Setting up channels via ethtool (\fBethtool \-L\fP) is not supported
when the TCs are configured using mqprio.
.UNINDENT
.UNINDENT
.IP 3. 3
Enable hardware TC offload on the interface:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> hw\-tc\-offload on
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.INDENT 0.0
.IP 2. 3
Add clsact qdisc to enable adding ingress/egress filters for Rx/Tx:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev <ethX> clsact
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
Verify successful TC creation after qdisc is created:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc show dev <ethX> ingress
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
TRAFFIC CLASS EXAMPLES:
.sp
See the tc and tc\-flower man pages for more information on traffic control and
TC flower filters.
.INDENT 0.0
.IP \(bu 2
To set up two TCs (tc0 and tc1), with 16 queues each, priorities 0\-3 for
tc0 and 4\-7 for tc1, and max Tx rate set to 1Gbit for tc0 and 3Gbit for tc1:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
16@0 16@16 hw 1 mode channel shaper bw_rlimit max_rate 1Gbit 3Gbit
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 2.0
.TP
.B map 0 0 0 0 1 1 1 1
Sets priorities 0\-3 to use tc0 and 4\-7 to use tc1
.TP
.B queues 16@0 16@16
Assigns 16 queues to tc0 at offset 0 and 16 queues
to tc1 at offset 16
.UNINDENT
.IP \(bu 2
To create 8 TCs with 256 queues spread across all the TCs, when ADQ is
enabled:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev <ethX> root mqprio num_tc 8 map 0 1 2 3 4 5 6 7
queues 2@0 4@2 8@6 16@14 32@30 64@62 128@126 2@254 hw 1 mode channel
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To set a minimum rate for a TC:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
4@0 8@4 hw 1 mode channel shaper bw_rlimit min_rate 25Gbit 50Gbit
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To set a maximum data rate for a TC:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
4@0 8@4 hw 1 mode channel shaper bw_rlimit max_rate 25Gbit 50Gbit
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To set both minimum and maximum data rates together:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
4@0 8@4 hw 1 mode channel shaper bw_rlimit min_rate 10Gbit 20Gbit
max_rate 25Gbit 50Gbit
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.SS Creating Traffic Class Filters
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
These instructions are not specific to ADQ configuration.
.IP \(bu 2
Disable FW\-LLDP before using tc command to add filter. Refer to the
\fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for more information on disabling
FW\-LLDP.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
After creating traffic classes, use the tc command to create filters for
traffic. Refer to the tc and tc\-flower man pages for more information.
.sp
To view all TC filters:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter show dev <ethX> ingress
tc filter show dev <ethX> egress
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For detailed configuration information, supported fields, and example code for
switchdev mode on Intel Ethernet 800 Series devices, refer to the configuration
guide at
\fI\%https://edc.intel.com/content/www/us/en/design/products/ethernet/appnote\-e810\-eswitch\-switchdev\-mode\-config\-guide/\fP\&.
.sp
TC FILTER EXAMPLES:
.sp
To configure TCP TC filters, where:
.INDENT 0.0
.TP
.B protocol
Encapsulation protocol (valid options are IP and 802.1Q).
.TP
.B prio
Priority.
.TP
.B flower
Flow\-based traffic control filter.
.TP
.B dst_ip
IP address of the device.
.TP
.B ip_proto
IP protocol to use (TCP or UDP).
.TP
.B dst_port
Destination port.
.TP
.B src_port
Source port.
.TP
.B skip_sw
Flag to add the rule only in hardware.
.TP
.B hw_tc
Route incoming traffic flow to this hardware TC. The TC count
starts at 0. For example, \fBhw_tc 1\fP indicates that the filter
is on the second TC.
.TP
.B vlan_id
VLAN ID.
.UNINDENT
.INDENT 0.0
.IP \(bu 2
TCP: Destination IP + L4 Destination Port
.sp
To route incoming TCP traffic with a matching destination IP address and
destination port to the given TC:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <ethX> protocol ip ingress prio 1 flower dst_ip
<ip_address> ip_proto tcp dst_port <port_number> skip_sw hw_tc 1
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
TCP: Source IP + L4 Source Port
.sp
To route outgoing TCP traffic with a matching source IP address and
source port to the given TC associated with the given priority:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <ethX> protocol ip egress prio 1 flower src_ip
<ip_address> ip_proto tcp src_port <port_number> action skbedit priority 1
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
TCP: Destination IP + L4 Destination Port + VLAN Protocol
.sp
To route incoming TCP traffic with a matching destination IP address and
destination port to the given TC using the VLAN protocol (802.1Q):
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <ethX> protocol 802.1Q ingress prio 1 flower
dst_ip <ip address> eth_type ipv4 ip_proto tcp dst_port <port_number>
vlan_id <vlan_id> skip_sw hw_tc 1
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To add a GTP filter:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <ethX> protocol ip parent ffff: prio 1 flower
src_ip 16.0.0.0/16 ip_proto udp dst_port 5678 enc_dst_port 2152
enc_key_id <tunnel_id> skip_sw hw_tc 1
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 2.0
.TP
.B dst_port
inner destination port of application (5678)
.TP
.B enc_dst_port
outer destination port (for GTP user data tunneling occurs
on UDP port 2152)
.TP
.B enc_key_id
tunnel ID (vxlan ID)
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
You can add multiple filters to the device, using the same recipe
(and requires no additional recipe resources), either on the same interface
or on different interfaces. Each filter uses the same fields for matching,
but can have different match values.
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <ethX> protocol ip ingress prio 1 flower ip_proto
tcp dst_port <port_number> skip_sw hw_tc 1

tc filter add dev <ethX> protocol ip egress prio 1 flower ip_proto tcp
src_port <port_number> action skbedit priority 1
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev ens4f0 protocol ip ingress prio 1 flower ip_proto
tcp dst_port 5555 skip_sw hw_tc 1

tc filter add dev ens4f0 protocol ip egress prio 1 flower ip_proto
tcp src_port 5555 action skbedit priority 1
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SS Using TC Filters to Forward to a Queue
.sp
The ice driver supports directing traffic based on L2/L3/L4 fields in the
packet to specific Rx queues, using the TC filter\(aqs class ID.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
This functionality can be used with or without ADQ.
.IP \(bu 2
Disable FW\-LLDP before using tc command to add filter. Refer to the
\fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for more information on disabling
FW\-LLDP.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To add filters for the desired queue, use the following tc command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <ethX> ingress prio 1 protocol all flower src_mac
<mac_address> skip_sw classid ffff:<queue_id>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <mac_address>
the MAC address(es) you want to direct to the Rx queue
.TP
.B <queue_id>
the Rx queue ID number in hexadecimal
.UNINDENT
.sp
For example, to direct a single MAC address to queue 10:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K ens801 hw\-tc\-offload on
tc qdisc add dev ens801 clsact
tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
68:dd:ac:dc:19:00 skip_sw classid ffff:b
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To direct 4 source MAC addresses to Rx queues 10\-13:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K ens801 hw\-tc\-offload on
tc qdisc add dev ens801 clsact
tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
68:dd:ac:dc:19:00 skip_sw classid ffff:b
tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
68:dd:ac:dc:19:01 skip_sw classid ffff:c
tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
68:dd:ac:dc:19:02 skip_sw classid ffff:d
tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
68:dd:ac:dc:19:03 skip_sw classid ffff:e
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Intel(R) Ethernet Flow Director
.sp
The Intel(R) Ethernet Flow Director (Intel(R) Ethernet FD) performs the
following tasks:
.INDENT 0.0
.IP \(bu 2
Directs receive packets according to their flows to different queues
.IP \(bu 2
Enables tight control on routing a flow in the platform
.IP \(bu 2
Matches flows and CPU cores for flow affinity
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
An included script (\fBset_irq_affinity\fP) automates setting the
IRQ to CPU affinity.
.UNINDENT
.UNINDENT
.sp
This driver supports the following flow types:
.INDENT 0.0
.IP \(bu 2
IPv4
.IP \(bu 2
TCPv4
.IP \(bu 2
UDPv4
.IP \(bu 2
SCTPv4
.IP \(bu 2
IPv6
.IP \(bu 2
TCPv6
.IP \(bu 2
UDPv6
.IP \(bu 2
SCTPv6
.UNINDENT
.sp
Each flow type supports valid combinations of IP addresses (source or
destination) and UDP/TCP/SCTP ports (source and destination). You can supply
only a source IP address, a source IP address and a destination port, or any
combination of one or more of these four parameters.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
This driver allows you to filter traffic based on a user\-defined
flexible two\-byte pattern and offset by using the ethtool user\-def and
mask fields. Only L3 and L4 flow types are supported for user\-defined
flexible filters. For a given flow type, you must clear all Intel Ethernet
Flow Director filters before changing the input set (for that flow type).
.UNINDENT
.UNINDENT
.sp
Intel Ethernet Flow Director filters impact only LAN traffic. RDMA filtering
occurs before Intel Ethernet Flow Director, so Intel Ethernet Flow Director
filters will not impact RDMA.
.sp
See the Intel(R) Ethernet Adapters and Devices User Guide for a table that
summarizes supported Intel Ethernet Flow Director features across Intel(R)
Ethernet controllers.
.SS Intel Ethernet Flow Director Filters
.sp
Intel Ethernet Flow Director filters are used to direct traffic that matches
specified characteristics. They are enabled through ethtool\(aqs ntuple interface.
To enable or disable the Intel Ethernet Flow Director and these filters:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> ntuple <off|on>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When you disable ntuple filters, all the user programmed filters are
flushed from the driver cache and hardware. All needed filters must be
re\-added when ntuple is re\-enabled.
.UNINDENT
.UNINDENT
.sp
To display all of the active filters:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-u <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To add a new filter:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U <ethX> flow\-type <type> src\-ip <ip> [m <ip_mask>] dst\-ip <ip>
[m <ip_mask>] src\-port <port> [m <port_mask>] dst\-port <port> [m <port_mask>]
action <queue>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <ethX>
The Ethernet device to program
.TP
.B <type>
Can be ip4, tcp4, udp4, sctp4, ip6, tcp6, udp6, sctp6
.TP
.B <ip>
The IP address to match on
.TP
.B <ip_mask>
The IPv4 address to mask on
.sp
\fBNOTE:\fP
.INDENT 7.0
.INDENT 3.5
These filters use inverted masks. An inverted mask with 0
means exactly match while with 0xF means DON\(aqT CARE. Please
refer to the examples for more details about inverted masks.
.UNINDENT
.UNINDENT
.TP
.B <port>
The port number to match on
.TP
.B <port_mask>
The 16\-bit integer for masking
.sp
\fBNOTE:\fP
.INDENT 7.0
.INDENT 3.5
These filters use inverted masks.
.UNINDENT
.UNINDENT
.TP
.B <queue>
The queue to direct traffic toward (\-1 discards the matched
traffic)
.UNINDENT
.sp
To delete a filter:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U <ethX> delete <N>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where \fB<N>\fP is the filter ID displayed when printing all the active filters,
and may also have been specified using \fBloc <N>\fP when adding the filter.
.sp
EXAMPLES:
.sp
To add a filter that directs packet to queue 2:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U <ethX> flow\-type tcp4 src\-ip 192.168.10.1 dst\-ip \e
192.168.10.2 src\-port 2000 dst\-port 2001 action 2 [loc 1]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To set a filter using only the source and destination IP address:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U <ethX> flow\-type tcp4 src\-ip 192.168.10.1 dst\-ip \e
192.168.10.2 action 2 [loc 1]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To set a filter based on a user\-defined pattern and offset, where the value of
the \fBuser\-def\fP field contains the offset (4 bytes) and the pattern (0xffff):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U <ethX> flow\-type tcp4 src\-ip 192.168.10.1 dst\-ip \e
192.168.10.2 user\-def 0x4FFFF action 2 [loc 1]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To match TCP traffic sent from 192.168.0.1, port 5300, directed to 192.168.0.5,
port 80, and then send it to queue 7:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U enp130s0 flow\-type tcp4 src\-ip 192.168.0.1 dst\-ip 192.168.0.5 \e
src\-port 5300 dst\-port 80 action 7
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To add a TCPv4 filter with a partial mask for a source IP subnet. Here the
matched src\-ip is 192.*.*.* (inverted mask):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U <ethX> flow\-type tcp4 src\-ip 192.168.0.0 m 0.255.255.255 dst\-ip \e
192.168.5.12 src\-port 12600 dst\-port 31 action 12
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
For each flow\-type, the programmed filters must all have the same matching
input set. For example, issuing the following two commands is acceptable:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U enp130s0 flow\-type ip4 src\-ip 192.168.0.1 src\-port 5300 action 7
ethtool \-U enp130s0 flow\-type ip4 src\-ip 192.168.0.5 src\-port 55 action 10
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Issuing the next two commands, however, is not acceptable, since the first
specifies \fBsrc\-ip\fP and the second specifies \fBdst\-ip\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U enp130s0 flow\-type ip4 src\-ip 192.168.0.1 src\-port 5300 action 7
ethtool \-U enp130s0 flow\-type ip4 dst\-ip 192.168.0.5 src\-port 55 action 10
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The second command will fail with an error. You may program multiple filters
with the same fields, using different values, but, on one device, you may not
program two tcp4 filters with different matching fields.
.sp
The ice driver does not support matching on a subportion of a field, thus
partial mask fields are not supported.
.sp
The IPv4 filter type will not match TCP, UDP or SCTP traffic. To match those
types of traffic, create separate filters for TCP, UDP and SCTP as desired
or use a different type of filtering.
.UNINDENT
.UNINDENT
.SS Flex Byte Intel Ethernet Flow Director Filters
.sp
The driver also supports matching user\-defined data within the packet payload.
This flexible data is specified using the \fBuser\-def\fP field of the ethtool
command in the following way:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
| 31    28    24    20    16 | 15    12    8    4    0  |
+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
| offset into packet payload | 2 bytes of flexible data |
+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
\&... user\-def 0x4FFFF ...
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
tells the filter to look 4 bytes into the payload and match that value against
0xFFFF. The offset is based on the beginning of the payload, and not the
beginning of the packet. Thus:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
flow\-type tcp4 ... user\-def 0x8BEAF ...
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
would match TCP/IPv4 packets which have the value 0xBEAF 8 bytes into the
TCP/IPv4 payload.
.sp
Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload.
Thus to match the first byte of the payload, you must actually add 4 bytes to
the offset. Also note that ip4 filters match both ICMP frames as well as raw
(unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame.
.sp
The maximum offset is 64. The hardware will only read up to 64 bytes of data
from the payload. The offset must be even because the flexible data is 2 bytes
long and must be aligned to byte 0 of the packet payload.
.sp
The user\-defined flexible offset is also considered part of the input set and
cannot be programmed separately for multiple filters of the same type. However,
the flexible data is not part of the input set and multiple filters may use the
same offset but match against different data.
.SS RSS Hash Flow
.sp
Allows you to set the hash bytes per flow type and any combination of one or
more options for Receive Side Scaling (RSS) hash byte configuration.
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-N <ethX> rx\-flow\-hash <type> <option>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where \fB<type>\fP is:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B tcp4
signifying TCP over IPv4
.TP
.B udp4
signifying UDP over IPv4
.TP
.B tcp6
signifying TCP over IPv6
.TP
.B udp6
signifying UDP over IPv6
.UNINDENT
.UNINDENT
.UNINDENT
.sp
And \fB<option>\fP is one or more of:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B s
Hash on the IP source address of the Rx packet.
.TP
.B d
Hash on the IP destination address of the Rx packet.
.TP
.B f
Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet.
.TP
.B n
Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
For example, to hash on the source and destination IP address for TCP IPv4
traffic, use the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-N <ethX> rx\-flow\-hash tcp4 sd
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To hash on the source and destination ports for UDP IPv6 traffic, use the
following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-N <ethX> rx\-flow\-hash udp6 sdfn
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Accelerated Receive Flow Steering (aRFS)
.sp
Devices based on the Intel(R) Ethernet 800 Series support Accelerated Receive
Flow Steering (aRFS) on the PF. aRFS is a load\-balancing mechanism that allows
you to direct packets to the same CPU where an application is running or
consuming the packets in that flow.
.INDENT 0.0
.IP \(bu 2
aRFS requires that ntuple filtering is enabled via ethtool.
.IP \(bu 2
aRFS support is limited to the following packet types:
.INDENT 2.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
TCP over IPv4 and IPv6
.IP \(bu 2
UDP over IPv4 and IPv6
.IP \(bu 2
Nonfragmented packets
.UNINDENT
.UNINDENT
.UNINDENT
.IP \(bu 2
aRFS only supports Intel Ethernet Flow Director filters, which consist of the
source/destination IP addresses and source/destination ports.
.IP \(bu 2
aRFS and ethtool\(aqs ntuple interface both use the device\(aqs Intel Ethernet Flow
Director. aRFS and ntuple features can coexist, but you may encounter
unexpected results if there\(aqs a conflict between aRFS and ntuple requests.
See \fI\%Intel(R) Ethernet Flow Director\fP for additional information.
.UNINDENT
.sp
To set up aRFS:
.INDENT 0.0
.IP 1. 3
Enable the Intel Ethernet Flow Director and ntuple filters using ethtool:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> ntuple on
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Set up the number of entries in the global flow table. For example:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
NUM_RPS_ENTRIES=16384
echo $NUM_RPS_ENTRIES > /proc/sys/net/core/rps_sock_flow_entries
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
Set up the number of entries in the per\-queue flow table. For example:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
NUM_RX_QUEUES=64
for file in /sys/class/net/$IFACE/queues/rx\-*/rps_flow_cnt; do
echo $(($NUM_RPS_ENTRIES/$NUM_RX_QUEUES)) > $file;
done
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 4. 3
Disable the IRQ balance daemon (this is only a temporary stop of the service
until the next reboot):
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
systemctl stop irqbalance
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 5. 3
Configure the interrupt affinity:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
set_irq_affinity <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To disable aRFS using ethtool:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> ntuple off
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
This command will disable ntuple filters and clear any aRFS filters
in software and hardware.
.UNINDENT
.UNINDENT
.sp
Example Use Case:
.INDENT 0.0
.IP 1. 3
Set the server application on the desired CPU (e.g., CPU 4):
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
taskset \-c 4 netserver
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Use netperf to route traffic from the client to CPU 4 on the server with
aRFS configured. This example uses TCP over IPv4:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
netperf \-H <Host IPv4 Address> \-t TCP_STREAM
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.SS Enabling Virtual Functions (VFs) for SR\-IOV
.sp
Use sysfs to enable virtual functions (VF).
.sp
For example, you can create 4 VFs as follows:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 4 > /sys/class/net/<ethX>/device/sriov_numvfs
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To disable VFs, write 0 to the same file:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/class/net/<ethX>/device/sriov_numvfs
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The maximum number of VFs for the ice driver is 256 total (all ports). To check
how many VFs each PF supports, use the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/class/net/<ethX>/device/sriov_totalvfs
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The VF driver will not block teaming/bonding/link aggregation, but this is not
a supported feature. Do not expect failover or load balancing on the VF
interface.
.SS SR\-IOV Live Migration
.sp
You can use VFIO Device Migration to move an active virtual machine (VM)
between different physical machines so it does not lose its network connection.
After migrating, the virtual function (VF) will continue most Ethernet
operations without further interruption. During migration, data and VIRTCHNL
operations are sent to a buffer so the can br recreated when the migration
completes. If the memory allocated for the command buffer is exceeded, the
system will drop the buffer and disable the live migration capability for the
VF. You must reset the VF for live migration to be re\-enabled.
.INDENT 0.0
.IP \(bu 2
Live migration requires kernel version 5.15 to 5.17
.IP \(bu 2
You cannot migrate a VM if it has a VF that is using RDMA.
.IP \(bu 2
You can only migrate the VF to a device in the same family with a similar
firmware version. For example, you can migrate a VF from one 810 device to
another, but not from an 810 device to an 820 device.
.IP \(bu 2
Any VF properties that are set by the PF will not be migrated. Make sure that
both devices have the same PF\-set properties.
.UNINDENT
.sp
Refer to \fI\%https://qemu.readthedocs.io/en/latest/devel/vfio\-migration.html\fP for
more details.
.SS Displaying VF Statistics on the PF
.sp
Use the following command to display the statistics for the PF and its VFs:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip \-s link show dev <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
The output of this command can be very large due to the maximum number
of possible VFs.
.UNINDENT
.UNINDENT
.sp
The PF driver will display a subset of the statistics for the PF and for all
VFs that are configured. The PF will always print a statistics block for each
of the possible VFs, and it will show zero for all unconfigured VFs.
.SS Configuring VLAN Tagging on SR\-IOV Enabled Adapter Ports
.sp
To configure VLAN tagging for the ports on an SR\-IOV enabled adapter, use the
following command. The VLAN configuration should be done before the VF driver
is loaded or the VM is booted. The VF is not aware of the VLAN tag being
inserted on transmit and removed on received frames (sometimes called \(dqport
VLAN\(dq mode).
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set dev <ethX> vf <id> vlan <vlan id>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example, the following will configure PF eth0 and the first VF on VLAN 10:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set dev eth0 vf 0 vlan 10
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Enabling a VF Link If the Port Is Disconnected
.sp
If the physical function (PF) link is down, you can force link up (from the
host PF) on any virtual functions (VF) bound to the PF. Note that this requires
kernel support (Red Hat kernel 3.10.0\-327 or newer, upstream kernel 3.11.0 or
newer) and associated iproute2 user space support.
.sp
For example, to force link up on VF 0 bound to PF eth0:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set eth0 vf 0 state enable
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
If the command does not work, it may not be supported by your system.
.UNINDENT
.UNINDENT
.SS Setting the MAC Address for a VF
.sp
To change the MAC address for the specified VF:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set <ethX> vf 0 mac <address>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set <ethX> vf 0 mac 00:01:02:03:04:05
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
This setting lasts until the PF is reloaded.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
For untrusted VFs, assigning a MAC address for a VF from the host
will disable any subsequent requests to change the MAC address from within
the VM. This is a security feature. The VM is not aware of this restriction,
so if this is attempted in the VM, it will trigger MDD events. Trusted VFs
are allowed to change the MAC address from within the VM.
.UNINDENT
.UNINDENT
.SS Trusted VFs and VF Promiscuous Mode
.sp
This feature allows you to designate a particular VF as trusted and allows that
trusted VF to request selective promiscuous mode on the Physical Function (PF).
.sp
To set a VF as trusted or untrusted, enter the following command in the
Hypervisor:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set dev <ethX> vf 1 trust [on|off]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
It\(aqs important to set the VF to trusted before setting promiscuous
mode. If the VM is not trusted, the PF will ignore promiscuous mode requests
from the VF. If the VM becomes trusted after the VF driver is loaded, you
must make a new request to set the VF to promiscuous.
.UNINDENT
.UNINDENT
.sp
Once the VF is designated as trusted, use the following commands in the VM to
set the VF to promiscuous mode.
.INDENT 0.0
.IP \(bu 2
For promiscuous all, where \fB<ethX>\fP is a VF interface in the VM:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set <ethX> promisc on
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
For promiscuous multicast, where \fB<ethX>\fP is a VF interface in the VM:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set <ethX> allmulticast on
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
By default, the ethtool private flag \fBvf\-true\-promisc\-support\fP is
set to \(dqoff,\(dq meaning that promiscuous mode for the VF will be limited. To
set the promiscuous mode for the VF to true promiscuous and allow the VF to
see all ingress traffic, use the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> vf\-true\-promisc\-support on
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The \fBvf\-true\-promisc\-support\fP private flag does not enable promiscuous mode;
rather, it designates which type of promiscuous mode (limited or true) you will
get when you enable promiscuous mode using the \fBip link\fP commands above. You
can toggle the \fBvf\-true\-promisc\-support\fP flag separately for all PFs.
.sp
Next, add a VLAN interface on the VF interface. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link add link eth2 name eth2.100 type vlan id 100
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Note that the order in which you set the VF to promiscuous mode and add the
VLAN interface does not matter (you can do either first). The result in this
example is that the VF will get all traffic that is tagged with VLAN 100.
.SS LLDP Support on the VF
.sp
The ice driver supports the Link Layer Discovery Protocol (LLDP) on the VF.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
You must disable the firmware\-based LLDP agent on the port to use
LLDP packets on the VF. See \fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for
how to disable the FW\-LLDP Agent.
.UNINDENT
.UNINDENT
.sp
When the FW\-LLDP Agent is disabled:
.INDENT 0.0
.IP \(bu 2
The driver allows a trusted VF to configure L2 filters containing an LLDP
multicast address. See \fI\%Trusted VFs and VF Promiscuous Mode\fP in this README for how to set a VF
as trusted.
.IP \(bu 2
In switchdev mode, you can use the tc\-flower command to configure L2 filters
containing an LLDP multicast address. See \fI\%Switchdev Mode\fP in this
README for more information.
.IP \(bu 2
The ice driver uses the \fBtransmit_lldp\fP parameter in sysfs to enable a VF
to transmit LLDP packets.
.UNINDENT
.sp
Only a single VF per port is allowed to transmit LLDP packets. The PF is not
allowed to transmit LLDP packets.
.sp
To enable LLDP transmit on the VF, use the following command on the
PF:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 1 > /sys/bus/pci/devices/<VF\(aqs PCI device ID>/transmit_lldp
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 1 > /sys/bus/pci/devices/0000:ad:01.0/transmit_lldp
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To enable \fBtransmit_lldp\fP on a different port, you must first disable it on
the original port. For example, if it is enabled on \fB0000:ad:01.0\fP but you
want to want to change it to port \fB0000:ad:01.1\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 1 > /sys/bus/pci/devices/0000:ad:01.0/transmit_lldp
echo 0 > /sys/bus/pci/devices/0000:ad:01.0/transmit_lldp  # disables it on this port
echo 1 > /sys/bus/pci/devices/0000:ad:01.1/transmit_lldp
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Virtual Function (VF) Tx Rate Limit
.sp
Use the ip command to configure the maximum or minimum Tx rate limit for a VF
from the PF interface.
.sp
For example, to set a maximum Tx rate limit of 8000Mbps for VF 0:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set eth0 vf 0 max_tx_rate 8000
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example, to set a minimum Tx rate limit of 1000Mbps for VF 0:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set eth0 vf 0 min_tx_rate 1000
.ft P
.fi
.UNINDENT
.UNINDENT
.INDENT 0.0
.IP \(bu 2
If DCB or ADQ are enabled on a PF, you cannot set a minimum Tx rate on the
VFs associated with that PF.
.IP \(bu 2
If both DCB and ADQ are disabled on a PF, then you can set a minimum Tx rate
on the VFs associated with that PF.
.IP \(bu 2
If you set a minimum Tx rate limit on a PF for SR\-IOV VFs and then apply a
DCB or ADQ configuration, the PF cannot guarantee the minimum Tx rate limits
for those VFs.
.IP \(bu 2
If you set a minimum Tx rate on VFs across multiple ports that have an
aggregate bandwidth over 100Gbps, the PFs cannot guarantee the minimum Tx
rate set for the VFs.
.UNINDENT
.SS Malicious Driver Detection (MDD) for VFs
.sp
Some Intel Ethernet devices use Malicious Driver Detection (MDD) to detect
malicious traffic from the VF and disable Tx/Rx queues or drop the offending
packet until a VF driver reset occurs. You can view MDD messages in the PF\(aqs
system log using the dmesg command.
.INDENT 0.0
.IP \(bu 2
If the PF driver logs MDD events from the VF, confirm that the correct VF
driver is installed.
.IP \(bu 2
To restore functionality, you can manually reload the VF or VM or enable
automatic VF resets.
.IP \(bu 2
When automatic VF resets are enabled, the PF driver will immediately reset
the VF and re\-enable queues when it detects MDD events on either the receive
or transmit path.
.IP \(bu 2
If automatic VF resets are disabled, the PF will not automatically reset the
VF when it detects MDD events.
.UNINDENT
.sp
To enable or disable automatic VF resets, use the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> mdd\-auto\-reset\-vf on|off
.ft P
.fi
.UNINDENT
.UNINDENT
.SS MAC and VLAN Anti\-Spoofing Feature for VFs
.sp
When a malicious driver on a Virtual Function (VF) interface attempts to send a
spoofed packet, it is dropped by the hardware and not transmitted.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
This feature can be disabled for a specific VF:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set <ethX> vf <vf id> spoofchk {off|on}
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SS VLAN Pruning
.sp
The ice driver allows you to enable or disable VLAN pruning for the VF VSI
using the ethtool private flag \fBvf\-vlan\-pruning\fP\&.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
You cannot change this private flag while any VFs are active.
.IP \(bu 2
If a port VLAN is configured, VLAN pruning will always be enabled.
.IP \(bu 2
When VLAN pruning is enabled, the interface will:
.INDENT 2.0
.IP \(bu 2
Discard all packets with a VLAN tag when Rx VLAN filtering is disabled.
.IP \(bu 2
Discard untagged packets when Rx VLAN filtering is enabled.
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To disable or enable VLAN pruning on all VFs, do the following:
.INDENT 0.0
.IP 1. 3
Deinitialize any VFs.
.IP 2. 3
On the PF, use the following command:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> vf\-vlan\-pruning on|off
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 3.0
.TP
.B on
Enables VLAN pruning
.TP
.B off
Disables VLAN pruning (default)
.UNINDENT
.IP 3. 3
Initialize and configure any VFs.
.UNINDENT
.sp
VLAN pruning will then be disabled or enabled on any of these VFs, depending on
the flag you set.
.SS Flexible VF Loopback Pacing
.sp
The ice driver supports adjusting the loopback rate for a designated port,
which allows you to prioritize that port for maximum bandwidth and achieve
higher speeds.
.sp
Use the devlink command and the \fBloopback\fP parameter to change this setting.
After changing the \fBloopback\fP parameter, the driver will reconfigure all
underlying VFs to align to the desired port settings and add more bandwidth
to VF\-to\-VF traffic. Setting this parameter to \fBprioritized\fP enables higher
hairpin\-bandwidth on related PFs.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
This configuration applies only for 8x10G and 4x25G adapter cards.
.IP \(bu 2
Typically you would set some ports to prioritized loopback and then
disable loopback on other ports, to allow the driver to utilize spare
bandwidth for VF\-to\-VF traffic.
.IP \(bu 2
Intel recommends using the \fBprioritized\fP loopback setting on a port
with minimal network traffic.
.IP \(bu 2
You should first configure loopback on the PF and then configure any
other settings, such as VFs/VMs or assigning MAC addresses.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To change the loopback setting, use the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set <pci/D:b:d.f> name loopback value <setting> \e
      cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <pci/D:b:d.f>
The PCI address of the device (pci/Domain:bus:device.function).
.TP
.B <setting>
The desired setting for the \fBloopback\fP parameter.
Supported values are:
.INDENT 7.0
.TP
.B enabled
Loopback traffic is allowed on the designated port
(default).
.TP
.B disabled
Loopback traffic is not allowed on the designated
port.
.TP
.B prioritized
Loopback traffic is prioritized on the designated
port. \fBNote\fP: This value is not supported on
single port adapters.
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set pci/0000:b2:00.3 name loopback value prioritized cmode runtime

devlink dev param set pci/0000:b2:00.3 name loopback value enabled cmode runtime

devlink dev param set pci/0000:b2:00.3 name loopback value disabled cmode runtime
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Switchdev Mode
.sp
The PF driver supports legacy and switchdev eSwitch modes. Switchdev mode
allows the driver to create additional port representor netdevs that enable a
control plane running on the host to configure filters for the VFs and also
handle default/exception traffic from the uplink and the VFs.
.sp
The driver loads in legacy mode by default. You can configure eSwitch modes
independently per physical port using the devlink command. You can change
between eSwitch modes only if no VFs have been created. If SR\-IOV is enabled
and VFs are bound to the PF, you must do the following before changing between
switchdev and legacy mode:
.INDENT 0.0
.IP \(bu 2
Unload all VFs that were bound
.IP \(bu 2
Set the number of VFs on the PF to zero
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
ADQ, trusted VFs, and L2 forwarding are not supported in switchdev
mode.
.IP \(bu 2
Switchdev mode is not persistent across reboots or driver reloads.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To configure the device in switchdev mode, enter the following, where
\fB<pci/0000:##:##.#>\fP is the PCI address of the PF:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev eswitch set <pci/0000:##:##.#> mode switchdev
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev eswitch set pci/0000:17:00.0 mode switchdev
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To configure the device in legacy mode:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev eswitch set <pci/0000:##:##.#> mode legacy
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To check the current eSwitch mode:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev eswitch show <pci/0000:##:##.#>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The ice driver supports the following hardware offloads in switchdev mode:
.INDENT 0.0
.IP \(bu 2
Supported filter conditions:
.INDENT 2.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
L2: Source/Destination MAC addresses, VLAN ID
.IP \(bu 2
L3: Source/Destination IP addresses (IPv4, IPv6), IP protocol (TCP, UDP),
ToS (IPv4), Traffic Class (IPv6), TTL (IPv4), PPPoE (IPv4, IPv6, TCP, UDP)
.IP \(bu 2
L4: Source and Destination port, L2TPv3 (IP)
.IP \(bu 2
VXLAN/GRETAP/GENEVE: VNI/GRE Key, Outer Destination IP, Inner Source IP,
Inner Destination IP, Inner Destination MAC, TCP/UDP Source port and
Destination port
.IP \(bu 2
GTP: TEID, PDU type, QFI, Outer Destination IP, Outer Source IP
.IP \(bu 2
PPPoE: session ID, Protocol
.IP \(bu 2
L2TPv3: session ID
.UNINDENT
.UNINDENT
.UNINDENT
.IP \(bu 2
Supported filter actions: redirect, drop, mirror
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
L2TPv3 over UDP is not supported.
.IP \(bu 2
GTP, L2TPv3, and PPPoE are only supported with a DDP package that supports
these protocols, such as the Comms package.
.IP \(bu 2
GTP support requires kernel 5.18 and iproute2 5.18 or newer. On older
kernel versions, the DCF method provides the same functionality.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
For detailed configuration information, supported fields, and example code for
switchdev mode on Intel Ethernet 800 Series devices, refer to the configuration
guide at
\fI\%https://edc.intel.com/content/www/us/en/design/products/ethernet/appnote\-e810\-eswitch\-switchdev\-mode\-config\-guide/\fP\&.
.sp
At a high level, do the following to offload TC filters to the hardware and
create switch rules in switchdev mode:
.INDENT 0.0
.IP 1. 3
Verify that switchdev mode is enabled.
.IP 2. 3
Enable \fBhw\-tc\-offload\fP on the VF port representor (VF_PR).
.IP 3. 3
For tunnel interfaces: Use the \fBip link\fP command to create the tunnel.
.IP 4. 3
Use the tc\-flower command to create the switch rule.
.IP 5. 3
Verify the offloaded flow in hardware.
.UNINDENT
.sp
Switchdev mode supports the following \fBip link\fP commands to configure the VF:
.INDENT 0.0
.IP \(bu 2
mac
.IP \(bu 2
vlan, vxlan, geneve, gre, nvgre, gtp, qos, proto
.IP \(bu 2
max_tx_rate
.IP \(bu 2
min_tx_rate
.IP \(bu 2
spoofchk
.IP \(bu 2
query_rss
.IP \(bu 2
state
.IP \(bu 2
node_guid
.IP \(bu 2
port_guid
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
\fBtrust\fP is not supported. \fBrate\fP is supported but deprecated; use
\fBmax_tx_rate\fP instead.
.UNINDENT
.UNINDENT
.sp
To limit the VF\(aqs interrupt rate for Rx and Tx in switchdev mode, use the
following command, where \fB<vf_pr>\fP is the designated VF port representor and
\fB<N>\fP is the desired cap for the interrupt rate:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-C <vf_pr> rx\-usecs\-high <N>
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Traffic Mirroring of VF
.sp
The ice driver supports traffic mirroring (also known as port mirroring), for
both ingress and egress traffic. This feature allows network traffic entering
and leaving a VF to be duplicated and sent to another specified VF that resides
on the same PF.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
This feature is only supported in switchdev mode. See
\fI\%Switchdev Mode\fP in this README for more information.
.IP \(bu 2
Disable FW\-LLDP before using tc command to add filter. Refer to the
\fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for more information on disabling
FW\-LLDP.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Use the tc\-mirred command to configure the mirror rules. For example, to add
a minimal filter:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
tc filter add dev <VF1_PR> ingress flower skip_sw action mirred egress mirror dev <VF2_PR>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.IP \(bu 2
\fB<VF1_PR>\fP and \fB<VF2_PR>\fP are the port representor netdevs for VF1 and VF2.
.IP \(bu 2
See \fI\%Creating Traffic Class Filters\fP in this README for an explanation of \fBskip_sw\fP
and \fBingress flower\fP\&.
.UNINDENT
.SS Switching Modes
.sp
Devices based on the Intel(R) Ethernet 800 Series support Virtual Ethernet
Bridging (VEB) and Virtual Ethernet Port Aggregator (VEPA) switching modes.
.sp
In Virtual Ethernet Bridging (VEB) switching mode:
.INDENT 0.0
.IP \(bu 2
Functionality: VEB acts as an internal switch within the Intel Ethernet
800 Series, managing the interconnectivity and traffic flow between various
Virtual Functions (VFs) on the same physical device.
.IP \(bu 2
Network Topology and Loopback: It is VLAN\-aware, capable of segregating and
routing traffic based on VLAN tags. Importantly, VEB handles loopback traffic
directly on the network controller, allowing for efficient internal
communication without needing to send traffic out of the host system and
back in.
.IP \(bu 2
Use Case Example: In a scenario with multiple virtual machines (VMs) on a
single server, each assigned a different VF, VEB facilitates direct, efficient
communication between these VMs at the hardware level. This is particularly
beneficial for applications requiring low latency and high\-speed internal
data transfer.
.UNINDENT
.sp
In Virtual Ethernet Port Aggregator (VEPA) switching mode:
.INDENT 0.0
.IP \(bu 2
Functionality: VEPA, in contrast, forwards all inter\-VF traffic to an external
network switch, relying on this external entity for traffic management and
routing.
.IP \(bu 2
Network Topology: It is typically used in environments where centralized
control and analysis of traffic, such as for security or policy enforcement,
are conducted at an external switch.
.IP \(bu 2
Use Case Example: In data centers where external traffic monitoring and
policy enforcement are essential, VEPA enables the aggregation of traffic
from various VFs to an external switch, which then manages routing,
monitoring, and policy application.
.UNINDENT
.sp
Key differences between VEB and VEPA:
.INDENT 0.0
.IP 1. 3
Traffic Management: VEB provides efficient internal traffic management
between VFs and handles loopback on the Network Controller. VEPA, in
contrast, depends on external devices for managing and routing inter\-VF
traffic.
.IP 2. 3
Efficiency vs. Control: VEB is more efficient for internal traffic and
loopback scenarios, while VEPA offers advantages in centralized external
control and monitoring of traffic.
.IP 3. 3
Application Use Cases: VEB is suited for environments needing high\-speed,
low\-latency internal communication, such as in dense VM deployments. VEPA
is preferred in scenarios where external traffic analysis and policy
enforcement are prioritized.
.UNINDENT
.sp
Use the following commands to set and show the hardware switch mode:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
bridge link set dev <ethX> hwmode {vepa|veb}
bridge link show dev <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Jumbo Frames
.sp
Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
to a value larger than the default value of 1500.
.sp
Use the ip command to increase the MTU size. For example, enter the following
where \fB<ethX>\fP is the interface number:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link set mtu 9000 dev <ethX>
ip link set up dev <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
This setting is not saved across reboots.
.sp
Add \fBMTU=9000\fP to the following file to make the setting change permanent:
.INDENT 0.0
.IP \(bu 2
For RHEL: \fB/etc/sysconfig/network\-scripts/ifcfg\-<ethX>\fP
.IP \(bu 2
For SLES: \fB/etc/sysconfig/network/<config_file>\fP
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
The maximum MTU setting for jumbo frames is 9702. This corresponds to
the maximum jumbo frame size of 9728 bytes.
.IP \(bu 2
This driver will attempt to use multiple page sized buffers to receive
each jumbo packet. This should help to avoid buffer starvation issues when
allocating receive packets.
.IP \(bu 2
Packet loss may have a greater impact on throughput when you use jumbo
frames. If you observe a drop in performance after enabling jumbo frames,
enabling flow control may mitigate the issue.
.UNINDENT
.UNINDENT
.UNINDENT
.SS Speed and Duplex Configuration
.sp
You cannot set speed, duplex, or autonegotiation settings using ethtool.
.sp
To see the speed configurations your device supports, run the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To have your device advertise supported speeds, use the following, where \fBN\fP
is a bitmask of the desired speeds:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-s <ethX> advertise N
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example, to have your device advertise 10000baseSR Full, use:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-s <ethX> advertise 0x80000000000
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For more details, please refer to the ethtool man page.
.SS Data Center Bridging (DCB)
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
The kernel assumes that TC0 is available, and will disable Priority
Flow Control (PFC) on the device if TC0 is not available. To fix this,
ensure TC0 is enabled when setting up DCB on your switch.
.UNINDENT
.UNINDENT
.sp
DCB is a configuration Quality of Service implementation in hardware. It uses
the VLAN priority tag (802.1p) to filter traffic. That means that there are 8
different priorities that traffic can be filtered into. It also enables
priority flow control (802.1Qbb) which can limit or eliminate the number of
dropped packets during network stress. Bandwidth can be allocated to each of
these priorities, which is enforced at the hardware level (802.1Qaz).
.sp
DCB is normally configured on the network using the DCBX protocol (802.1Qaz), a
specialization of LLDP (802.1AB). The ice driver supports the following
mutually exclusive variants of DCBX support:
.INDENT 0.0
.IP \(bu 2
Firmware\-based LLDP Agent
.IP \(bu 2
Software\-based LLDP Agent
.UNINDENT
.sp
In firmware\-based mode, firmware intercepts all LLDP traffic and handles DCBX
negotiation transparently for the user. In this mode, the adapter operates in
\(dqwilling\(dq DCBX mode, receiving DCB settings from the link partner (typically a
switch). The local user can only query the negotiated DCB configuration. For
information on configuring DCBX parameters on a switch, please consult the
switch manufacturer\(aqs documentation.
.sp
In software\-based mode, LLDP traffic is forwarded to the network stack and user
space, where a software agent can handle it. In this mode, the adapter can
operate in either \(dqwilling\(dq or \(dqnonwilling\(dq DCBX mode and DCB configuration can
be both queried and set locally. This mode requires the FW\-based LLDP Agent to
be disabled.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
You can enable and disable the firmware\-based LLDP Agent using an ethtool
private flag. Refer to the \fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for more
information.
.IP \(bu 2
In software\-based DCBX mode, you can configure DCB parameters using
software LLDP/DCBX agents that interface with the Linux kernel\(aqs DCB
Netlink API. We recommend using OpenLLDP as the DCBX agent when running
in software mode. For more information, see the OpenLLDP man pages and
\fI\%https://github.com/intel/openlldp\fP\&.
.IP \(bu 2
The driver implements the DCB netlink interface layer to allow the user
space to communicate with the driver and query DCB configuration for the
port.
.IP \(bu 2
iSCSI with DCB is not supported.
.UNINDENT
.UNINDENT
.UNINDENT
.SS L3 QoS mode
.sp
The ice driver supports setting DSCP\-based Layer 3 Quality of Service (L3 QoS)
in the PF driver. The driver initializes in L2 QoS mode. L3 QoS mode is:
.INDENT 0.0
.IP \(bu 2
Automatically enabled when the first DSCP/ToS to TC mapping is defined
.IP \(bu 2
Automatically disabled when the last DSCP/ToS to TC mapping is removed
.UNINDENT
.sp
The following is an example of how to map a DSCP/ToS to a TC:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
lldptool \-T \-i <ethX> \-V APP app=<prio>,<sel>,<pid>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
where:
.INDENT 0.0
.TP
.B <prio>
The TC assigned to the DSCP/ToS code point
.TP
.B <sel>
5 for DSCP to TC mapping
.TP
.B <pid>
The DSCP/ToS code point
.UNINDENT
.sp
For example, to map packets containing DSCP value 63 to traffic class 0 on
interface eth0:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
lldptool \-T \-i eth0 \-V APP app=63,5,0
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To remove a mapping, use the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
lldptool \-T \-I <ethX> \-V APP \-d app=<prio>,<sel>,<pid>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To view the currently configured mappings, use the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
lldptool \-t \-i <ethX> \-V APP \-c
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
L3 QoS mode is not available when FW\-LLDP is enabled. You also cannot
enable FW\-LLDP if L3 QoS mode is active. Disable FW\-LLDP before switching
to L3 QoS mode. Refer to the \fI\%FW\-LLDP (Firmware Link Layer Discovery Protocol)\fP section in this README for
more information on disabling FW\-LLDP.
.IP \(bu 2
Once a mapping has been submitted for a DSCP value, another mapping for
that value will not be accepted until the first one has been deleted.
.UNINDENT
.UNINDENT
.UNINDENT
.SS FW\-LLDP (Firmware Link Layer Discovery Protocol)
.sp
Use ethtool to change FW\-LLDP settings. The FW\-LLDP setting is per port and
persists across boots.
.sp
To enable LLDP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> fw\-lldp\-agent on
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To disable LLDP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> fw\-lldp\-agent off
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To check the current LLDP setting:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-show\-priv\-flags <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
You must enable the UEFI HII \fBLLDP Agent\fP attribute for this
setting to take effect. If \fBLLDP AGENT\fP is set to disabled, you cannot
enable it from the OS.
.UNINDENT
.UNINDENT
.SS Forward Error Correction (FEC)
.sp
Allows you to set the Forward Error Correction (FEC) mode. FEC improves link
stability, but increases latency. Many high quality optics, direct attach
cables, and backplane channels provide a stable link without FEC.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
For devices to benefit from this feature, link partners must have
FEC enabled.
.UNINDENT
.UNINDENT
.sp
If you enable the flag \fBallow\-no\-fec\-modules\-in\-auto\fP, Auto FEC negotiation
will include \fBNo FEC\fP in case your link partner does not have FEC enabled or
is not FEC capable:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-set\-priv\-flags <ethX> allow\-no\-fec\-modules\-in\-auto on
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
On kernels older than 4.14, use the following private flags to disable
FEC modes:
.INDENT 0.0
.TP
.B rs\-fec
0 to disable, 1 to enable
.TP
.B base\-r\-fec
0 to disable, 1 to enable
.UNINDENT
.sp
On kernel 4.14 or later, use ethtool to get/set the following FEC modes:
.INDENT 0.0
.IP \(bu 2
No FEC
.IP \(bu 2
Auto FEC
.IP \(bu 2
BASE\-R FEC
.IP \(bu 2
RS\-FEC
.UNINDENT
.SS Link\-Level Flow Control (LFC)
.sp
Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enable
receiving and transmitting pause frames for ice. When transmit is enabled,
pause frames are generated when the receive packet buffer crosses a predefined
threshold. When receive is enabled, the transmit unit will halt for the time
delay specified when a pause frame is received.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
You must have a flow control capable link partner.
.UNINDENT
.UNINDENT
.sp
Flow Control is disabled by default.
.sp
Use ethtool to change the flow control settings.
.sp
To enable or disable Rx or Tx Flow Control:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-A <ethX> rx <on|off> tx <on|off>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
This command only enables or disables Flow Control if auto\-negotiation
is disabled. If auto\-negotiation is enabled, this command changes the
parameters used for auto\-negotiation with the link partner.
.UNINDENT
.UNINDENT
.sp
Flow Control auto\-negotiation is part of link auto\-negotiation. Depending
on your device, you may not be able to change the auto\-negotiation setting.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
The ice driver requires flow control on both the port and link partner.
If flow control is disabled on one of the sides, the port may appear to
hang on heavy traffic.
.IP \(bu 2
You may encounter issues with link\-level flow control (LFC) after disabling
DCB. The LFC status may show as enabled but traffic is not paused. To
resolve this issue, disable and reenable LFC using ethtool:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-A <ethX> rx off tx off
ethtool \-A <ethX> rx on tx on
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SS Limiting the Maximum Bitrate for a Transmit Queue
.sp
The ice driver supports limiting the transmit queue bit rate with the
\fBtx_maxrate\fP sysfs entry. Use this entry to set a maximum bitrate in Mbps.
A value of zero means no limiting.
.sp
For example, to set the bit rate for transmit queue 1 to 300 Mbps:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 300 > /sys/class/net/<ethx>/queues/tx\-1/tx_maxrate
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To remove the limit:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/class/net/<ethx>/queues/tx\-1/tx_maxrate
.ft P
.fi
.UNINDENT
.UNINDENT
.SS NAPI
.sp
This driver supports NAPI (Rx polling mode). For more information on NAPI, see
\fI\%https://docs.kernel.org/networking/napi.html\fP\&.
.SS MACVLAN
.sp
This driver supports MACVLAN. Kernel support for MACVLAN can be tested by
checking if the MACVLAN driver is loaded. You can run \fBlsmod | grep macvlan\fP
to see if the MACVLAN driver is loaded or run \fBmodprobe macvlan\fP to try to
load the MACVLAN driver.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
In passthru mode, you can only set up one MACVLAN device. It will
inherit the MAC address of the underlying PF (Physical Function) device.
.UNINDENT
.UNINDENT
.sp
ice devices support L2 Forwarding Offload. This will offload the processing
required for L2 Forwarding from the system processors to the ice device.
Perform the following steps to enable L2 Forwarding Offload:
.INDENT 0.0
.IP 1. 3
Enable L2 Forwarding offload:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> l2\-fwd\-offload on
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Create the MACVLAN netdevs and bind them to the PF.
.IP 3. 3
Bring up/enable the MACVLAN netdevs.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
MACVLAN offloads and ADQ are mutually exclusive. System instability
may occur if you enable \fBl2\-fwd\-offload\fP and then set up ADQ, or if you
set up ADQ and then enable \fBl2\-fwd\-offload\fP\&.
.UNINDENT
.UNINDENT
.SS IEEE 802.1ad (QinQ) Support
.sp
The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
\(dqtags,\(dq and multiple VLAN IDs are thus referred to as a \(dqtag stack.\(dq Tag stacks
allow L2 tunneling and the ability to separate traffic within a particular VLAN
ID, among other uses.
.sp
The following are examples of how to configure 802.1ad (QinQ), where \(dq24\(dq and
\(dq371\(dq are example VLAN IDs:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
802.1ad (QinQ) is supported in 3.19 and later kernels.
.IP \(bu 2
VLAN protocols use the following EtherTypes:
.INDENT 2.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
802.1Q = EtherType 0x8100
.IP \(bu 2
802.1ad = EtherType 0x88A8
.UNINDENT
.UNINDENT
.UNINDENT
.IP \(bu 2
For QinQ traffic to work at MTU 1500, the L2 peer (switch port or another
NIC) should be able to receive Ethernet frames of 1526 bytes. Some
third\-party NICs support a maximum Ethernet frame size of 1522 bytes at
MTU 1500, which will cause QinQ traffic to fail. To work around this issue,
restrict the MTU on the Intel Ethernet device to 1496.
.UNINDENT
.UNINDENT
.UNINDENT
.SS Double VLANs
.sp
Devices based on the Intel(R) Ethernet 800 Series can process up to two VLANs
in a packet when all the following are installed:
.INDENT 0.0
.IP \(bu 2
ice driver version 1.4.0 or later
.IP \(bu 2
NVM version 2.4 or later
.IP \(bu 2
ice DDP package version 1.3.21 or later
.UNINDENT
.sp
If you don\(aqt use the versions above, the only supported VLAN configuration is
single 802.1Q VLAN traffic.
.sp
When two VLAN tags are present in a packet, the outer VLAN tag can be either
802.1Q or 802.1ad. The inner VLAN tag must always be 802.1Q.
.sp
Note the following limitations:
.INDENT 0.0
.IP \(bu 2
For each VF, the PF can only allow VLAN hardware offloads (insertion and
stripping) of one type, either 802.1Q or 802.1ad.
.IP \(bu 2
You can\(aqt enable or disable outer or single 802.1Q or 802.1ad filtering
separately. They are either both on or both off.
.IP \(bu 2
In SR\-IOV mode, the VF may not receive all network traffic based on the inner
VLAN header when VF true promiscuous mode (\fBvf\-true\-promisc\-support\fP) and
double VLANs are enabled.
.UNINDENT
.sp
To enable outer or single 802.1Q VLAN insertion and stripping and disable
802.1ad VLAN insertion and stripping:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> rxvlan on txvlan on rx\-vlan\-stag\-hw\-parse off
tx\-vlan\-stag\-hw\-insert off
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To enable outer or single 802.1ad VLAN insertion and stripping and disable
802.1Q VLAN insertion and stripping:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> rxvlan off txvlan off rx\-vlan\-stag\-hw\-parse on
tx\-vlan\-stag\-hw\-insert on
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To enable outer or single VLAN filtering:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> rx\-vlan\-filter on rx\-vlan\-stag\-filter on
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To disable outer or single VLAN filtering:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> rx\-vlan\-filter off rx\-vlan\-stag\-filter off
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Combining QinQ with SR\-IOV VFs
.sp
We recommend you always configure a port VLAN for the VF from the PF. If a port
VLAN is not configured, the VF driver may only offload VLANs via software. The
PF allows all VLAN traffic to reach the VF, and the VF manages all VLAN traffic.
.sp
When the device is configured for double VLANs and the PF has configured a port
VLAN:
.INDENT 0.0
.IP \(bu 2
The VF can only offload guest VLANs for 802.1Q traffic.
.IP \(bu 2
The VF can only configure VLAN filtering rules for guest VLANs using 802.1Q
traffic.
.UNINDENT
.sp
However, when the device is configured for double VLANs and the PF has NOT
configured a port VLAN:
.INDENT 0.0
.IP \(bu 2
You must use iavf driver version 4.1.0 or later to offload and filter VLANs.
.IP \(bu 2
The PF turns on VLAN pruning and antispoof in the VF\(aqs VSI by default. The VF
will not transmit or receive any tagged traffic until the VF requests a VLAN
filter.
.IP \(bu 2
The VF can offload (insert and strip) the outer VLAN tag of 802.1Q or 802.1ad
traffic.
.IP \(bu 2
The VF can create filter rules for the outer VLAN tag of both 802.1Q and
802.1ad traffic.
.UNINDENT
.sp
If the PF does not support double VLANs, the VF can hardware offload single
802.1Q VLANs without a port VLAN.
.sp
When the PF is enabled for double VLANs, for iavf drivers before version 4.1.x:
.INDENT 0.0
.IP \(bu 2
VLAN hardware offloads and filtering are supported only when the PF has
configured a port VLAN.
.IP \(bu 2
VLAN filtering, insertion, and stripping will be software offloaded when no
port VLAN is configured.
.UNINDENT
.sp
To see VLAN filtering and offload capabilities, use the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-k <ethX> | grep vlan
.ft P
.fi
.UNINDENT
.UNINDENT
.SS IEEE 1588 Precision Time Protocol (PTP) Hardware Clock (PHC)
.sp
Precision Time Protocol (PTP) is used to synchronize clocks in a computer
network. PTP support varies among Intel devices that support this driver. Use
the following command to get a definitive list of PTP capabilities supported by
the device:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-T <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
A detailed user guide is available for the following devices. Refer to it for
advanced configuration of this feature.
.INDENT 0.0
.IP \(bu 2
Intel(R) Ethernet Network Adapter E810\-XXV\-4T:
\fI\%https://cdrdv2.intel.com/v1/dl/getContent/646265\fP
.IP \(bu 2
Intel(R) Ethernet Network Adapter E810\-C\-Q2T:
\fI\%https://cdrdv2.intel.com/v1/dl/getContent/722960\fP
.UNINDENT
.sp
Intel Ethernet 800 Series devices support hardware\-generated timestamps. The
ice driver uses these timestamps to synchronize clocks on the platform and
report precise timestamps on packets. Use the following \fBhwstamp_ctl\fP command,
which is available in the linuxptp utility, to enable this setting:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
hwstamp_ctl \-i <ethX> \-t 1 \-r 1
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
All Intel Ethernet 800 Series (except E810) devices support hardware\-generated
cross timestamps, which capture PHC and x86 ART time simultaneously in HW.
This is available through kernel PTP device\(aqs PTP_SYS_OFFSET_PRECISE IOCTL.
.sp
E810 devices support only SW cross timestamping, which captures PHC and x86 TSC
times in software with high enough precision. This is only available to VFs by
using VIRTCHNL_OP_1588_PTP_GET_SW_CROSS_TSTAMP and
VIRTCHNL_OP_1588_PTP_GET_PHC_TO_CPU_DYNAMIC_RATIO commands.
.SS SyncE Support
.sp
On hardware that supports Synchronous Ethernet (SyncE), the ice driver has
interfaces that allow you to synchronize frequencies with other SyncE\-supported
ports. After you manually configure SyncE, the device dynamically selects the
best quality signal from the ones that are available. Then, once the signal is
locked, it synchronizes its frequency clock to it. The best quality signal is
determined based on the topology configured with the ice SyncE interfaces.
.sp
A detailed user guide is available for the following devices. Refer to it for
advanced configuration of this feature.
.INDENT 0.0
.IP \(bu 2
Intel(R) Ethernet Network Adapter E810\-XXV\-4T:
\fI\%https://cdrdv2.intel.com/v1/dl/getContent/646265\fP
.IP \(bu 2
Intel(R) Ethernet Network Adapter E810\-C\-Q2T:
\fI\%https://cdrdv2.intel.com/v1/dl/getContent/722960\fP
.UNINDENT
.SS Unmanaged DPLL Support
.sp
The managed DPLL support for supported hardware is described in external guides,
as mentioned in \fI\%SyncE Support\fP\&.
.sp
Hardware variants of E830 may support an unmanaged DPLL:
.INDENT 0.0
.IP \(bu 2
Intel® Ethernet Network Adapter E830\-XXVDA8F for OCP 3.0,
.IP \(bu 2
Intel® Ethernet Network Adapter E830\-XXVDA4F.
.UNINDENT
.sp
In the case of the unmanaged DPLL, the configuration is hardcoded within the
hardware and firmware, meaning users cannot modify settings. However,
users can check the DPLL lock status and obtain configuration information
through the Linux DPLL subsystem.
.sp
When present, the DPLL device locks to an external signal provided through the
PCIe/OCP pin. The expected input signal is 1PPS (1 Pulse Per Second) embedded
on a 10MHz reference clock.
The DPLL produces output:
.INDENT 0.0
.IP \(bu 2
for MAC (Media Access Control) & PHY (Physical Layer) clocks,
.IP \(bu 2
1PPS for synchronization of onboard PHC (Precision Hardware Clock) timer.
.UNINDENT
.sp
Requirements: The Linux kernel must have support for both the DPLL Subsystem
and the Embedded Sync patch series.
.sp
Example output of querying the Linux DPLL subsystem can be found below.
.sp
Dumping the DPLL pins
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ./tools/net/ynl/cli.py \-\-spec Documentation/netlink/specs/dpll.yaml \-\-dump pin\-get
[{\(aqboard\-label\(aq: \(aq1588\-TIME_SYNC\(aq,
  \(aqcapabilities\(aq: set(),
  \(aqclock\-id\(aq: 282574471561216,
  \(aqesync\-frequency\(aq: 1,
  \(aqesync\-frequency\-supported\(aq: [{\(aqfrequency\-max\(aq: 1, \(aqfrequency\-min\(aq: 1}],
  \(aqesync\-pulse\(aq: 25,
  \(aqfrequency\(aq: 10000000,
  \(aqid\(aq: 13,
  \(aqmodule\-name\(aq: \(aqice\(aq,
  \(aqparent\-device\(aq: [{\(aqdirection\(aq: \(aqinput\(aq,
                     \(aqparent\-id\(aq: 6,
                     \(aqstate\(aq: \(aqconnected\(aq}],
  \(aqphase\-adjust\-max\(aq: 0,
  \(aqphase\-adjust\-min\(aq: 0,
  \(aqtype\(aq: \(aqext\(aq},
  {\(aqboard\-label\(aq: \(aqMAC\-PHY\-CLK\(aq,
    \(aqcapabilities\(aq: set(),
  \(aqclock\-id\(aq: 282574471561216,
  \(aqfrequency\(aq: 156250000,
  \(aqid\(aq: 14,
  \(aqmodule\-name\(aq: \(aqice\(aq,
  \(aqparent\-device\(aq: [{\(aqdirection\(aq: \(aqoutput\(aq,
                     \(aqparent\-id\(aq: 6,
                     \(aqstate\(aq: \(aqconnected\(aq}],
  \(aqphase\-adjust\-max\(aq: 0,
  \(aqphase\-adjust\-min\(aq: 0,
  \(aqtype\(aq: \(aqsynce\-eth\-port\(aq},
{\(aqboard\-label\(aq: \(aq1588\-TIME_REF\(aq,
  \(aqcapabilities\(aq: set(),
  \(aqclock\-id\(aq: 282574471561216,
  \(aqfrequency\(aq: 1,
  \(aqid\(aq: 15,
  \(aqmodule\-name\(aq: \(aqice\(aq,
  \(aqparent\-device\(aq: [{\(aqdirection\(aq: \(aqoutput\(aq,
                     \(aqparent\-id\(aq: 6,
                     \(aqstate\(aq: \(aqconnected\(aq}],
  \(aqphase\-adjust\-max\(aq: 0,
  \(aqphase\-adjust\-min\(aq: 0,
  \(aqtype\(aq: \(aqint\-oscillator\(aq}]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Dumping the DPLL devices
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ./tools/net/ynl/cli.py \-\-spec Documentation/netlink/specs/dpll.yaml \-\-dump device\-get
[{\(aqclock\-id\(aq: 282574471561216,
  \(aqid\(aq: 6,
  \(aqlock\-status\(aq: \(aqlocked\(aq,
  \(aqmode\(aq: \(aqautomatic\(aq,
  \(aqmode\-supported\(aq: [\(aqautomatic\(aq],
  \(aqmodule\-name\(aq: \(aqice\(aq,
  \(aqtype\(aq: \(aqeec\(aq}]
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Earliest TxTime First (ETF) Offloads
.sp
Intel Ethernet E83x Series devices support Earliest TxTime First (ETF) offloads.
ETF offloads enable precise packet transmission scheduling, which is crucial
for time\-sensitive network applications that require strict timing control.
The ice driver uses the \fBSO_TXTIME\fP socket option to schedule the
transmission. Applications must set this option when sending packets to
leverage the precise transmission timing provided by ETF offloads.
.sp
To enable and configure ETF qdisc offload on the queues:
.INDENT 0.0
.IP 1. 3
Use the \fBmqprio\fP qdisc to classify packets into different traffic
classes. For example:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc add dev <ethX> handle 100: parent root mqprio num_tc 3 \e
  map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \e
  queues 1@0 1@1 2@2 \e
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Replace \fB<ethX>\fP with the name of your network interface.
.IP 2. 3
Set ETF as the queueing discipline for your desired traffic class. For
example, to set the ETF qdisc for traffic class number 0:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
tc qdisc replace dev <ethX> parent 100:1 etf \e
  clockid CLOCK_TAI delta 300000 offload
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
For a detailed explanation of the ETF qdisc and its options, see the tc\-etf
man page.
.SS Configuring Checksum Offloads
.sp
You can configure checksum offloads using ethtool.
.sp
To see the supported and configured checksum features for your device, run
the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-k <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To enable or disable Tx and Rx checksum offload features, run the following:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> [tx|rx] [on|off]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Intel Ethernet E83x Series devices support generic Tx and Rx checksums.
.INDENT 0.0
.IP \(bu 2
To change generic Rx checksum offloads, use the above Rx checksum
enable/disable command.
.IP \(bu 2
To change Tx generic checksum offloads, use the following:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> tx\-checksum\-ip\-generic [on|off]
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
On Intel Ethernet E83x Series devices, the generic Tx checksum offload
feature (tx\-checksum\-ip\-generic) cannot be enabled simultaneously with
the following features:
.INDENT 0.0
.IP \(bu 2
TCP Segmentation Offload (TSO), or
.IP \(bu 2
IP checksum offload (tx\-checksum\-ipv4 and tx\-checksum\-ipv6)
.UNINDENT
.sp
Ensure that TSO and IP checksum offloads are disabled before enabling
generic Tx checksum offloads.
.UNINDENT
.UNINDENT
.sp
For more details, please refer to the ethtool man page.
.SS Tunnel/Overlay Stateless Offloads
.sp
Supported tunnels and overlays include VXLAN, GENEVE, and others depending on
hardware and software configuration. Stateless offloads are enabled by default.
.sp
To view the current state of all offloads:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-k <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.SS UDP Segmentation Offload
.sp
Allows the adapter to offload transmit segmentation of UDP packets with
payloads up to 64K into valid Ethernet frames. Because the adapter hardware is
able to complete data segmentation much faster than operating system software,
this feature may improve transmission performance.
.sp
In addition, the adapter may use fewer CPU resources.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
UDP transmit segmentation offload requires Linux kernel 4.18 or later.
.IP \(bu 2
The application sending UDP packets must support UDP segmentation offload.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To enable/disable UDP Segmentation Offload, issue the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> tx\-udp\-segmentation [off|on]
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Runtime Control of CRC/FCS Stripping
.sp
The frame check sequence (FCS) is a four\-octet cyclic redundancy check (CRC)
that allows the driver to detect corrupted data within a received Ethernet
frame.
.sp
The ice driver allows you to disable or enable FCS/CRC stripping using the
ethtool command.
.INDENT 0.0
.IP \(bu 2
FCS/CRC stripping is enabled by default.
.IP \(bu 2
The driver enforces valid combinations of FCS/CRC and VLAN stripping. You can
only disable FCS/CRC stripping if VLAN stripping is also disabled on the PF.
.IP \(bu 2
Disabling FCS/CRC stripping may help when debugging issues. XDP programs can
also use FCS/CRC for their purposes.
.UNINDENT
.sp
Use the following ethtool command to enable or disable FCS/CRC stripping:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-K <ethX> rx\-fcs on|off
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To check the status of FCS/CRC stripping, look for the \fBrx\-fcs\fP information
reported from ethtool:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-k <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Using Devlink to update a device\(aqs NVM
.sp
When you update the NVM on some devices, the update may use the devlink
interface, rather than the ethtool interface. This will happen if the following
are true:
.INDENT 0.0
.IP \(bu 2
You are updating an Intel Ethernet 800 Series device.
.IP \(bu 2
Your system is running a distro that supports the \fBdevlink dev flash\fP command.
.IP \(bu 2
The firmware currently installed on the device supports it.
.IP \(bu 2
The new NVM conforms to the correct PLDM format.
.UNINDENT
.sp
Most of the functionality and commands are the same with the following
exceptions:
.INDENT 0.0
.IP \(bu 2
You cannot update a device in Recovery Mode. (To update a device in recovery
mode, you must download and install the Intel Ethernet driver set.)
.IP \(bu 2
You cannot update the OROM or Netlist as a separate update, only as part of
a full NVM update.
.IP \(bu 2
If you specified a preservation level of \fBPRESERVE_ALL\fP, the system will
immediately perform an EMPR reset after the NVM update.
.UNINDENT
.sp
On devices that support it, you can also use the devlink command line directly
to update the device NVM. However, we recommend that you use NVMUpdate.
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev flash pci/0000:3b:00.0 file filename.bin
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B pci/0000:3b:00.0
The device you wish to update. You can get a list of devices
with the \fBdevlink dev info\fP command.
.TP
.B filename.bin
The file that contains the new NVM image.
.UNINDENT
.SS Port Split Configuration Using Devlink
.sp
Most CVL devices support changing their port split configuration to suit your
needs. For example, a dual port device may support two 100Gbps links, two
50Gbps links, and (with the correct cables) four 25Gbps links, etc. The
supported port split configurations are defined in the device\(aqs NVM.
.sp
You can use a tool like Intel\(aqs Ethernet Port Configuration Tool (EPCT) to
query and set this configuration. If no such tool is available, you can use
devlink to cycle through a device\(aqs possible prt split configurations.
.sp
If you use devlink to change the configuration, you must check the log to
determine which configuration was selected. If you use devlink, you specify
the number of ports you want configured on the device. Each time you call
devlink with that port count, the driver will check the device\(aqs current
configuration and then move to the next configuration with the specified
number of ports. For example, if your device has two four\-port configurations
defined in its NVM, the first time you called devlink, it would select the
first configuration. The second time you called devlink, it would select
the second configuration. If you called devlink again, it would select the
first configuration.
.sp
There is no direct feedback mechanism; you must check the log to determine
which configuration was set. Use the following command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port split <pci/D:b:d.f>/0 count <num>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <pci/D:b:d.f>/0
The PCI address of the device (pci/Domain:bus:device.function).
\fB/0\fP is the \fBPORT_INDEX\fP\&.
.TP
.B <num>
The desired port split count.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
If you successfully change a port\(aqs configuration, the driver logs an
information message: \(dqReboot required to finish port split\(dq and the port
split configuration selected. This is the only indication of success.
.IP \(bu 2
If you request an unsupported count value parameter in devlink port split,
the driver logs an information message: \(dqPort split requested unsupported
port config.\(dq
.IP \(bu 2
If you try to change the configuration on a PF that is not PF 0, the
driver returns the error \(dqPort cannot be split.\(dq
.UNINDENT
.UNINDENT
.UNINDENT
.sp
For example, if your device had the following configurations defined in its NVM:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ice 0000:16:00.0:  Status  Split      Quad 0         Quad 1
ice 0000:16:00.0:          count  L0  L1  L2  L3  L4  L5  L6  L7
ice 0000:16:00.0: Active   2     100   \-   \-   \- 100   \-   \-   \-
ice 0000:16:00.0:          2      50   \-  50   \-   \-   \-   \-   \-
ice 0000:16:00.0:          4      25  25  25  25   \-   \-   \-   \-
ice 0000:16:00.0:          4      25  25   \-   \-  25  25   \-   \-
ice 0000:16:00.0:          8      10  10  10  10  10  10  10  10
ice 0000:16:00.0:          1     100   \-   \-   \-   \-   \-   \-   \-
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
If you call:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port split pci/0000.16:00.0/0 count 4
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Your device will be configured for:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ice 0000:16:00.0:          4      25  25  25  25   \-   \-   \-   \-
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
If you call the same command again, your device will be configured for:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ice 0000:16:00.0:          4      25  25   \-   \-  25  25   \-   \-
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
If you call the same command a third time, your device will cycle back to the
top of its 4\-port configurations (because there are only two 4\-port
configurations defined it its NVM) and will be set to:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ice 0000:16:00.0:          4      25  25  25  25   \-   \-   \-   \-
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Firmware Logs
.sp
The ice driver allows you to generate firmware logs for supported categories of
events, to help debug issues with Customer Support. Firmware logs are enabled
by default. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for
an overview of this feature and additional tips.
.INDENT 0.0
.IP \(bu 2
The driver supports firmware logging via the debugfs interface on PF 0 only.
.IP \(bu 2
The firmware running on the Ethernet device must support firmware logging; if
the firmware does not support firmware logging, the \fBfwlog\fP file will not
get created in the ice \fBdebugfs\fP directory.
.IP \(bu 2
Firmware logs are stored in a data file in binary form.
.UNINDENT
.sp
At a high level, you must do the following to capture a firmware log (see the
subsections below for details):
.INDENT 0.0
.IP 1. 3
Set log levels. For example:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo normal > /sys/kernel/debug/ice/0000:18:00.0/fwlog/modules/all
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Turn on firmware logging:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo 1 > /sys/kernel/debug/ice/0000:18:00.0/fwlog/enable
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 3. 3
Perform the necessary steps to generate the issue you are trying to debug.
.IP 4. 3
Turn off firmware logging:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/kernel/debug/ice/0000:18:00.0/fwlog/enable
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 5. 3
Save data to a file:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/0000:18:00.0/fwlog/data > fwlog.bin
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 6. 3
Work with Customer Support to debug your issue.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
Firmware logs are generated in a binary format and MUST be decoded by
Customer Support. Information collected is related only to firmware and
hardware for debug purposes.
.IP \(bu 2
You must have admin permissions and be logged in as root to change firmware
logging settings.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The driver groups firmware events into categories, called \(dqmodules.\(dq
The modules are instantiated under the \fBfwlog/modules\fP directory.
.sp
To configure modules and verbosity levels:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo <log_level> > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/<module>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.IP \(bu 2
\fB<log_level>\fP is the desired verbosity level of the firmware logs. Supported
values include:
.INDENT 2.0
.IP \(bu 2
none
.IP \(bu 2
error
.IP \(bu 2
warning
.IP \(bu 2
normal
.IP \(bu 2
verbose
.UNINDENT
.sp
You can set only one log level per module, and each level includes the
verbosity levels lower than it. For instance, setting the level to \fBnormal\fP
will also log warning and error messages.
.IP \(bu 2
\fB<module>\fP is a name that represents the module to receive events for.
Supported values include:
.INDENT 2.0
.IP \(bu 2
\fBgeneral\fP \- General
.IP \(bu 2
\fBctrl\fP \- Control
.IP \(bu 2
\fBlink\fP \- Link Management
.IP \(bu 2
\fBlink_topo\fP \- Link Topology Detection
.IP \(bu 2
\fBdnl\fP \- Link Control Technology
.IP \(bu 2
\fBi2c\fP \- I2C
.IP \(bu 2
\fBsdp\fP \- SDP
.IP \(bu 2
\fBmdio\fP \- MDIO
.IP \(bu 2
\fBadminq\fP \- Admin Queue
.IP \(bu 2
\fBhdma\fP \- Host DMA
.IP \(bu 2
\fBlldp\fP \- LLDP
.IP \(bu 2
\fBdcbx\fP \- DCBx
.IP \(bu 2
\fBdcb\fP \- DCB
.IP \(bu 2
\fBxlr\fP \- XLR (function\-level resets)
.IP \(bu 2
\fBnvm\fP \- NVM
.IP \(bu 2
\fBauth\fP \- Authentication
.IP \(bu 2
\fBvpd\fP \- Vital Product Data
.IP \(bu 2
\fBiosf\fP \- Intel On\-Chip System Fabric
.IP \(bu 2
\fBparser\fP \- Parser
.IP \(bu 2
\fBsw\fP \- Switch
.IP \(bu 2
\fBscheduler\fP \- Scheduler
.IP \(bu 2
\fBtxq\fP \- TX Queue Management
.IP \(bu 2
\fBrsvd\fP \- ACL (Access Control List)
.IP \(bu 2
\fBpost\fP \- Post
.IP \(bu 2
\fBwatchdog\fP \- Watchdog
.IP \(bu 2
\fBtask_dispatch\fP \- Task Dispatcher
.IP \(bu 2
\fBmng\fP \- Manageability
.IP \(bu 2
\fBsynce\fP \- SyncE
.IP \(bu 2
\fBhealth\fP \- Health
.IP \(bu 2
\fBtsdrv\fP \- Time Sync
.IP \(bu 2
\fBpfreg\fP \- PF Registration
.IP \(bu 2
\fBmdlver\fP \- Module Version
.IP \(bu 2
\fBall\fP \- Allows you to set all of the modules to the specified \fBlog_level\fP or
to read the \fBlog_level\fP of all of the modules
.UNINDENT
.UNINDENT
.sp
EXAMPLES:
.sp
To set a single module to \fBverbose\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo verbose > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/link
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To set multiple modules and then issue the command multiple times:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo verbose > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/link
echo warning > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/ctrl
echo none > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/dcb
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To set all the modules to the same value:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo normal > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/all
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To read the \fBlog_level\fP of a specific module (for example, the \fBgeneral\fP
module):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/general
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To read the \fBlog_level\fP of all the modules:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/modules/all
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
You can configure modules, but firmware will \fBnot\fP send firmware logging
events to the driver until it explicitly receives the \fBfwlog\eenable\fP setting.
.sp
To enable firmware logging:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 1 > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/enable
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To disable firmware logging:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/enable
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To retrieve the firmware log data, read the data from \fBfwlog\edata\fP\&. You can
clear the contents of the firmware log data by writing any value to
\fBfwlog\edata\fP\&.
.sp
To retrieve the firmware log data and output it to a binary file:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/data > fwlog.bin
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To clear the contents of the firmware log data:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/data
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
You can only clear the data when firmware logging is disabled.
.IP \(bu 2
Firmware logs are generated in a binary format and MUST be decoded by
Customer Support. Information collected is related only to firmware and
hardware for debug purposes.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The driver receives firmware log data from the Admin Receive Queue (ARQ).
To change the frequency that the firmware sends the ARQ events, write a value
to \fBfwlog/nr_messages\fP\&.
.INDENT 0.0
.IP \(bu 2
The range is 1\-128 (1 means push every log message; 128 means push only when
the max AQ command buffer is full).
.IP \(bu 2
The suggested value is 10.
.UNINDENT
.sp
An example to set the value is:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 50 > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/nr_messages
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To see the currently configured value:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/nr_messages
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The driver stores firmware log data within the driver. The default size of the
memory used to store the data is 1MB, but some use cases may require more or less
data.
.sp
To change the amount of memory allocated for firmware log data, write to
\fBfwlog/log_size\fP\&. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 128K > /sys/kernel/debug/ice/0000\e:18\e:00.0/fwlog/log_size
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The value must be one of: 128K, 256K, 512K, 1M, or 2M.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
Firmware logging must be disabled to change the \fBfwlog/log_size\fP
value.
.UNINDENT
.UNINDENT
.SS Debug Dump
.sp
Intel Ethernet 800 Series devices support debug dump, which allows you to
obtain runtime register values from the firmware for \(dqclusters\(dq of events and
then write the results to a single dump file, for debugging complicated issues
in the field.
.sp
This debug dump contains a snapshot of the device and its existing hardware
configuration, such as switch tables, transmit scheduler tables, and other
information. Debug dump captures the current state of the specified cluster(s)
and is a stateless snapshot of the whole device.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
Like with firmware logs, the contents of the debug dump are not
human\-readable. You must work with Customer Support to decode the file.
.IP \(bu 2
Debug dump is per device, not per PF.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The ice driver uses the debugfs interface for debug dump. To generate a debug
dump file in Linux, do the following:
.INDENT 0.0
.IP 1. 3
Specify the cluster(s) to include in the dump file, using one of the
following commands. You can either set a single cluster or all clusters.
.INDENT 3.0
.IP \(bu 2
To dump all clusters:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo > /sys/kernel/debug/ice/<pci_addr>/fw/dump_cluster_id
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To dump a single cluster:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo <cluster ID> > /sys/kernel/debug/ice/<pci_addr>/fw/dump_cluster_id
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The cluster ID values vary depending on the Intel Ethernet controller family. Use the appropriate reference table below based on your device type.
.sp
Possible values for \fB<cluster ID>\fP are:
.TS
center;
|l|l|l|.
_
T{
Cluster Function
T}	T{
E810/E82x
T}	T{
E83x Series
T}
_
T{
Switch
T}	T{
0
T}	T{
100
T}
_
T{
ACL
T}	T{
1
T}	T{
101
T}
_
T{
Tx Scheduler
T}	T{
2
T}	T{
102
T}
_
T{
Profile Configuration
T}	T{
3
T}	T{
103
T}
_
T{
EMP DRAM
T}	T{
4
T}	T{
104
T}
_
T{
Link
T}	T{
5
T}	T{
105
T}
_
T{
DCB
T}	T{
7
T}	T{
107
T}
_
T{
L2P
T}	T{
8
T}	T{
108
T}
_
T{
Queue Manageability
T}	T{
9
T}	T{
109
T}
_
T{
CSR Space
T}	T{
21
T}	T{
121
T}
_
T{
Manageability Transactions
T}	T{
22
T}	T{
122
T}
_
.TE
.UNINDENT
.sp
For example, to dump the CSR space, use the following:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
echo 21 > /sys/kernel/debug/ice/<pci_addr>/fw/dump_cluster_id
.ft P
.fi
.UNINDENT
.UNINDENT
.IP 2. 3
Save the debug dump to a file, using one of the following. Replace the
\fB\&.bin\fP name with the file name you want to use. For example:
.INDENT 3.0
.IP \(bu 2
For a single cluster:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/<pci_addr>/fw/debug_dump > ~/single_cluster_dump.bin
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
For all clusters:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/kernel/debug/ice/<pci_addr>/fw/debug_dump > ~/all_cluster_dump.bin
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SS Debugging PHY Statistics
.sp
The ice driver supports the ability to obtain the values of the PHY registers,
to debug link and connection issues during runtime.
.sp
The driver allows you to obtain information about:
.INDENT 0.0
.IP \(bu 2
Rx and Tx Equalization parameters
.IP \(bu 2
RS FEC correctable and uncorrectable block counts
.UNINDENT
.sp
Use ethtool to read the PHY registers:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-d <ethX> [raw on|off] [hex on|off] [file name]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The output of the command is similar to:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
Offset          Values
\-\-\-\-\-\-          \-\-\-\-\-\-
0x0000:         00 00 00 00 0a 00 00 00 00 00 00 00 00 00 00 00
0x0010:         00 00 00 00 02 00 a9 b1 00 10 00 00 19 df 00 00
0x0020:         b7 00 00 00 d4 00 00 00 00 00 00 00 00 00 00 00
0x0030:         0a 00 00 00 19 00 00 00 00 00 00 00 0e 00 00 00
\&...
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
The contents of the registers are not human\-readable. You must work
with Customer Support to decode the file.
.UNINDENT
.UNINDENT
.sp
Use ethtool to read the RS FEC correctable and uncorrectable block counts:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-show\-fec <ethX>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The output of the command is similar to:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
FEC parameters for ethX:
Supported/Configured FEC encodings: Auto RS BaseR
Active FEC encoding: RS
Statistics:
  corrected_blocks: 0
  uncorrectable_blocks: 0
.ft P
.fi
.UNINDENT
.UNINDENT
.SS Hierarchical QoS (HQoS) Transmit Scheduler
.sp
You can configure a custom transmit scheduler tree structure to shape transmit
traffic for specific needs. You change the tree structure by creating parent
nodes on the device and then assigning child nodes (VFs)to the parent node. You
can also change the transmit rate management configuration for each node.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
Reconfiguring the scheduler topology should only be done by an expert.
Modifying the scheduler topology may adversly impact your device\(aqs network
availability and throughput. Do not do this unless you are willing to take
these risks. After modifying the scheduler topology, if your device does
not perform as expected, you should return the device to the default
topology.
.IP \(bu 2
Modifying the Hierarchical QoS (HQoS) Transmit Scheduler requires Kernel
6.2, or later.
.IP \(bu 2
Modifying the Hierarchical QoS (HQoS) Transmit Scheduler is not compatible
with ADQ, DCB, RDMA, or other custom scheduler tree features.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To create a devlink\-rate parent group:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port function rate add <dev/port>/<group>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
where:
.INDENT 0.0
.TP
.B <dev/port>
The pci bus:device:function of the device
.TP
.B <group>
A new parent group
.UNINDENT
.sp
For example, the following creates the \fBoperators\fP group on the specified
device:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port function rate add pci/0000:03:00.0/operators
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To create a new child node in a parent group:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port function rate add <dev/port>/<child> parent <group>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
where:
.INDENT 0.0
.TP
.B <dev/port>
The pci bus:device:function of the device
.TP
.B <child>
A new child node
.TP
.B <group>
An existing parent group
.UNINDENT
.sp
For example, the following creates the \fBclass_1\fP child node in the
\fBoperators\fP parent group:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port function rate add pci/0000:03:00.0/class_1 parent operators
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To display a device\(aqs current tree structure, where \fB<dev/port>\fP is the pci
bus:device:function of the device:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port function rate show <dev/port>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink port function rate show pci/0000:03:00.0
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Example output:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
pci/0000:03:00.0/node_0 type node (root)
pci/0000:03:00.0/operators type node tx_share 20Mbit tx_max 100Mbit
tx_priority 2 tx_weight 5
pci/0000:03:00.0/class_1 type node parent operators
pci/0000:03:00.0/1 type leaf parent class_1
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Refer to the devlink\-rate man page and other documentation for details.
.SS Types of supported resets
.sp
The driver supports a few types of resets that the user can perform using ethtool.
Ethtool reset flags are mapped to the E810 reset type:
.sp
PF reset:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-reset <ethX> irq dma filter offload
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
CORE reset:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-reset <ethX> irq\-shared dma\-shared filter\-shared offload\-shared \e
ram\-shared
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
GLOBAL reset:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-reset <ethX> irq\-shared dma\-shared filter\-shared offload\-shared \e
mac\-shared phy\-shared ram\-shared
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Calling the same set of flags on a port representor triggers a VF reset.
.SH PERFORMANCE OPTIMIZATION
.sp
Driver defaults are meant to fit a wide variety of workloads, but if further
optimization is required, we recommend experimenting with the following
settings.
.SS Transmit/Receive Queue Allocation
.sp
The driver allocates a number of transmit/receive queue pairs equal to the
number of local node CPU threads with the following constraints:
.INDENT 0.0
.IP \(bu 2
The driver will allocate a minimum of 8 queue pairs, or the total number of
CPUs, whichever is lower.
.IP \(bu 2
The driver will allocate a maximum of 64 queue pairs, or 256 for the iavf
driver.
.UNINDENT
.sp
You can set the number of symmetrical (Rx/Tx) or asymmetrical (mix of combined
and Tx or Rx) queues using the \fBethtool \-L\fP command. Use the \fBcombined\fP
parameter to set the symmetrical part of the configuration, and then use
either \fBrx\fP or \fBtx\fP to set the remaining asymmetrical part of the
configuration. For example:
.INDENT 0.0
.IP \(bu 2
To set 16 queue pairs, regardless of what the previous configuration was:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-L <ethX> combined 16 rx 0 tx 0
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 2.0
.INDENT 3.5
If the current configuration is already symmetric, you can omit
the \fBrx\fP and \fBtx\fP parameters. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-L <ethX> combined 16
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.IP \(bu 2
To set 16 Tx queues and 8 Rx queues:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-L <ethX> combined 8 tx 8
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
You cannot configure less than 1 queue pair. Attempts to do so will
be rejected by the kernel.
.IP \(bu 2
You cannot configure more Tx/Rx queues than there are MSI\-X interrupts
available. Attempts to do so will be rejected by the driver.
.IP \(bu 2
\fBethtool\fP preserves the previous values of \fBcombined\fP, \fBrx\fP, and
\fBtx\fP independently, same as it handles flags. If you do not specify
a certain value in the command, it will stay the same instead of being set
to zero.
.IP \(bu 2
Tx/Rx queues cannot exist outside of queue pairs simultaneously, so either
\fBrx\fP or \fBtx\fP parameter has to be zero.
.UNINDENT
.UNINDENT
.UNINDENT
.SS IRQ to Adapter Queue Alignment
.sp
Pin the adapter\(aqs IRQs to specific cores by disabling the irqbalance service
and using the included \fBset_irq_affinity\fP script. Please see the script\(aqs
help text for further options.
.INDENT 0.0
.IP \(bu 2
The following settings will distribute the IRQs across all the cores
evenly:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
scripts/set_irq_affinity \-x all <interface1> , [ <interface2>, ... ]
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
The following settings will distribute the IRQs across all the cores that
are local to the adapter (same NUMA node):
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
scripts/set_irq_affinity \-x local <interface1> ,[ <interface2>, ... ]
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
For very CPU\-intensive workloads, we recommend pinning the IRQs to all
cores.
.UNINDENT
.SS Rx Descriptor Ring Size
.sp
To reduce the number of Rx packet discards, increase the number of Rx
descriptors for each Rx ring using ethtool.
.INDENT 0.0
.IP \(bu 2
Check if the interface is dropping Rx packets due to buffers being full
(\fBrx_dropped.nic\fP can mean that there is no PCIe bandwidth):
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-S <ethX> | grep \(dqrx_dropped\(dq
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
If the previous command shows drops on queues, it may help to increase
the number of descriptors using \fBethtool \-G\fP, where \fB<N>\fP is the desired
number of ring entries/descriptors:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-G <ethX> rx <N>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
This can provide temporary buffering for issues that create latency while
the CPUs process descriptors.
.UNINDENT
.SS Interrupt Rate Limiting
.sp
This driver supports an adaptive interrupt throttle rate (ITR) mechanism that
is tuned for general workloads. The user can customize the interrupt rate
control for specific workloads, via ethtool, adjusting the number of
microseconds between interrupts.
.sp
To set the interrupt rate manually, you must disable adaptive mode:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-C <ethX> adaptive\-rx off adaptive\-tx off
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For lower CPU utilization:
.INDENT 0.0
.IP \(bu 2
Disable adaptive ITR and lower Rx and Tx interrupts. The examples below
affect every queue of the specified interface.
.IP \(bu 2
Setting \fBrx\-usecs\fP and \fBtx\-usecs\fP to 80 will limit interrupts to about
12,500 interrupts per second per queue:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-C <ethX> adaptive\-rx off adaptive\-tx off rx\-usecs 80 tx\-usecs 80
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
For reduced latency:
.INDENT 0.0
.IP \(bu 2
Disable adaptive ITR and ITR by setting \fBrx\-usecs\fP and \fBtx\-usecs\fP to 0
using ethtool:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-C <ethX> adaptive\-rx off adaptive\-tx off rx\-usecs 0 tx\-usecs 0
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Per\-queue interrupt rate settings:
.INDENT 0.0
.IP \(bu 2
The following examples are for queues 1 and 3, but you can adjust other
queues.
.IP \(bu 2
To disable Rx adaptive ITR and set static Rx ITR to 10 microseconds or
about 100,000 interrupts/second, for queues 1 and 3:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-per\-queue <ethX> queue_mask 0xa \-\-coalesce adaptive\-rx off rx\-usecs 10
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
To show the current coalesce settings for queues 1 and 3:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-\-per\-queue <ethX> queue_mask 0xa \-\-show\-coalesce
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Bounding interrupt rates using \fBrx\-usecs\-high\fP:
.INDENT 0.0
.IP \(bu 2
Valid Range: 0\-236 (0=no limit)
.sp
The range of 0\-236 microseconds provides an effective range of 4,237 to
250,000 interrupts per second. The value of \fBrx\-usecs\-high\fP can be set
independently of \fBrx\-usecs\fP and \fBtx\-usecs\fP in the same ethtool command,
and is also independent of the adaptive interrupt moderation algorithm. The
underlying hardware supports granularity in 4\-microsecond intervals, so
adjacent values may result in the same interrupt rate.
.IP \(bu 2
The following command would disable adaptive interrupt moderation, and allow
a maximum of 5 microseconds before indicating a receive or transmit was
complete. However, instead of resulting in as many as 200,000 interrupts per
second, it limits total interrupts per second to 50,000 via the
\fBrx\-usecs\-high\fP parameter:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-C <ethX> adaptive\-rx off adaptive\-tx off rx\-usecs\-high 20
rx\-usecs 5 tx\-usecs 5
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.SS Virtualized Environments
.sp
In addition to the other suggestions in this section, the following may be
helpful to optimize performance in VMs.
.INDENT 0.0
.IP \(bu 2
Using the appropriate mechanism (vcpupin) in the VM, pin the CPUs to
individual LCPUs, making sure to use a set of CPUs included in the
device\(aqs \fBlocal_cpulist\fP:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
/sys/class/net/<ethX>/device/local_cpulist
.ft P
.fi
.UNINDENT
.UNINDENT
.IP \(bu 2
Configure as many Rx/Tx queues in the VM as available. (See the iavf driver
documentation for the number of queues supported.) For example:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-L <virt_interface> rx <max> tx <max>
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.SS Transmit Scheduling Layers
.sp
The ice hardware uses hierarchical scheduling for Tx with a fixed number of
layers in the scheduling tree. Each of them are decision points. Root node
represents a port, while all the leaves represent the queues. This way of
configuring the Tx scheduler allows features like DCB or devlink\-rate
(documented below) to configure how much bandwidth is given to any given queue
or group of queues, enabling fine\-grained control because scheduling parameters
can be configured at any given layer of the tree.
.sp
The default 9\-layer tree topology was deemed best for most workloads, as it
gives an optimal ratio of performance to configurability. However, for some
specific cases, this 9\-layer topology might not be desired.  One example would
be sending traffic to queues that are not a multiple of 8. Because the maximum
radix is limited to 8 in 9\-layer topology, the 9th queue has a different parent
than the rest, and it\(aqs given more bandwidth credits. This causes a problem
when the system is sending traffic to 9 queues:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
| tx_queue_0_packets: 24163396
| tx_queue_1_packets: 24164623
| tx_queue_2_packets: 24163188
| tx_queue_3_packets: 24163701
| tx_queue_4_packets: 24163683
| tx_queue_5_packets: 24164668
| tx_queue_6_packets: 23327200
| tx_queue_7_packets: 24163853
| tx_queue_8_packets: 91101417 < Too much traffic is sent from 9th
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To address this need, you can switch to a 5\-layer topology, which
changes the maximum topology radix to 512. With this enhancement,
the performance characteristic is equal as all queues can be assigned
to the same parent in the tree. The obvious drawback of this solution
is a lower configuration depth of the tree.
.sp
Use the \fBtx_scheduling_layer\fP parameter with the devlink command to change
the transmit scheduler topology. To use 5\-layer topology, use a value of 5.
For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param set pci/0000:16:00.0 name tx_scheduling_layers
value 5 cmode permanent
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Use a value of 9 to set it back to the default value. You must do PCI slot
powercycle for the selected topology to take effect.
.sp
To verify that value has been set:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink dev param show pci/0000:16:00.0 name tx_scheduling_layers
.ft P
.fi
.UNINDENT
.UNINDENT
.SS MSI\-X Vector Allocation
.sp
The ice driver automatically allocates MSI\-X vectors for PF, VF, and RDMA from
a pool of 2048 vectors. If there are 8, or fewer, local node CPU threads, the
driver will automatically allocate 8 vectors for each PF. This scales up by
allocating one vector per local node CPU thread, up to 64 vectors.
.sp
The driver will not automatically allocate more than 64 MSI\-X vectors for each
PF. RDMA requires one more MSI\-X vector than the PF allocation, so the driver
will automatically allocate 9\-65 MSI\-X vectors for RDMA.
.sp
You can use sysfs to override the automatic MSI\-X vector allocation for a
particular PF or RDMA function, or for the pool of vectors used by the VFs
bound to a PF:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink resource set <pci/D:b:d.f> path msix/<parameter> size <num>
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <pci/D:b:d.f>
The PCI address of the device (pci/Domain:bus:device.function).
.TP
.B <parameter>
Is one of the following:
.INDENT 7.0
.IP \(bu 2
For a PF, use the \fBmsix_eth\fP parameter.
.IP \(bu 2
For an RDMA function, use the \fBmsix_rdma\fP parameter.
.IP \(bu 2
For the pool of vectors used by the VFs, use the \fBmsix_vf\fP parameter.
.UNINDENT
.TP
.B <num>
The number of MSI\-X vector to assign to the function.
.UNINDENT
.sp
For example, to set a PF to use 320 MSI\-X vectors:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink resource set pci/0000:31:00.1 path msix/msix_eth size 320
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
For this change to take effect, you must reinitialize the driver
after you make this change. Reinitializing the driver may drop some netdev
configurations, including reset or downtime. Refer to the Devlink Reload
documentation for more information.
.UNINDENT
.UNINDENT
.sp
You can set the allocation for a particular VF with the \fBsriov_vf_msix_count\fP
sysfs parameter:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo <num> > /sys/bus/pci/devices/D:b:d.f/sriov_vf_msix_count
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Where:
.INDENT 0.0
.TP
.B <D:b:d.f>
The PCI address of the device (Domain:bus:device.function)
.TP
.B <num>
The number of MSI\-X vectors to allocate to the particular VF
.UNINDENT
.sp
For example, to set a VF to 64 MSI\-X vectors, use:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 64 > /sys/bus/pci/devices/0000:31:00.2/sriov_vf_msix_count
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
For some Linux distributions, the \fBsriov_vf_msix_count\fP
parameter is not available, and it is recommended to use devlink parameter.
.UNINDENT
.UNINDENT
.sp
You can check the current MSI\-X vector allocation by using the \fBdevlink
resource show\fP parameter. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
devlink resource show pci/0000:31:00.1
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Might return:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
name: msix size 520 occ 262 unit entry dpipe_tables none
resources:
  name msix_misc size 4 unit entry dpipe_tables none
  name: msix_eth size 48 occ 24 unit
  name: msix_vf size 48 occ 24 unit
  name: msix_rdma size 48 occ 24 unit
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The ice driver supports changing the automatic MSI\-X vector and queue allocation
for PFs and VFs to spread the RSS load across more cores.
.sp
Each PF is by default given ownership of the PF look\-up table (LUT) with 2048
entries, and each VF is given a VSI LUT with 64 entries. There are 16 global
LUTs with 512 entries each, which allows for the use of 64 queue pairs and
can be allocated for both VFs and PFs.
.sp
The PF default MSI\-X count is equal to the number of local CPU cores, which
allows for the use of up to 256 queue pairs.
.sp
The VF default is 16 MSI\-X vectors, which allows for the use of 16 queue pairs.
This can further be limited by the available number of local CPU cores. Usually
1 MSI\-X vector maps to one TX queue pair.
.sp
Several MSI\-X vectors are required for other functionality on the PF, such as
basic control and RDMA.
.sp
Use the \fBrss_lut_pf_attr\fP and \fBrss_lut_vf_attr\fP sysfs parameters to manage
the LUT sizes for the VF and PF.
.INDENT 0.0
.IP \(bu 2
You can change the MSI\-X count and LUT size for both the PF and VF separately.
.IP \(bu 2
You can assign a PF LUT to a bound VF after increasing the VF\(aqs MSI\-X vector
limit to the intended number of queue pairs and decreasing the PF\(aqs LUT to
the global LUT size of 512.
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
Before changing \fBrss_lut_vf_attr\fP, you must first set
\fBsriov_drivers_autoprobe\fP to zero. After changing \fBrss_lut_vf_attr\fP,
you can set \fBsriov_drivers_autoprobe\fP back to 1.
.IP \(bu 2
You must reload the iavf driver after making these changes.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
To set a VF\(aqs queue pair limit up to 64, using the global LUT:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/bus/pci/devices/<ethx>/sriov_drivers_autoprobe
echo 512 > /sys/bus/pci/devices/<ethx>/rss_lut_vf_attr
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To set a VF to use its PF\(aqs LUT:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/bus/pci/devices/<ethx>/sriov_drivers_autoprobe
echo 512 > /sys/bus/pci/devices/<ethx>/rss_lut_pf_attr
echo 2048 > /sys/bus/pci/devices/<ethx>/rss_lut_vf_attr
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To set a PF back to using its PF LUT:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo 0 > /sys/bus/pci/devices/<ethx>/sriov_drivers_autoprobe
echo 64 > /sys/bus/pci/devices/<ethx>/rss_lut_vf_attr
echo 2048 > /sys/bus/pci/devices/<ethx>/rss_lut_pf_attr
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The ice driver support for monitoring the internal temperature of the chip.
.sp
Match the PCI address to identify the device\(aqs hwmon interface.
For example use:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
tree /sys/class/hwmon/
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Might return:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
/sys/class/hwmon/
├── <hwmonX> \-> ../../devices/pci0000:<bus>/<pci\-addresss\-id>/hwmon/hwmon0
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The temperature data can be accessed through the sysfs interface:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
cat /sys/class/hwmon/<hwmonX>/temp1_input
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The temperature value is reported in millidegrees Celsius.
.SH KNOWN ISSUES/TROUBLESHOOTING
.SS Receive Error counts may be higher than the actual packet error count
.sp
When a packet is received with more than one error, two bad packets may be
reported. This affects all devices based on 10G, or faster, controllers.
.SS Dynamic Debug
.sp
If you encounter unexpected issues during driver load, some of the most useful
information for developers to receive in a bug report can include driver
logging. This logging uses a kernel feature called Dynamic Debug, which is
generally enabled in most kernel configurations (\fBCONFIG_DYNAMIC_DEBUG=y\fP).
.sp
To load the driver with dynamic debug enabled, run modprobe with the dyndbg
parameter:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
modprobe ice dyndbg=+p
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The driver will then load and print debugging information into the kernel log
(dmesg) and is usually logged into the system log viewable by journalctl or in
\fB/var/log/messages\fP\&. Saving this information to a file and attaching it to any
bug report can help shorten the reproduction and debugging time for a developer.
.sp
To enable dynamic debug during runtime operation of the driver, use this
command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
echo \(dqmodule ice +p\(dq > /sys/kernel/debug/dynamic_debug/control
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
For more details, see the Dynamic Debug documentation included in the Linux
kernel instructions.
.SS PF Message Queue Overflow
.sp
The device driver can detect some types of anomalous behavior. When it does, it
will log the VF MAC address and associated PF MAC address. Using this
information, you can check the virtual machine (VM) that is using the VF MAC
address to ensure that the VM is operating correctly.
.SS \fBethtool \-S\fP does not display Tx/Rx packet statistics
.sp
Issuing the command \fBethtool \-S\fP does not display Tx/Rx packet statistics.
This is by convention. Use other tools (such as the ip command) that display
standard netdev statistics such as Tx/Rx packet statistics.
.SS \fBethtool \-S\fP rx_bytes and ip stats rx_bytes don\(aqt match statistics
.sp
The \fBrx_bytes\fP value of ethtool does not match the \fBrx_bytes\fP value of
Netdev, due to the 4\-byte CRC being stripped by the device. The difference
between the two \fBrx_bytes\fP values will be 4 x the number of Rx packets. For
example, if Rx packets are 10 and Netdev (software statistics) displays
\fBrx_bytes\fP as \(dqX\(dq, then ethtool (hardware statistics) will display \fBrx_bytes\fP
as \(dqX+40\(dq (4 bytes CRC x 10 packets).
.SS Unexpected issues when the device driver and DPDK share a device
.sp
Unexpected issues may result when an ice device is in multi driver mode and the
kernel driver and DPDK driver are sharing the device. This is because access to
the global NIC resources is not synchronized between multiple drivers. Any
change to the global NIC configuration (writing to a global register, setting
global configuration by AQ, or changing switch modes) will affect all ports and
drivers on the device. Loading DPDK with the \fBmulti\-driver\fP module parameter
may mitigate some of the issues.
.SS Fiber optics and auto\-negotiation
.sp
Modules based on 100GBASE\-SR4, active optical cable (AOC), and active copper
cable (ACC) do not support auto\-negotiation per the IEEE specification. To
obtain link with these modules, you must turn off auto\-negotiation on the link
partner\(aqs switch ports.
.SS \fBethtool \-a\fP autonegotiate result may vary between drivers
.sp
For kernel versions 4.6 or higher, \fBethtool \-a\fP will show the advertised and
negotiated autoneg settings. For kernel versions below 4.6, ethtool will only
report the negotiated link status.
.sp
The issue is cosmetic and does not affect functionality. Installing the latest
ice driver and upgrading your kernel to version 4.6 or higher will resolve the
issue.
.SS AF_XDP fails to allocate buffers
.sp
On kernels older than 5.3, you may see an undesirable CPU load during packet
processing if you enable AF_XDP in native mode and the Rx ring size is larger
than the UMEM fill queue. This is due to a known issue in the kernel and was
fixed in 5.3. To address the issue, upgrade your kernel to 5.3 or newer.
.SS SCTP checksum offloads aren\(aqt indicated on Geneve tunnel
.sp
For SCTP traffic over a Geneve tunnel, the SCTP checksum isn\(aqt offloaded to the
device, even when tx\-checksum\-sctp is on. This is due to a limitation in the
Linux kernel. However, for Rx traffic, the SCTP checksum is verified if
rx\-checksumming is on. For both Tx and Rx traffic, you can offload the outer
UDP checksum to the device.
.SS CentOS* 7.2 Issues
.sp
The following issues are specific to CentOS* 7.2:
.INDENT 0.0
.IP \(bu 2
\fBbase\-r\-fec\fP mode is supposed to be on by default. On CentOS 7.2, Ethtool
\fB\-\-show\-priv\-flags\fP shows that it is off, instead of on.
.IP \(bu 2
\fBethtool \-m <ethX>\fP does not display optical module information as expected.
.IP \(bu 2
You cannot create an ipv6 Intel(R) Ethernet Flow Director rule. For example,
the following returns a bad syntax error:
.INDENT 2.0
.INDENT 3.5
.sp
.nf
.ft C
ethtool \-U p1p1 flow\-type tcp6 src\-ip 3001:1::2:1:1 dst\-ip 3001:1::1:1:1
src\-port 22 dst\-port 23 action 10
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Upgrading to the latest version of the operating system will resolve these
issues.
.SS Incorrect link speed reported on older VF drivers
.sp
Linux distributions with older iavf or i40evf drivers (including Red Hat
Enterprise Linux 8) may show an incorrect link speed on VF interfaces. This
issue is cosmetic and does not affect VF functionality. To resolve the issue,
download the latest iavf driver.
.SS Older VF drivers on Intel Ethernet 800 Series adapters
.sp
Some Windows* VF drivers from Release 22.9 or older may encounter errors when
loaded on a PF based on the Intel Ethernet 800 Series on Linux KVM. You may see
errors and the VF may not load. This issue does not occur starting with the
following Windows VF drivers:
.INDENT 0.0
.IP \(bu 2
v40e64, v40e65: Version 1.5.65.0 and newer
.UNINDENT
.sp
To resolve this issue, download and install the latest iavf driver.
.SS \(dqVF X failed opcode 24\(dq error message in dmesg on host
.sp
With a Microsoft Windows Server 2019 guest machine running on a Linux host, you
may see \fBVF <vf_number> failed opcode 24\fP error messages in dmesg on the host.
This error is benign and does not affect traffic. Installing the latest iavf
driver in the guest will resolve the issue.
.SS Windows guest OSs on a Linux host may not pass traffic across VLANs
.sp
The VF is not aware of the VLAN configuration if you use Load Balancing and
Failover (LBFO) to configure VLANs in a Windows guest. VLANs configured using
LBFO on a VF driver may result in failure to pass traffic.
.SS SR\-IOV virtual functions have identical MAC addresses
.sp
When you create multiple SR\-IOV virtual functions, the VFs may have identical
MAC addresses. Only one VF will pass traffic, and all traffic on other VFs with
identical MAC addresses will fail. This is related to the
\fBMACAddressPolicy=persistent\fP setting in
\fB/usr/lib/systemd/network/99\-default.link\fP\&.
.sp
To resolve this issue, edit the \fB/usr/lib/systemd/network/99\-default.link\fP
file and change the MACAddressPolicy line to \fBMACAddressPolicy=none\fP\&. For more
information, see the systemd.link man page.
.SS MDD events in dmesg when creating maximum number of VLANs on the VF
.sp
When you create the maximum number of VLANs on the VF, you may see MDD events
in dmesg on the host. This is due to the asynchronous design of the iavf
driver. It always reports success to any VLAN requests, but the requests may
fail later. The guest OS could try to send traffic on a VLAN that is not
configured on the VF, which will cause a Malicious Driver Detection (MDD) event
in dmesg on the host.
.sp
This issue is cosmetic. You do not need to reload the PF driver.
.SS \fBip address\fP or \fBip link\fP command displays an error on a single\-port NIC with 245+ VFs
.sp
When you use the \fBip address\fP or \fBip link\fP command on a Linux host configured
with 245 or more VFs on a single\-port adapter, you may encounter a \(dqBuffer too
small for object\(dq error. This is due to a known issue in the iproute2 tools.
Please use ifconfig instead of iproute2. You can install ifconfig via the
net\-tools\-deprecated package.
.SS Symbols mismatch with in\-tree irdma driver
.sp
When out\-of\-tree ice driver is installed alongside in\-tree irdma driver kernel will report
missing symbols after loading out\-of\-tree ice driver:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
irdma: Unknown symbol ice_del_rdma_qset
irdma: Unknown symbol ice_add_rdma_qset
irdma: Unknown symbol ice_rdma_update_vsi_filter
irdma: Unknown symbol ice_rdma_request_reset
irdma: Unknown symbol ice_get_qos_params
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Missing symbols will also be reported by depmod preventing creation of weak\-updates
symlinks when installing signed binary releases of ice.ko driver (available for SLES
and RHEL distributions):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
Warning: weak\-updates symlinks might not be created
depmod: WARNING: /lib/modules/5.14.21\-150400.22\-default/kernel/drivers/infiniband/hw/irdma/irdma.ko.zst needs unknown symbol ice_del_rdma_qset
depmod: WARNING: /lib/modules/5.14.21\-150400.22\-default/kernel/drivers/infiniband/hw/irdma/irdma.ko.zst needs unknown symbol ice_add_rdma_qset
depmod: WARNING: /lib/modules/5.14.21\-150400.22\-default/kernel/drivers/infiniband/hw/irdma/irdma.ko.zst needs unknown symbol ice_rdma_update_vsi_filter
depmod: WARNING: /lib/modules/5.14.21\-150400.22\-default/kernel/drivers/infiniband/hw/irdma/irdma.ko.zst needs unknown symbol ice_rdma_request_reset
depmod: WARNING: /lib/modules/5.14.21\-150400.22\-default/kernel/drivers/infiniband/hw/irdma/irdma.ko.zst needs unknown symbol ice_get_qos_params
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
To supress those messages user have to manually remove in\-tree irdma driver or install
compatible out\-of\-tree irdma driver.
.SH SUPPORT
.sp
For general information, go to the Intel support website at
\fI\%https://www.intel.com/support/\fP
.sp
or the Intel Ethernet Linux project hosted by GitHub at
\fI\%https://github.com/intel/ethernet\-linux\-ice\fP
.sp
If an issue is identified with the released source code on a supported kernel
with a supported adapter, contact Intel Customer Support at
\fI\%https://www.intel.com/content/www/us/en/support/products/36773/ethernet\-products.html\fP
.SH LICENSE
.sp
This program is free software; you can redistribute it and/or modify it under
the terms and conditions of the GNU General Public License, version 2, as
published by the Free Software Foundation.
.sp
This program is distributed in the hope it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.
.sp
You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
St \- Fifth Floor, Boston, MA 02110\-1301 USA.
.sp
The full GNU General Public License is included in this distribution in the
file called \(dqCOPYING\(dq.
.SH TRADEMARKS
.sp
Intel is a trademark or registered trademark of Intel Corporation or its
subsidiaries in the United States and/or other countries.
.sp
Other names and brands may be claimed as the property of others.
.SH AUTHOR
Intel
.SH COPYRIGHT
2017 - 2024, Intel Corporation
.\" Generated by docutils manpage writer.
.